Docker is a platform.

Docker runs natively on Linux or on OS X and Windows through a helper application called boot2docker that creates a Linux Virtual Machine, by using only RAM, to run Docker.

Docker‘s main goal is to allow you to ship distributed applications, that you’ve created previously, by running them as isolated process on what is known as a container, thus it avoids the need of Virtual Machines that implies a resources saving in the involved machines. The isolation also allows to run several containers simultaneously.

So, Docker is an open platform for developing, shipping, and running applications.

The key point is that you can separate your application from the infrastructure and also treats the infrastructure as an application. Everything can be packaged, distributed and deployed anywhere and quickly. It’s said that Docker eliminates the friction between development, QA, and production environments.

Docker has two major components:

• Docker, the container virtualization platform, that has a daemon running on a host server and a client (the docker binary) to talk to the Docker daemon.
• Docker Hub, the SaaS platform for sharing and managing Docker containers.

Docker concepts

• Image (build component of Docker): a Docker environment template, it consists in files (a copy of what is expected to contain) and metadata (information such as environment variables, port mappings and so on) and it has a name (“ubuntu”, for instance)
• Registry (distribution component of Docker): a public or private store that holds images. The public Docker registry is called Docker Hub.
• Container (run component of Docker): a running instance of a Docker image. It’s created from a Docker image, and it can be run, started, stopped, moved, and deleted. Each container is an isolated and secure application platform.

How Does Docker work?

An image is a serie of layers that are combined into a single image by Docker using union file systems (UnionFS, a file system service that allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system)

From a base image, you can add and modify additional layers and there’s no need of rebuilding the entire image if there is a change, you only need replace the added or updated layer.

A container is built from an image, that is read-only. For running the container, Docker adds a read-write layer on top of the image (using UnionFS) and then allocates a network/bridge interface, an IP address, and finally, executes the specified process and captures input and provides output.

The container is isolated thanks to a technology called namespaces (Namespace isolation, where groups of processes are separated such that they cannot “see” resources in other groups)

Docker use these namespaces:

• The pid namespace, for process isolation.
• The net namespace, for managing network interfaces.
• The ipc namespace, for Inter-Process Communication.
• The mnt namespace, for managing mount-points.
• The uts namespace, for changing the hostname.

In order to achieve isolation on running application, Docker uses another technology called control groups (cgroups, it limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes)

Installing Docker

I’m going to use Ubuntu (as Sheldon Cooper said, my favorite Linux-based operating system)

There are three packages available, two of them from Ubuntu, an older KDE3/GNOME2 called docker (warning, this is the suggested package when you try docker version before having installed: “The program ‘docker’ is currently not installed. You can install it by typing: sudo apt-get install docker”) and a newer called docker.io, that is not the most recent Docker release. The third one is a PPA (Personal Package Archives for Ubuntu) for Docker and it’s called lxc-docker.

If you don’t want extra repositories and you don’t want the latest version, just install docker.io (I recommend you to enable tab-completion of Docker commands in BASH if you install it):

$ sudo apt-get update
$ sudo apt-get install docker.io
$ source /etc/bash_completion.d/docker.io

To validate the installation, type:

$ sudo docker version
Client version: 1.0.1
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 990021a
Server version: 1.0.1
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 990021a
$

You need to use sudo because the docker daemon always runs as the root user, that’s because the docker daemon binds to a Unix socket (instead of a TCP port, as it happened until version 0.5.2). By default that Unix socket is owned by the user root, so you need to use the docker command with sudo. You can solve it by giving non-root access to Docker or, even better, you can create a docker group and add users to it.

Otherwise, if you run docker without sudo you will obtain this error message:

$ docker version
Client version: 1.0.1
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 990021a
2015/03/20 11:40:30 Get http:///var/run/docker.sock/v1.12/version: dial unix /var/run/docker.sock: permission denied
$

In order to have the latest version, use the lxc-docker package, that is the one maintained by Docker itself.

There is a script to install the Docker package on Ubuntu, but I’d rather to do it manually. But you can try with:

$ curl -sSL https://get.docker.com/ubuntu/ | sudo sh

To install lxc-docker, your APT system have to deal with https (you can if the file /usr/lib/apt/methods/https exists, it you don’t have it, install the apt-transport-https package)

Then, just add the Docker repository key to the local keychain, and to the apt sources list. Finally, update and install the lxc-docker package:

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
$ sudo sh -c "echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
$ sudo apt-get update
$ sudo apt-get install lxc-docker

Now we have the last version:

$ sudo docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
$

And we can run the basic example, that downloads the ubuntu image, and then start bash in a container. When you’re done, type exit.

$ sudo docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
511136ea3c5a: Pull complete 
f3c84ac3a053: Pull complete 
511136ea3c5a: Download complete 
f3c84ac3a053: Download complete 
a1a958a24818: Download complete 
9fec74352904: Download complete 
d0955f21bf24: Download complete 
Status: Downloaded newer image for ubuntu:latest
root@0f5e5a64c583:/# ls
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr
root@0f5e5a64c583:/# exit
exit
$

Having said that, Docker has changed the instructions for Installing Docker on Ubuntu… today!

It seems easier, but I don’t like it very much. If you have wget installed, you can get the latest Docker package with:

$ wget -qO- https://get.docker.com/ | sh

It prompts for the password and then it downloads and installs… the lxc-docker package.

And then, verify the installation with:

$ sudo docker run hello-world
[sudo] password for Javier: 
Unable to find image 'hello-world:latest' locally
31cbccb51277: Pull complete 
e45a5af57b00: Pull complete 
511136ea3c5a: Already exists 
hello-world:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (Assuming it was not already locally available.)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

For more examples and ideas, visit:
 http://docs.docker.com/userguide/
$

Using Docker

Running applications inside containers

Docker allows you to run applications inside containers with the command docker run.

The command docker run creates a new container from the image name that you specify (the mandatory parameter for run) and then runs the command in it by performing these steps: Docker looks for the image in this computer, if it isn’t installed yet, Docker searches the image at Docker Hub to download it and install it. Once the image is installed, Docker creates a new container and starts the program.

We’ve seen above one example:

$ sudo docker run -t -i ubuntu:14.04.2 /bin/bash

It creates a container from the Official Ubuntu base image (tag 14.04.2) and then it runs a Bash shell command, with a terminal assigned (flag -t), and grabs the standard in of the container(flag -i)

For executing in the background (or daemonized), use the -d flag:

$ sudo docker run -d --name="Javier" ubuntu:14.04 /bin/sh -c "while true; do echo hello world; sleep 1; done"
e84ae64138881b9eaf6ac743c6f0076cfc414f017daef9e59c3d2e1d591eb7b9

Here, the command will run forever. Besides, I have assigned a name to the container (OK, my very name, some egocentricity here) to easily discover it later (Docker automatically names any containers that you start, but I’d rather to specify the name by myself). Furthermore, Docker returns the container ID (e84ae6413888…).

You can find both the ID and the name when listing containers with the command docker ps (flag -a to show all regardless they are running or not)

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
e84ae6413888        ubuntu:14.04        "/bin/sh -c 'while t   5 days ago          Up 2 minutes                            Javier              
$

The ports in the container can be exposed randomly with the -P flag or manually with -p. In any case, you can see in the column PORTS of the docker ps command or with the docker port command. Let’s see an example with a sample web application image.

$ sudo docker run -d -P training/webapp python app.py
Unable to find image 'training/webapp:latest' locally
Pulling repository training/webapp
31fa814ba25a: Download complete 
511136ea3c5a: Download complete 
f10ebce2c0e1: Download complete 
82cdea7ab5b5: Download complete 
5dbd9cb5a02f: Download complete 
74fe38d11401: Download complete 
64523f641a05: Download complete 
0e2afc9aad6e: Download complete 
e8fc7643ceb1: Download complete 
733b0e3dbcee: Download complete 
a1feb043c441: Download complete 
e12923494f6a: Download complete 
a15f98c46748: Download complete 
Status: Downloaded newer image for training/webapp:latest
a03ec94ea8087789d605ca91d6689d8026e7806e9138b8b9b7ed7f5a1295db85
$ sudo docker ps
CONTAINER ID        IMAGE                    COMMAND             CREATED             STATUS              PORTS                     NAMES
a03ec94ea808        training/webapp:latest   "python app.py"     25 seconds ago      Up 24 seconds       0.0.0.0:49153->5000/tcp   happy_pare          
$ sudo docker port happy_pare
5000/tcp -> 0.0.0.0:49153
$ curl http://localhost:49153
Hello world!
$

This means that you can access the application running in the container by using the port 49153 locally.

To manage the container, you can use the next commands:

• docker logs to see the standard output of a container (-f to view the new additions, press Ctrl+C to exit)
• docker top to see the process in the container.
• docker inspect to see configuration and status information about a container.
• docker stop/kill to stop or kills (respectively) a running container.
• docker start to restart a stopped container (remember the command docker ps -a).
• docker rm to remove a container.
• docker version to see the current version of the program, its programming language (Go) and so on.

$ sudo docker logs happy_pare
 * Running on http://0.0.0.0:5000/
172.17.42.1 - - [31/Mar/2015 09:51:38] "GET / HTTP/1.1" 200 -
$ sudo docker stop happy_pare
happy_pare
$ sudo docker start happy_pare
happy_pare
$ sudo docker kill happy_pare
happy_pare
$ sudo docker rm happy_pare
happy_pare
$ sudo docker top happy_pare
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                8580                1372                0                   12:00               ?                   00:00:00            python app.py
$ sudo docker inspect hungry_kirch
[{
    "AppArmorProfile": "",
    "Args": [
        "app.py"
    ],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "python",
            "app.py"
        ],
        "CpuShares": 0,
        "Cpuset": "",
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "HOME=/",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "ExposedPorts": {
            "5000/tcp": {}
        },
        "Hostname": "468cd593eada",
        "Image": "training/webapp",
        "MacAddress": "",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OnBuild": null,
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": null,
        "WorkingDir": "/opt/webapp"
    },
    "Created": "2015-03-31T10:00:32.697111567Z",
    "Driver": "aufs",
    "ExecDriver": "native-0.2",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "CapAdd": null,
        "CapDrop": null,
        "ContainerIDFile": "",
        "Devices": [],
        "Dns": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "IpcMode": "",
        "Links": null,
        "LxcConf": [],
        "NetworkMode": "bridge",
        "PidMode": "",
        "PortBindings": {},
        "Privileged": false,
        "PublishAllPorts": true,
        "ReadonlyRootfs": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": ""
        },
        "SecurityOpt": null,
        "VolumesFrom": null
    },
    "HostnamePath": "/var/lib/docker/containers/468cd593eadaea6d18441a33ca6c1ea42f1b398d6fd028fa5b557181a5cf36f3/hostname",
    "HostsPath": "/var/lib/docker/containers/468cd593eadaea6d18441a33ca6c1ea42f1b398d6fd028fa5b557181a5cf36f3/hosts",
    "Id": "468cd593eadaea6d18441a33ca6c1ea42f1b398d6fd028fa5b557181a5cf36f3",
    "Image": "31fa814ba25ae3426f8710df7a48d567d4022527ef2c14964bb8bc45e653417c",
    "MountLabel": "",
    "Name": "/hungry_kirch",
    "NetworkSettings": {
        "Bridge": "docker0",
        "Gateway": "172.17.42.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.10",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "fe80::42:acff:fe11:a",
        "LinkLocalIPv6PrefixLen": 64,
        "MacAddress": "02:42:ac:11:00:0a",
        "PortMapping": null,
        "Ports": {
            "5000/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "49155"
                }
            ]
        }
    },
    "Path": "python",
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/468cd593eadaea6d18441a33ca6c1ea42f1b398d6fd028fa5b557181a5cf36f3/resolv.conf",
    "RestartCount": 0,
    "State": {
        "Error": "",
        "ExitCode": 0,
        "FinishedAt": "0001-01-01T00:00:00Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 8580,
        "Restarting": false,
        "Running": true,
        "StartedAt": "2015-03-31T10:00:32.898205331Z"
    },
    "Volumes": {},
    "VolumesRW": {}
}
]
$

Working with Docker Images

As we have explained, Docker runs container by using images that are either installed in your system or exists at Docker Hub (in order to download it and install in your system)

You can see what images are already installed with the docker images command.

$ sudo docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu                  trusty-20150320     d0955f21bf24        11 days ago         188.3 MB
ubuntu                  14.04               d0955f21bf24        11 days ago         188.3 MB
ubuntu                  14.04.2             d0955f21bf24        11 days ago         188.3 MB
ubuntu                  latest              d0955f21bf24        11 days ago         188.3 MB
ubuntu                  trusty              d0955f21bf24        11 days ago         188.3 MB
training/webapp         latest              31fa814ba25a        10 months ago       278.8 MB
$

If you want to manually install an image before running it, you can use the docker pull command for images that you can find at Docker Hub or with the command docker search.

As a curiosity, if you look for images at Docker Hub, you can find Official repos, for instance, the Java OFFICIAL REPO.

You can add a new name to the image with the docker tag.

If you want to remove an image, you can do so with the docker rmi command.

A note about the official images

The images and relevant files are maintained at GitHub by an organization called docker-library (Docker is open source, and it’s maintained at GitHub by an organization called Docker, that has several repositories, including the one for docker).

The official images exist in a repository, Docker Official Images, that contains a folder for the library definitions, for instance, the one for java.

The image packages are also maintained by docker-library in each corresponding repository, for instance, the Docker Official Image packaging for Java (openJDK)

It’s worth to mention it because there is another organization at GitHub called (Trusted Automated Docker Builds) that has repositories for several Docker builds, for instance, the one for Java, that includes an image for Oracle Java 8 JDK (that I was looking for)

Creating your own images

There are two ways for updating and creating images.

1. Run a container from an image (docker run), then update it, and finally commit the results to an image, with the docker commit command.
2. Use a Dockerfile to create an image.

When you’re done, you can push the created image to Docker Hub with the docker push command.

Finally, you can remove images with the docker rmi command.

$ sudo docker run -t -i ubuntu:latest /bin/bash
root@98bec5327540:/# sudo apt-get install --reinstall software-properties-common
root@98bec5327540:/# sudo add-apt-repository ppa:webupd8team/java
root@98bec5327540:/# sudo apt-get update
root@98bec5327540:/# sudo apt-get install oracle-java8-installer
root@98bec5327540:/# sudo apt-get install nano
root@98bec5327540:/# wget http://apache.rediris.es/maven/maven-3/3.3.1/binaries/apache-maven-3.3.1-bin.tar.gz
root@98bec5327540:/# tar -xvf apache-maven-3.3.1-bin.tar.gz 
root@98bec5327540:/# cp -r apache-maven-3.3.1 /usr/local/apache-maven
root@98bec5327540:/# sudo nano /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/apache-maven/bin"
JAVA_HOME="/usr/lib/jvm/java-8-oracle"
MAVEN_OPTS="-Xms256m -Xmx512m"
root@98bec5327540:/# source /etc/environment 
root@98bec5327540:/# echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle
root@98bec5327540:/# java -version
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
root@98bec5327540:/# mvn -version
Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 2015-03-13T20:10:27+00:00)
Maven home: /usr/local/apache-maven
Java version: 1.8.0_40, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "3.13.0-48-generic", arch: "amd64", family: "unix"
root@98bec5327540:/# exit
$ sudo docker ps -a
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS                        PORTS               NAMES
98bec5327540        ubuntu:14.04             "/bin/bash"            47 minutes ago      Exited (0) 4 minutes ago                          determined_carson   
468cd593eada        training/webapp:latest   "python app.py"        2 hours ago         Exited (137) 2 minutes ago                        hungry_kirch        
e84ae6413888        ubuntu:14.04             "/bin/sh -c 'while t   5 days ago          Exited (137) 3 hours ago                          Javier              
0f5e5a64c583        ubuntu:14.04             "/bin/bash"            11 days ago         Exited (0) 11 days ago                            dreamy_hopper     

$ sudo docker commit -m "Ubuntu latest (14.04) with  Oracle Java 8 JDK and Apache Maven 3.3.1 (and nano editor)" -a "Javier Beneito Barquero" 98bec5327540 jbbarquero/ubuntu-java8_oracle-maven
e262639379c46afa53eca74de9df9bd2f81e0ab7839a3472cafb67d7e199d85e
$

$ sudo docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jbbarquero/ubuntu-java8_oracle-maven   latest              e262639379c4        56 seconds ago      828 MB
ubuntu                                 14.04               d0955f21bf24        11 days ago         188.3 MB
ubuntu                                 14.04.2             d0955f21bf24        11 days ago         188.3 MB
ubuntu                                 latest              d0955f21bf24        11 days ago         188.3 MB
ubuntu                                 trusty              d0955f21bf24        11 days ago         188.3 MB
ubuntu                                 trusty-20150320     d0955f21bf24        11 days ago         188.3 MB
centos                                 latest              88f9454e60dd        3 weeks ago         210 MB
hello-world                            latest              e45a5af57b00        12 weeks ago        910 B
training/webapp                        latest              31fa814ba25a        10 months ago       278.8 MB
$ sudo docker tag jbbarquero/ubuntu-java8_oracle-maven my-java8
$ sudo docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
jbbarquero/ubuntu-java8_oracle-maven   latest              e262639379c4        About a minute ago   828 MB
my-java8                               latest              e262639379c4        About a minute ago   828 MB
ubuntu                                 trusty-20150320     d0955f21bf24        11 days ago          188.3 MB
ubuntu                                 14.04               d0955f21bf24        11 days ago          188.3 MB
ubuntu                                 14.04.2             d0955f21bf24        11 days ago          188.3 MB
ubuntu                                 latest              d0955f21bf24        11 days ago          188.3 MB
ubuntu                                 trusty              d0955f21bf24        11 days ago          188.3 MB
centos                                 latest              88f9454e60dd        3 weeks ago          210 MB
hello-world                            latest              e45a5af57b00        12 weeks ago         910 B
training/webapp                        latest              31fa814ba25a        10 months ago        278.8 MB

$ sudo docker run -t -i my-java8 /bin/bash
root@da1045b88138:/# mvn -version
Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 2015-03-13T20:10:27+00:00)
Maven home: /usr/local/apache-maven
Java version: 1.8.0_40, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "3.13.0-48-generic", arch: "amd64", family: "unix"
root@da1045b88138:/# exit
exit
$ 

$ sudo docker push jbbarquero/ubuntu-java8_oracle-maven
The push refers to a repository [jbbarquero/ubuntu-java8_oracle-maven] (len: 1)
Sending image list

Please login prior to push:
Username: jbbarquero
Password: 
Email: jbbarquero@gmail.com
Login Succeeded
The push refers to a repository [jbbarquero/ubuntu-java8_oracle-maven] (len: 1)
Sending image list
Pushing repository jbbarquero/ubuntu-java8_oracle-maven (1 tags)
511136ea3c5a: Image already pushed, skipping 
f3c84ac3a053: Image already pushed, skipping 
a1a958a24818: Image already pushed, skipping 
9fec74352904: Image already pushed, skipping 
d0955f21bf24: Image already pushed, skipping 
e262639379c4: Image successfully pushed 
Pushing tag for rev [e262639379c4] on {https://cdn-registry-1.docker.io/v1/repositories/jbbarquero/ubuntu-java8_oracle-maven/tags/latest}
$

You can find this image at jbbarquero / ubuntu-java8_oracle-maven

How to write a Dockerfile is out of the bounds of this first post, but I can create a very basic one. Just for fun.

Having this Dockerfile:

# This is a comment
FROM jbbarquero/ubuntu-java8_oracle-maven
MAINTAINER Javier Beneito Barquero <jbbarquero@gmail.com>
RUN apt-get update && apt-get install -y git

Just build it (and then push it)

$ sudo docker build -t jbbarquero/ubuntu-java8_oracle-maven-git .
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon 
Step 0 : FROM jbbarquero/ubuntu-java8_oracle-maven
 ---> e262639379c4
Step 1 : MAINTAINER Javier Beneito Barquero 
 ---> Running in d1be5fc2ed6e
 ---> 162fdb8b3a3d
Removing intermediate container d1be5fc2ed6e
Step 2 : RUN apt-get update && apt-get install -y git
 ---> Running in b7c2cf318638

... Lot of installation messages here

 ---> f63b88cf14ab
Removing intermediate container b7c2cf318638
Successfully built f63b88cf14ab
$ sudo docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jbbarquero/ubuntu-java8_oracle-maven-git   latest              f63b88cf14ab        17 seconds ago      876.5 MB
...

$ sudo docker run -t -i jbbarquero/ubuntu-java8_oracle-maven-git /bin/bash
root@bb9caa298372:/# git --version
git version 1.9.1
root@bb9caa298372:/# mvn -version
Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 2015-03-13T20:10:27+00:00)
Maven home: /usr/local/apache-maven
Java version: 1.8.0_40, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "3.13.0-48-generic", arch: "amd64", family: "unix"
root@bb9caa298372:/# exit
exit

The new image can be found in Docker Hub, find it at jbbarquero / ubuntu-java8_oracle-maven-git.

Closing words

That’s enough for now. There are a couple of interesting topics, but we’ll leave for another time:

• Linking Containers, for sending information between containers in Docker.
• Data in containers, for managing data volumes.

Resources

• Docker official documentation
• Docker 10-minute tutorial
• Docker installation: Ubuntu
• Docker in Action. By Jeff Nickoloff (Manning)
• Docker in Practice. By Ian Miell and Aidan Hobson Sayers (Manning)
• Install oracle java 8 in ubuntu via ppa repository [jdk8]
• How to add a PPA on a server?
• Nano at Ubuntu
• Download Apache Maven 3.3.1

Post scríptum

When re-starting the container jbbarquero/ubuntu-java8_oracle-maven, source /etc/environment is not called, so the values that I wrote there are not applied. For solving this issue, I edited /etc/bash.bashrc and I put there the environment variables.

root@98bec5327540:/# sudo nano /etc/bash.bashrc
$ sudo docker start -i 98bec5327540
root@98bec5327540:/# sudo nano /etc/bash.bashrc

JAVA_HOME="/usr/lib/jvm/java-8-oracle"
export JAVA_HOME

M2_HOME=/usr/local/apache-maven
export M2_HOME
M2=$M2_HOME/bin
export M2

PATH=$PATH:$JAVA_HOME
PATH=$PATH:$M2
export PATH

root@98bec5327540:/# exit

Maybe a Dockerfile is a better idea to create containers with environment variables using ENV.
so the values that I wrote there

We also saw a sample application and two ways of running it: by invoking a JobLauncher bean or by using CommandLineJobRunner from the command line.

In this blog entry, we’ll see two additional ways to run a Spring Batch job:

1. Using JobOperator, in order to have control of the batch process, from start a job to monitoring tasks such as stopping, restarting, or summarizing a Job. We’ll only pay attention to the start operation, but once a JobOperator is configured, you can use for the remaining monitoring tasks.
2. Using Spring Boot, the new convention-over-configuration centric framework from the Spring team, that allows you to create with a few lines of code applications that “just run”, because Spring Boot provides a lot of features based on what you have in your classpath.

As usual, all the source code is available at GitHub.

Running the sample: JobOperator

JobOperator is an interface that provides operations for inspecting and controlling jobs, mainly for a command-line client or a remote launcher like a JMX console.

The implementation that Spring Batch provides, SimpleJobOperator, uses JobLauncher, JobRepository, JobExplorer, and JobRegistry for performing its operations. They are created by the @EnableBatchProcessing annotation, so we can create an additional batch configuration file with these dependencies autowired and later import it in the batch configuration file (not without issues, due to the way Spring loads the application context, we’ll see this shortly):

@Configuration
public class AdditionalBatchConfiguration {

    @Autowired
    JobRepository jobRepository;
    @Autowired
    JobRegistry jobRegistry;
    @Autowired
    JobLauncher jobLauncher;
    @Autowired
    JobExplorer jobExplorer;

    @Bean
    public JobOperator jobOperator() {
        SimpleJobOperator jobOperator = new SimpleJobOperator();
        jobOperator.setJobExplorer(jobExplorer);
        jobOperator.setJobLauncher(jobLauncher);
        jobOperator.setJobRegistry(jobRegistry);
        jobOperator.setJobRepository(jobRepository);
        return jobOperator;
    }

}

And the @Import:

@Configuration
@EnableBatchProcessing
@Import(AdditionalBatchConfiguration.class)
public class BatchConfiguration {

	// Omitted

}

Now it seems easy to run the job with a main class:

@Component
public class MainJobOperator {

    @Autowired
    JobOperator jobOperator;

    @Autowired
    Job importUserJob;

    public static void main(String... args) throws JobParametersInvalidException, JobInstanceAlreadyExistsException, NoSuchJobException, DuplicateJobException, NoSuchJobExecutionException {

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ApplicationConfiguration.class);

        MainJobOperator main = context.getBean(MainJobOperator.class);
        long executionId = main.jobOperator.start(main.importUserJob.getName(), null);

        MainHelper.reportResults(main.jobOperator, executionId);
        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

        context.close();

        System.out.printf("\nFIN %s", main.getClass().getName());

    }
}

But there’s a little problem… it doesn’t work:

Exception in thread "main" org.springframework.batch.core.launch.NoSuchJobException: No job configuration with the name [importUserJob] was registered
	at org.springframework.batch.core.configuration.support.MapJobRegistry.getJob(MapJobRegistry.java:66)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy14.getJob(Unknown Source)
	at org.springframework.batch.core.launch.support.SimpleJobOperator.start(SimpleJobOperator.java:310)
	at com.malsolo.springframework.batch.sample.MainJobOperator.main(MainJobOperator.java:15)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

Process finished with exit code 1

The problem here, No job configuration with the name [importUserJob] was registered, is due to the way that JobOperator.start(String jobName, String parameters) works.

The main difference with JobLauncher.run(Job job, JobParameters jobParameters) is that the former has String as parameters while the latter uses objects directly.

So JobOperator, actually SimpleJobOperator, has to obtain a Job with the provided name. In order to do so, it uses the JobRegistry.getJob(String name) method. The available Spring Batch implementation is MapJobRegistry that uses a ConcurrentMap to store using the job name as the key, a JobFactory to create the Job when requested.

The problem is that this map has not been populated.

The first solution is easy: the JobRegistry allows you to register at runtime a JobFactory to later obtain the Job as explained above. So, we only need to create this JobFactory…

@Configuration
public class AdditionalBatchConfiguration {
//    Rest omitted
    @Autowired
    Job importUserJob;

    @Bean
    public JobFactory jobFactory() {
        return new ReferenceJobFactory(importUserJob);
    }
}

…and register it in the main method:

@Component
public class MainJobOperator {

    @Autowired
    JobFactory jobFactory;
    @Autowired
    JobRegistry jobRegistry;
    @Autowired
    JobOperator jobOperator;

    @Autowired
    Job importUserJob;

    public static void main(String... args) throws JobParametersInvalidException, JobInstanceAlreadyExistsException, NoSuchJobException, DuplicateJobException, NoSuchJobExecutionException {

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ApplicationConfiguration.class);

        MainJobOperator main = context.getBean(MainJobOperator.class);
        main.jobRegistry.register(main.jobFactory);
        long executionId = main.jobOperator.start(main.importUserJob.getName(), null);

        MainHelper.reportResults(main.jobOperator, executionId);
        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

        context.close();

        System.out.printf("\nFIN %s", main.getClass().getName());

    }
}

And now it works:

***********************************************************
JobExecution: id=0, version=2, startTime=2014-09-04 13:03:37.964, endTime=2014-09-04 13:03:38.141, lastUpdated=2014-09-04 13:03:38.141, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=0, version=0, Job=[importUserJob]], jobParameters=[{}]
* Steps executed:
StepExecution: id=0, version=3, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
***********************************************************
***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************

But I don’t like this approach, it’s too manual.

I’d rather to populate the JobRegistry automatically, and Spring Batch provides two mechanisms for doing so, a bean post-processor, JobRegistryBeanPostProcessor, and a component that loads and unloads Jobs by creating child context and registering jobs from those contexts as they are created, AutomaticJobRegistrar.

We’ll see the post-processor approach, because it’s very easy. Just declare the bean in the Batch configuration and run the original main class.

@Configuration
@EnableBatchProcessing
@Import(AdditionalBatchConfiguration.class)
public class BatchConfiguration {

	// Omitted

    @Bean
    public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
        JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
        jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
        return jobRegistryBeanPostProcessor;
    }

}

The bean post processor has to be declared in this configuration file for registering the job when it’s created (this is the issue that I mentioned before, if you declare the post processor in another java file configuration, for instance in the AdditionalBatchConfiguration it will never receive the job bean). It uses the same JobRegistry that uses the JobOperator to launch the Job. Actually the only that exists, but it’s good to know this.

It also works:

***********************************************************
JobExecution: id=0, version=2, startTime=2014-09-04 13:20:07.343, endTime=2014-09-04 13:20:07.522, lastUpdated=2014-09-04 13:20:07.522, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=0, version=0, Job=[importUserJob]], jobParameters=[{}]
* Steps executed:
StepExecution: id=0, version=3, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
***********************************************************
***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************

Running the sample: Spring Boot

We’d like to try how quickly and easy is Spring Boot for launching Spring Batch applications once we already have a functional configuration.

And it seems to be a piece of cake (the problem here is to know what is happening under the hood)

package com.malsolo.springframework.batch.sample;
@ComponentScan(excludeFilters = {@ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, value = ApplicationConfiguration.class)})
@EnableAutoConfiguration
public class MainBoot {

    public static void main(String... args) {

        ApplicationContext context = SpringApplication.run(MainBoot.class);

        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

    }
}

Actually, Spring Boot is out of the bounds of this topic, it deserves its own entry (even, an entire book) but we can summarize the important code here:

• Line 3: @EnableAutoConfiguration, with this annotation you want Spring Boot to instantiate the beans that you’re going to need based on the libraries on your classpath.
• Line 8: the run method, to bootstrap the application by passing the class itself (in our case, MainBoot) that serves as the primary Spring component.

It’s enough to run the Batch application:

.   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.1.4.RELEASE)

2014-09-04 16:58:55.109  INFO 15646 --- [           main] c.m.s.batch.sample.MainBoot              : Starting MainBoot on jbeneito-Latitude-3540 with PID 15646 (/home/jbeneito/Documents/git/spring-batch-sample/target/classes started by jbeneito in /home/jbeneito/Documents/git/spring-batch-sample)
2014-09-04 16:58:55.237  INFO 15646 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@6366ebe0: startup date [Thu Sep 04 16:58:55 CEST 2014]; root of context hierarchy
2014-09-04 16:58:56.039  INFO 15646 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'jdbcTemplate': replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=batchConfiguration; factoryMethodName=jdbcTemplate; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/malsolo/springframework/batch/sample/BatchConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=mainForSpringInfo; factoryMethodName=jdbcTemplate; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/malsolo/springframework/batch/sample/MainForSpringInfo.class]]
2014-09-04 16:58:56.691  WARN 15646 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.stepScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean Javadoc for complete details
2014-09-04 16:58:56.724  WARN 15646 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.jobScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean Javadoc for complete details
2014-09-04 16:58:56.729  INFO 15646 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2014-09-04 16:58:56.975  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'batchConfiguration' of type [class com.malsolo.springframework.batch.sample.BatchConfiguration$$EnhancerBySpringCGLIB$$c3ec56ab] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.145  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [class org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$5a15b25b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.238  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'transactionAttributeSource' of type [class org.springframework.transaction.annotation.AnnotationTransactionAttributeSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.294  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'transactionInterceptor' of type [class org.springframework.transaction.interceptor.TransactionInterceptor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.301  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.config.internalTransactionAdvisor' of type [class org.springframework.transaction.interceptor.BeanFactoryTransactionAttributeSourceAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.374  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceConfiguration' of type [class com.malsolo.springframework.batch.sample.DataSourceConfiguration$$EnhancerBySpringCGLIB$$18a97d02] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.455  INFO 15646 --- [           main] o.s.j.d.e.EmbeddedDatabaseFactory        : Creating embedded database 'testdb'
2014-09-04 16:58:58.156  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [schema-all.sql]
2014-09-04 16:58:58.178  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [schema-all.sql] in 20 ms.
2014-09-04 16:58:58.178  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2014-09-04 16:58:58.189  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 11 ms.
2014-09-04 16:58:58.198  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSource' of type [class org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory$EmbeddedDataSourceProxy] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.206  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration$DataSourceInitializerConfiguration' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration$DataSourceInitializerConfiguration$$EnhancerBySpringCGLIB$$c0608242] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.257  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.datasource.CONFIGURATION_PROPERTIES' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.260  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from URL [file:/home/jbeneito/Documents/git/spring-batch-sample/target/classes/schema-all.sql]
2014-09-04 16:58:58.262  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from URL [file:/home/jbeneito/Documents/git/spring-batch-sample/target/classes/schema-all.sql] in 2 ms.
2014-09-04 16:58:58.263  WARN 15646 --- [           main] o.s.b.a.jdbc.DataSourceInitializer       : Could not send event to complete DataSource initialization (ApplicationEventMulticaster not initialized - call 'refresh' before multicasting events via the context: org.springframework.context.annotation.AnnotationConfigApplicationContext@6366ebe0: startup date [Thu Sep 04 16:58:55 CEST 2014]; root of context hierarchy)
2014-09-04 16:58:58.263  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceInitializer' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.277  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration' of type [class org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$$EnhancerBySpringCGLIB$$85a27e41] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.320  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jobRegistry' of type [class com.sun.proxy.$Proxy25] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.795  INFO 15646 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: HSQL
2014-09-04 16:58:59.056  INFO 15646 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2014-09-04 16:58:59.361  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2014-09-04 16:58:59.381  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 20 ms.
2014-09-04 16:58:59.890  INFO 15646 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2014-09-04 16:58:59.926  INFO 15646 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: []
2014-09-04 16:59:00.023  INFO 15646 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=importUserJob]] launched with the following parameters: [{run.id=1}]
2014-09-04 16:59:00.060  INFO 15646 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [step1]
Converting (firstName: Jill, lastName: Doe) into (firstName: JILL, lastName: DOE)
Converting (firstName: Joe, lastName: Doe) into (firstName: JOE, lastName: DOE)
Converting (firstName: Justin, lastName: Doe) into (firstName: JUSTIN, lastName: DOE)
Converting (firstName: Jane, lastName: Doe) into (firstName: JANE, lastName: DOE)
Converting (firstName: John, lastName: Doe) into (firstName: JOHN, lastName: DOE)
2014-09-04 16:59:00.162  INFO 15646 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=importUserJob]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED]
2014-09-04 16:59:00.164  INFO 15646 --- [           main] c.m.s.batch.sample.MainBoot              : Started MainBoot in 5.963 seconds (JVM running for 8.019)
***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************
2014-09-04 16:59:00.258  INFO 15646 --- [       Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@6366ebe0: startup date [Thu Sep 04 16:58:55 CEST 2014]; root of context hierarchy
2014-09-04 16:59:00.262  INFO 15646 --- [       Thread-1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown

Process finished with exit code 0

As a side note, this class already scans for compontents, so we don’t need to make an additional scan for components, so we exclude the ApplicationConfiguration class with a filter.

Finally, we use the MainHelper class to show a summary of the results.

That’s all for now, because one more time this entry is growing really fast. Thus, we’ll see in the next post the last topic of Spring Batch that I want to talk about: JSR 352.

Resources

• Webinar: Spring Batch 3.0.0 by Michael Minella. Published on Jun 18, 2014.
• JSR-352, Spring Batch and You by Michael Minella. Published on Feb 3, 2014.
• Integrating Spring Batch and Spring Integration by Gunnar Hillert, Michael Minella. Published on Jul 9, 2014.
• Pro Spring Batch (Expert’s Voice in Spring) by Michael Minella. Published on July 12, 2011 by Apress.
• Spring Batch in Action by Arnaud Cogoluegnes, Thierry Templier, Gary Gregory, Olivier Bazoud. Published on October 10, 2011 by Manning Publications.
• Spring.io GETTING STARTED GUIDE: Creating a Batch Service.
• Spring Batch – Reference Documentation
• Spring Batch – API specification

Michael T. Minella, project lead of Spring Batch and also a member of the JSR 352 (Batch Applications for the Java Platform) expert group, wrote in his book Pro Spring Batch the next definition “Batch processing […] is defined as the processing of data without interaction or interruption. Once started, a batch process runs to some form of completion without any intervention“.

Typically Batch Jobs are long-running, non-interactive and process large volumes of data, more than fits in memory or a single transaction. Thus they usually run outside office hours and include logic for handling errors and restarting if necessary.

Spring Batch provides, among others, the next features:

• Transaction management, to allow you to focus on business processing.
• Chunk based processing, to process a large value of data by dividing it in small pieces.
• Start/Stop/Restart/Skip/Retry capabilities, to handle non-interactive management of the process.
• Web based administration interface (Spring Batch Admin), it provides an API for administering tasks.
• Based on Spring framework, so it includes all the configuration options, including Dependency Injection.
• Compliance with JSR 352: Batch Applications for the Java Platform.

Spring Batch concepts

Batch Stereotypes

• Job: an entity that encapsulates an entire batch process. It is composed of one or more ordered Steps and it has some properties such as restartability.
• Step: a domain object that encapsulates an independent, sequential phase of a batch job.
• Item: the individual piece of data that it’s been processed.
• Chunk: the processing style used by Spring Batch: read and process the item and then aggregate until reach a number of items, called “chunk” that will be finally written.


Chunk-Oriented Processing

• JobLauncher: the entry point to launch Spring Batch jobs with a given set of JobParameters.
• JobRepository: maintains all metadata related to job executions and provides CRUD operations for JobLauncher, Job, and Step implementations.

Running a Job

The JobLauncher interface has a basic implementation SimpleJobLauncher whose only required dependency is a JobRepository, in order to obtain an execution, so that you can use it for executing the Job.

JobLauncher

You can also launch a Job asynchronously by configuring a TaskExecutor. You can also this configuration for running Jobs from within a Web Container.

Job launcher sequence async

A JobLauncher uses the JobRepository to create new JobExecution objects and run them.

Job Repository

Running Jobs: concepts

The main concepts related with Job execution are

Job hierarchy with steps

• JobInstance: a logical run of a Job.
• JobParameters: a set of parameters used to start a batch job. It categorizes each JobInstance.
• JobExecution: physical runs of Jobs, in order to know what happens with the execution.
• StepExecution: a single attempt to execute a Step, that is created each time a Step is run and it also provides information regarding the result of the processing.
• ExecutionContext: a collection of key/value pairs that are persisted and controlled by the framework in order to allow developers a place to store persistent state that is scoped to a StepExecution or JobExecution.

Sample application

Now we are going to see a simple sample application that reads a POJO that represents a Person from a file containing People data and after processing each of them, that is just uppercase its attributes, saving them in a database.

All the code is available at GitHub.

Let’s begin with the basic domain class: Person, just a POJO.

package com.malsolo.springframework.batch.sample;

public class Person {
    private String lastName;
    private String firstName;
    //...
}

Then, let’s see the simple processor, PersonItemProcessor. It implements an ItemProcessor, with a Person both as Input and Output.

It provides a method to be overwritten, process, that allows you to write the custom transformation.

package com.malsolo.springframework.batch.sample;

import org.springframework.batch.item.ItemProcessor;

public class PersonItemProcessor implements ItemProcessor<Person, Person> {

    @Override
    public Person process(final Person person) throws Exception {
        final String firstName = person.getFirstName().toUpperCase();
        final String lastName = person.getLastName().toUpperCase();

        final Person transformedPerson = new Person(firstName, lastName);

        System.out.println("Converting (" + person + ") into (" + transformedPerson + ")");

        return transformedPerson;
    }

}

Once done this, we can proceed to configure the Spring Batch Application, for doing so, we’ll use Java Annotations in a BatchConfiguration file:

// Imports and package omitted
@Configuration
@EnableBatchProcessing
@Import(AdditionalBatchConfiguration.class)
public class BatchConfiguration {

	// Input, processor, and output definition
	
	@Bean
    public ItemReader<Person> reader() {
		FlatFileItemReader<Person> reader = new FlatFileItemReader<Person>();
		reader.setResource(new ClassPathResource("sample-data.csv"));
		reader.setLineMapper(new DefaultLineMapper<Person>() {{
			setLineTokenizer(new DelimitedLineTokenizer() {{
				setNames(new String[] {"firstName", "lastName"});
			}});
			setFieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
				setTargetType(Person.class);
			}});
			
		}});
		return reader;
	}
	
	@Bean
    public ItemProcessor<Person, Person> processor() {
        return new PersonItemProcessor();
    }
	
	@Bean
    public ItemWriter<Person> writer(DataSource dataSource) {
		JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<Person>();
		writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
		writer.setSql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)");
		writer.setDataSource(dataSource);
		return writer;
	}
	
	//  Actual job configuration
	
	@Bean
    public Job importUserJob(JobBuilderFactory jobs, Step s1) {
		return jobs.get("importUserJob")
				.incrementer(new RunIdIncrementer())
				.flow(s1)
				.end()
				.build();
	}
	
	@Bean
    public Step step1(StepBuilderFactory stepBuilderFactory, ItemReader<Person> reader,
            ItemWriter<Person> writer, ItemProcessor<Person, Person> processor) {
		return stepBuilderFactory.get("step1")
				.<Person, Person> chunk(10)
				.reader(reader)
				.processor(processor)
				.writer(writer)
				.build();
	}
	
	@Bean
    public JdbcTemplate jdbcTemplate(DataSource dataSource) {
        return new JdbcTemplate(dataSource);
    }
	
}

Highlights for this class are:

• Line 2: @Configuration, this class will be processed by the Spring container to generate bean definitions.
• Line 3: @EnableBatchProcessing, provides a base configuration for building batch jobs by creating the next beans beans available to be autowired:
  • JobRepository – bean name “jobRepository”
  • JobLauncher – bean name “jobLauncher”
  • JobRegistry – bean name “jobRegistry”
  • PlatformTransactionManager – bean name “transactionManager”
  • JobBuilderFactory – bean name “jobBuilders”
  • StepBuilderFactory – bean name “stepBuilders”

  We’ll see shortly how it works.

• Line 10: the reader bean, an instance of a FlatFileItemReader, that implements the ItemReader interface to read each Person from the file containing people. Spring Batch provides several implementations for this interface, being this implementation that read lines from one Resource one of them.You know, no need of custom code.
• Line 26: the processor bean, an instance of the previously defined PersonItemProcessor. See above.
• Line 31: the writer bean, an instance of a JdbcBatchItemWriter, that implements the ItemWriter interface to write the People already processed to the database. It’s also an implementation provided by Spring Batch, so no need of custom code again. In this case, you only have to provide an SQL, and a callback for the parameters. Since we are using named parameters, we’ve chosen a BeanPropertyItemSqlParameterSourceProvider. This bean also needs a DataSource, so we provided it by passing one as a method parameter in order to Spring inject the instance that it has registered.
• Line 42: a Job bean, that it’s built using the JobBuilderFactory that is autowired by passing it as method parameter for this @Bean method. When you call its get method, Spring Batch will create a job builder and will initialize its job repository, the JobBuilder is the convenience class for building jobs of various kinds as you can see in the code above. We also use a Step that is configured as the next Spring bean.
• Line 51: a Step bean, that it’s built using the StepBuilderFactory that is autowired by passing it as method parameter for this @Bean method, as well as the other dependencies: the reader, the processor and the writer previously defined. When calling the get method from the StepBuilderFactory, Spring Batch will create a step builder and will initialize its job repository and transaction manager, the StepBuilder is an entry point for building all kinds of steps as you can see in the code above.

This configuration is almost everything needed to configure a Batch process as defined in the concepts above.

Actually, only one configuration class needs to have the @EnableBatchProcessing annotation in order to have the base configuration for building batch jobs. Then you can define the job with their steps and the readers/processors/writers that they need.

But an additional data source is needed to be used by the JobRepository. For this sample we’ll use an in-memory one:

@Configuration
public class DataSourceConfiguration {

    @Bean
    public DataSource dataSource() {
        EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
        return builder
                .setType(HSQL)
                .addScript("schema-all.sql")
                .addScript("org/springframework/batch/core/schema-hsqldb.sql")
                .build();
    }

}

In this case we’ll use the same in-memory database, HSQL, with the schema for the application (line 9) and the schema for the job repository (line 10). The former is available as a resource of the application, the file called schema-all.sql, and the latter in the spring-batch-core jar (spring-batch-core-3.0.1.RELEASE.jar at the time of this writing)

Alternate Configuration

The official documentation shows an slightly different configuration by using the @Autowired annotation for the beans that @EnableBatchProcessing will create. Use the one that you like most. In this case they also imports the data base configuration.

@Configuration
@EnableBatchProcessing
@Import(DataSourceConfiguration.class)
public class AppConfig {

    @Autowired
    private JobBuilderFactory jobs;

    @Autowired
    private StepBuilderFactory steps;

    // Input, processor, and output definition omitted

    @Bean
    public Job importUserJob() {
        return jobs.get("importUserJob").incrementer(new RunIdIncrementer()).flow(step1()).end().build();
    }

    @Bean
    protected Step step1(ItemReader<Person> reader, ItemProcessor<Person, Person> processor, ItemWriter<Person> writer) {
        return steps.get("step1")
            .<Person, Person> chunk(10)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .build();
    }

}

We chose another approach: we load it when configuring the application in the main method as you’ll see shortly. besides, we imported an additional batch configuration (see line 28 at BatchConfiguration.java) to provide an alternate way to launch the application.

Enable Batch Processing: how it works

As we said before, we will go a little deeper in how the annotation @EnableBatchProcessing works.

To remind its goal, this annotation provides a base configuration for building batch jobs by creating a list of beans available to be autowired. An extract of the source code gives us a lot of information:

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(BatchConfigurationSelector.class)
public @interface EnableBatchProcessing {

	/**
	 * Indicate whether the configuration is going to be modularized into multiple application contexts. If true then
	 * you should not create any @Bean Job definitions in this context, but rather supply them in separate (child)
	 * contexts through an {@link ApplicationContextFactory}.
	 */
	boolean modular() default false;

}

As you can see at line 4, this annotation imports an implementation of an ImportSelector, one of the options to import beans in a configuration class, in particular, to selective import beans according to certain criteria.

This particular implementation, BatchConfigurationSelector, instantiates the expected beans for providing common structure for enabling and using Spring Batch based in the EnableBatchProcessing’s attribute modular.

There are two implementations depending on whether you want the configuration to be modularized into multiple application contexts so that they don’t interfere with each other with the naming and the uniqueness of beans (for instance, beans named reader) or not. They are ModularBatchConfiguration and SimpleBatchConfiguration respectively. Mainly they both do the same, but the former using an AutomaticJobRegistrar which is responsible for creating separate ApplicationContexts for register isolated jobs that are later accesible via the JobRegistry, and the latter just creates the main components as lazy proxies that only initialize when a method is called (in order to prevent configuration cycles)

The key concept here is that both extends AbstractBatchConfiguration that uses the core interface for this configuration: BatchConfigurer.

The default implementation, DefaultBatchConfigurer, provides the beans mentioned above (jobRepository, jobLauncher, jobRegistry, transactionManager, jobBuilders and stepBuilders), for doing so it doesn’t require a dataSource, it’s Autowired with required to false, so it will use a Map based JobRepository if its dataSource is null, but you have take care if you have a dataSource eligible for autowiring that doesn’t contain the expected database schema for the job repository: the batch process will fail in this case.

Spring Boot provides another implementation, BasicBatchConfigurer, but this is out of the scope of this entry.

With all this information, we already have a Spring Batch application configured, and we more or less know how this configuration is achieved using Java.

Now it’s time to run the application.

Running the sample: JobLauncher

We have all we need to launch a batch job, the Job to be launched and a JobLauncher, so wait no more and execute this main class: MainJobLauncher.

@Component
public class MainJobLauncher {

    @Autowired
    JobLauncher jobLauncher;

    @Autowired
    Job importUserJob;

    public static void main(String... args) throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException {

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ApplicationConfiguration.class);

        MainJobLauncher main = context.getBean(MainJobLauncher.class);

        JobExecution jobExecution = main.jobLauncher.run(main.importUserJob, new JobParameters());

        MainHelper.reportResults(jobExecution);
        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

        context.close();

    }

}

First things first. This is the way I like to write main classes. Some people from Spring are used to writing main classes annotated with @Configuration, but I’d rather to annotate them as @Components in order to separate the actual application and its configuration from the classes that test the functionality.

As Spring component (line 1), it only needs to have the dependencies @Autowired.

That’s the reason for the ApplicationConfiguration class. It’s a @Configuration class that also performs a @ComponentScan from its own package, that will find the very MainJobLauncher and the remain @Configuration classes, because they are also @Components: BatchConfiguration and DataSourceConfiguration.

As a main class, it creates the Spring Application Context (line 12), it gets the component as a Spring bean (line 14) and then it uses its methods (or attributes in this example. Line 16)

Let’s back to the Batch application: the line 16 is the call to the JobLauncher that will run the Spring Batch process.

The remaining lines are intended to show the results, both from the job execution and the results in the database.

It will be something like this:

***********************************************************
importUserJob finished with a status of  (COMPLETED).
* Steps executed:
	step1 : exitCode=COMPLETED;exitDescription=
StepExecution: id=0, version=3, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0
***********************************************************

***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************

Running the sample: CommandLineJobRunner

CommandLineJobRunner is a main class provided by Spring Batch as the primary entry point to launch a Spring Batch Job.

It requires at least two arguments: JobConfigurationXmlPath/JobConfigurationClassName and jobName. With the first, it will create an ApplicationContext by loading a Java Configuration from a class with the same name or by loading an XML Configuration file with the same name.

It has a JobLauncher attribute that is autowired with the application context via its AutowireCapableBeanFactory exposed, that is used to autowire the bean properties by type.

It accepts some options (“-restart”, “-next”, “-stop”, “-abandon”) as well as parameters for the JobLauncher that are converted with the DefaultJobParametersConverter as JobParametersConverter that expects a ‘name=value’ format.

You can declare this main class in the manifest file, directly or using some maven plugin as maven-jar-plugin, maven-shade-plugin or even exec-maven-plugin.

That is, you can invoke from your command line something like this:

$ java CommandLineJobRunner job.xml jobName parameter=value

Well, the sample code is a maven project that you can install (it’s enough if you package the application) and it allows to manage the dependencies (the mvn dependency:copy-dependencies command copies all the dependencies in the target/dependency directory)

To simplify, I’ll also copy the generated jar to the same directory of the dependencies in order to invoke the java command more easily:

~/Documents/git/spring-batch-sample$mvn clean install
[INFO] Scanning for projects...
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
...
~/Documents/git/spring-batch-sample$ mvn dependency:copy-dependencies
[INFO] Scanning for projects...
...
[INFO] Copying spring-batch-core-3.0.1.RELEASE.jar to ~/Documents/git/spring-batch-sample/target/dependency/spring-batch-core-3.0.1.RELEASE.jar
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
...
~/Documents/git/spring-batch-sample$ cp target/spring-batch-sample-0.0.1-SNAPSHOT.jar ./target/dependency/
~/Documents/git/spring-batch-sample$ java -classpath "./target/dependency/*" org.springframework.batch.core.launch.support.CommandLineJobRunner com.malsolo.springframework.batch.sample.ApplicationConfiguration importUserJob
...
12:32:17.039 [main] INFO  o.s.b.c.l.support.SimpleJobLauncher 
- Job: [FlowJob: [name=importUserJob]] 
completed with the following parameters: [{}] 
and the following status: [COMPLETED]
...

That’s all for now.

Since this entry is becoming very large, I’ll explain other ways to run Spring Batch Jobs in a next post.

Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project…: Fatal error compiling: invalid target release: 1.8.0_11 -> [Help 1]

To summarize, I want to compile a maven project using Java 8, that is possible thanks to the JAVA_HOME configuration:

$ mvn -version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T18:37:52+01:00)
Maven home: /home/jbeneito/Applications/apache-maven-3.2.1
Java version: 1.8.0_11, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-32-generic", arch: "amd64", family: "unix"

But I copied the maven compiler plugin configuration from another maven project without putting special attention:

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                    <encoding>${project.build.sourceEncoding}</encoding>
                </configuration>
            </plugin>

In programming the error usually contains the solution. I’m trying to use the current Java version (1.8.0_11) as the option for the javac command, that is not the expected one.

${java.version} is a environment variable whose value is currently 1.8.0_11, but the Maven Compiler Plugin expects the standard options of the javac command for Setting the -source and -target of the Java Compiler.

That is:

• 1.3 for Java SE 1.3
• 1.4 for Java SE 1.4
• 1.5 for Java SE 5
• 5 synonym for 1.5
• 1.6 for Java SE 6
• 6 synonym for 1.6
• 1.7 the default value. The compiler accepts code with features introduced in Java SE 7. (Yes, as it appears in the Java 8 official documentation)
• 7 Synonym for 1.7

Actually I only use the values with the format 1.X, that includes 1.8, that I assume is the default value for Java 8 SE.

Why did I use this variable?

Because in my previous project it wasn’t an environment variable, but a custom property in the pom.xml:

<properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
...
    </properties>

But this time I have another two custom properties:

<properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>

Thus the solution is easy:

1. Try to not copy-paste
2. Pay attention in any case
3. Just use the appropriate variables

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>${maven.compiler.target}</source>
                    <target>${maven.compiler.source}</target>
                    <encoding>${project.build.sourceEncoding}</encoding>
                </configuration>
            </plugin>

Introduction

Since a few years ago, I try to use several OS in order to improve my computing skills: Windows 7/Vista, OS X (Lion currently) and my favorite Linux distribution, Ubuntu (Sheldon Cooper dixit). You can also include Android (my wife’s smartphone) and iOS (my iPhone and our iPad)

Suddenly, I realize that I’ve been enriched, and you’re able to do a lot of more things because you have your mind really prepared for thinking instead of conditioned by a single line of sight.

I observed the same improvement when I began to use Maven for Java development. I didn’t quit my IDE, but regarding compilation, the important things are the source code and bytecode. You shouldn’t think about settings, preferences, project configuration and some other artificial stuff that the IDE needs, but that is not your concern.

So I’ve decided to try the 3 most popular IDEs for Java currently available.

Context

During the last 2 months, I’ve been changing between projects I work in more quickly than I used to be accustomed, and that has allowed me to choose the IDE for working.

The title says it’s been for fun, but I’m using them in professional environment. Besides, the goal is not to decide which is better (it’s hard to be that rude with the others) or worst (well, it seems it’s an easy decision, because the worst Java IDE is JDeveloper, doubtless). What I want to find out is which of them is more productive for me.

Let’s see the first impressions.

Eclipse

Actually I don’t use Eclipse, but Spring Tool Suite, “an Eclipse-based development environment that is customized for developing Spring applications”. You know, Eclipse on Spring steroids.

I’ve been using Eclipse as my main IDE for years, even before it was released (Visual Age for Java was the IDE that I used in my first professional project). Thus, I’m very used to writing programs with Eclipse.

The version that I’m currently using is 3.6.0, that is based on Eclipse 4.4 (Luna)

Let’s summarize my first impressions:

Pros:

• It comes in a compressed file, no native installer. That allows me to place the program in the same location on every OS (in my case, an Application folder within my user home)
• The shortcut keys
• Typing assistant. I love to type “main” and have an entire main method written, or “sysout” and obtain a complete log sentence.
• The view and the perspectives is the most interesting way of organizing the IDE

Cons:

• It crashes from time to time. Annoying.
• It seems to consume a lot of memory
• The TC server that it includes is based on Tomcat 7, so it doesn’t recognize Servlet 3.1 projects
• A custom way of building code. No native support for tools like maven, gradle or ant. There are appropriate plugins, but having to update Eclipse projects to reflect maven changes is somewhat annoying
• On Ubuntu and on OS X, the icon in the Docker runs the previously installed 3.5.1 version. Maybe is the OS fault, due to the lack of installation.

NetBeans

I’m talking about NetBeans 8.0.

It’s hard for me to evaluate Eclipse, because it is the IDE that I use on a daily basis since a long years ago.

On the other hand, it’s very easy to describe the first impressions with NetBeans.

The installer works fine and the application runs smoothly, but I miss the way Eclipse are organized and its shortcuts (delete line: Ctrl/Cmd+D vs Ctrl/Cmd+E)

However NetBeans is likely the best among the three IDEs (not so hard to decide, apparently)

I got used to NetBeans in a few hours. Yes, hours. NetBeans has the more natural appearance for programming.

Actually, it includes all the options that Eclipse has, but sometimes with different key combination.

Besides, NetBeans is pure Java using external tools for compiling (maven, for instance)

Furthermore, it comes with Tomcat and GlassFish and a several of useful tools easy to configure with pluggins.

Does it have flaws? Of course, sometimes it get locked in a long wait operation and you can do nothing but wait.

Another tricky issue is that it uses an external tool to compile, maven for instance, so the program doesn’t exist until the project is built and neither you have the dependencies until maven downloads them. But worst than that is, if you compile successfully, but later on you introduce a compilation error, when you execute again, it runs the previously compiled version, because the current can’t be compiled due to the error.

Eclipse shows you an alert if there are compilation errors in your project before executing it.

IntelliJ IDEA

I’m sure that this is the best IDE ever, it seems so, but it makes me a little bit unproductive.

Eclipse is some kind of a de facto standard, and NetBeans is an easy to use program. But thanks to IntelliJ IDEA 13.1, the JetBrains Web Help has been the most visited page this week.

Besides, I don’t know what is doing the IDE. I don’t know whether the files are saved automatically, I don’t know when compilation is performed, and so on.

The worst thing in my humble opinion is that it doesn’t have any application server in the community edition. Please, at least Tomcat. But it has not.

Regarding shortcuts, it uses Ctrl/Cmd+Y for deleting lines, seriously?, the universal shortcut for “redo”? Yepes. And the rest of shortcut keys are really hard to do (Ctrl+F4 for closing a file)

I will try it later again, but currently it’s not my preferred choice.

However, once you discover how to use it, it’s the one that runs better. It seems not consume a lot of memory and it never blocks nor has crashes.

On Ubuntu, I have to start it via command line.

Summary

Of course the three programs are very good IDEs, but one by one:

• I feel a little bit lost with IntelliJ IDEA. It’s great but it’s hard to find the standard options.
• I love Eclipse and its shortcut keys, and I don’t care its flaws because I’m used to them after all these years. It will be my preferred IDE for a while, but NetBeans is now here.
• I didn’t like NetBeans, but the version 8 is a great program. It doesn’t matter if you haven’t use it before, is a very natural program. It makes Java development the way it has to be. Now I like it a lot.

Some final words: they are all very good and easy to use for Java debugging. And I’m glad now I can use any of them if I had to.

An alternative for short projects is a text editor (Sublime Text, for instance) and a good build tool (maven or gradle, as you wish)

I have to try Web Based IDEs, as Alexander J Turner suggested.

That’s all for now.