We also saw a sample application and two ways of running it: by invoking a JobLauncher bean or by using CommandLineJobRunner from the command line.

In this blog entry, we’ll see two additional ways to run a Spring Batch job:

1. Using JobOperator, in order to have control of the batch process, from start a job to monitoring tasks such as stopping, restarting, or summarizing a Job. We’ll only pay attention to the start operation, but once a JobOperator is configured, you can use for the remaining monitoring tasks.
2. Using Spring Boot, the new convention-over-configuration centric framework from the Spring team, that allows you to create with a few lines of code applications that “just run”, because Spring Boot provides a lot of features based on what you have in your classpath.

As usual, all the source code is available at GitHub.

Running the sample: JobOperator

JobOperator is an interface that provides operations for inspecting and controlling jobs, mainly for a command-line client or a remote launcher like a JMX console.

The implementation that Spring Batch provides, SimpleJobOperator, uses JobLauncher, JobRepository, JobExplorer, and JobRegistry for performing its operations. They are created by the @EnableBatchProcessing annotation, so we can create an additional batch configuration file with these dependencies autowired and later import it in the batch configuration file (not without issues, due to the way Spring loads the application context, we’ll see this shortly):

@Configuration
public class AdditionalBatchConfiguration {

    @Autowired
    JobRepository jobRepository;
    @Autowired
    JobRegistry jobRegistry;
    @Autowired
    JobLauncher jobLauncher;
    @Autowired
    JobExplorer jobExplorer;

    @Bean
    public JobOperator jobOperator() {
        SimpleJobOperator jobOperator = new SimpleJobOperator();
        jobOperator.setJobExplorer(jobExplorer);
        jobOperator.setJobLauncher(jobLauncher);
        jobOperator.setJobRegistry(jobRegistry);
        jobOperator.setJobRepository(jobRepository);
        return jobOperator;
    }

}

And the @Import:

@Configuration
@EnableBatchProcessing
@Import(AdditionalBatchConfiguration.class)
public class BatchConfiguration {

	// Omitted

}

Now it seems easy to run the job with a main class:

@Component
public class MainJobOperator {

    @Autowired
    JobOperator jobOperator;

    @Autowired
    Job importUserJob;

    public static void main(String... args) throws JobParametersInvalidException, JobInstanceAlreadyExistsException, NoSuchJobException, DuplicateJobException, NoSuchJobExecutionException {

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ApplicationConfiguration.class);

        MainJobOperator main = context.getBean(MainJobOperator.class);
        long executionId = main.jobOperator.start(main.importUserJob.getName(), null);

        MainHelper.reportResults(main.jobOperator, executionId);
        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

        context.close();

        System.out.printf("\nFIN %s", main.getClass().getName());

    }
}

But there’s a little problem… it doesn’t work:

Exception in thread "main" org.springframework.batch.core.launch.NoSuchJobException: No job configuration with the name [importUserJob] was registered
	at org.springframework.batch.core.configuration.support.MapJobRegistry.getJob(MapJobRegistry.java:66)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy14.getJob(Unknown Source)
	at org.springframework.batch.core.launch.support.SimpleJobOperator.start(SimpleJobOperator.java:310)
	at com.malsolo.springframework.batch.sample.MainJobOperator.main(MainJobOperator.java:15)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

Process finished with exit code 1

The problem here, No job configuration with the name [importUserJob] was registered, is due to the way that JobOperator.start(String jobName, String parameters) works.

The main difference with JobLauncher.run(Job job, JobParameters jobParameters) is that the former has String as parameters while the latter uses objects directly.

So JobOperator, actually SimpleJobOperator, has to obtain a Job with the provided name. In order to do so, it uses the JobRegistry.getJob(String name) method. The available Spring Batch implementation is MapJobRegistry that uses a ConcurrentMap to store using the job name as the key, a JobFactory to create the Job when requested.

The problem is that this map has not been populated.

The first solution is easy: the JobRegistry allows you to register at runtime a JobFactory to later obtain the Job as explained above. So, we only need to create this JobFactory…

@Configuration
public class AdditionalBatchConfiguration {
//    Rest omitted
    @Autowired
    Job importUserJob;

    @Bean
    public JobFactory jobFactory() {
        return new ReferenceJobFactory(importUserJob);
    }
}

…and register it in the main method:

@Component
public class MainJobOperator {

    @Autowired
    JobFactory jobFactory;
    @Autowired
    JobRegistry jobRegistry;
    @Autowired
    JobOperator jobOperator;

    @Autowired
    Job importUserJob;

    public static void main(String... args) throws JobParametersInvalidException, JobInstanceAlreadyExistsException, NoSuchJobException, DuplicateJobException, NoSuchJobExecutionException {

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ApplicationConfiguration.class);

        MainJobOperator main = context.getBean(MainJobOperator.class);
        main.jobRegistry.register(main.jobFactory);
        long executionId = main.jobOperator.start(main.importUserJob.getName(), null);

        MainHelper.reportResults(main.jobOperator, executionId);
        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

        context.close();

        System.out.printf("\nFIN %s", main.getClass().getName());

    }
}

And now it works:

***********************************************************
JobExecution: id=0, version=2, startTime=2014-09-04 13:03:37.964, endTime=2014-09-04 13:03:38.141, lastUpdated=2014-09-04 13:03:38.141, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=0, version=0, Job=[importUserJob]], jobParameters=[{}]
* Steps executed:
StepExecution: id=0, version=3, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
***********************************************************
***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************

But I don’t like this approach, it’s too manual.

I’d rather to populate the JobRegistry automatically, and Spring Batch provides two mechanisms for doing so, a bean post-processor, JobRegistryBeanPostProcessor, and a component that loads and unloads Jobs by creating child context and registering jobs from those contexts as they are created, AutomaticJobRegistrar.

We’ll see the post-processor approach, because it’s very easy. Just declare the bean in the Batch configuration and run the original main class.

@Configuration
@EnableBatchProcessing
@Import(AdditionalBatchConfiguration.class)
public class BatchConfiguration {

	// Omitted

    @Bean
    public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
        JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
        jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
        return jobRegistryBeanPostProcessor;
    }

}

The bean post processor has to be declared in this configuration file for registering the job when it’s created (this is the issue that I mentioned before, if you declare the post processor in another java file configuration, for instance in the AdditionalBatchConfiguration it will never receive the job bean). It uses the same JobRegistry that uses the JobOperator to launch the Job. Actually the only that exists, but it’s good to know this.

It also works:

***********************************************************
JobExecution: id=0, version=2, startTime=2014-09-04 13:20:07.343, endTime=2014-09-04 13:20:07.522, lastUpdated=2014-09-04 13:20:07.522, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=0, version=0, Job=[importUserJob]], jobParameters=[{}]
* Steps executed:
StepExecution: id=0, version=3, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
***********************************************************
***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************

Running the sample: Spring Boot

We’d like to try how quickly and easy is Spring Boot for launching Spring Batch applications once we already have a functional configuration.

And it seems to be a piece of cake (the problem here is to know what is happening under the hood)

package com.malsolo.springframework.batch.sample;
@ComponentScan(excludeFilters = {@ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, value = ApplicationConfiguration.class)})
@EnableAutoConfiguration
public class MainBoot {

    public static void main(String... args) {

        ApplicationContext context = SpringApplication.run(MainBoot.class);

        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

    }
}

Actually, Spring Boot is out of the bounds of this topic, it deserves its own entry (even, an entire book) but we can summarize the important code here:

• Line 3: @EnableAutoConfiguration, with this annotation you want Spring Boot to instantiate the beans that you’re going to need based on the libraries on your classpath.
• Line 8: the run method, to bootstrap the application by passing the class itself (in our case, MainBoot) that serves as the primary Spring component.

It’s enough to run the Batch application:

.   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.1.4.RELEASE)

2014-09-04 16:58:55.109  INFO 15646 --- [           main] c.m.s.batch.sample.MainBoot              : Starting MainBoot on jbeneito-Latitude-3540 with PID 15646 (/home/jbeneito/Documents/git/spring-batch-sample/target/classes started by jbeneito in /home/jbeneito/Documents/git/spring-batch-sample)
2014-09-04 16:58:55.237  INFO 15646 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@6366ebe0: startup date [Thu Sep 04 16:58:55 CEST 2014]; root of context hierarchy
2014-09-04 16:58:56.039  INFO 15646 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'jdbcTemplate': replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=batchConfiguration; factoryMethodName=jdbcTemplate; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/malsolo/springframework/batch/sample/BatchConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=mainForSpringInfo; factoryMethodName=jdbcTemplate; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/malsolo/springframework/batch/sample/MainForSpringInfo.class]]
2014-09-04 16:58:56.691  WARN 15646 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.stepScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean Javadoc for complete details
2014-09-04 16:58:56.724  WARN 15646 --- [           main] o.s.c.a.ConfigurationClassEnhancer       : @Bean method ScopeConfiguration.jobScope is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @Resource and @PostConstruct within the method's declaring @Configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @Bean Javadoc for complete details
2014-09-04 16:58:56.729  INFO 15646 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2014-09-04 16:58:56.975  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'batchConfiguration' of type [class com.malsolo.springframework.batch.sample.BatchConfiguration$$EnhancerBySpringCGLIB$$c3ec56ab] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.145  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [class org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$5a15b25b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.238  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'transactionAttributeSource' of type [class org.springframework.transaction.annotation.AnnotationTransactionAttributeSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.294  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'transactionInterceptor' of type [class org.springframework.transaction.interceptor.TransactionInterceptor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.301  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.config.internalTransactionAdvisor' of type [class org.springframework.transaction.interceptor.BeanFactoryTransactionAttributeSourceAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.374  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceConfiguration' of type [class com.malsolo.springframework.batch.sample.DataSourceConfiguration$$EnhancerBySpringCGLIB$$18a97d02] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:57.455  INFO 15646 --- [           main] o.s.j.d.e.EmbeddedDatabaseFactory        : Creating embedded database 'testdb'
2014-09-04 16:58:58.156  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [schema-all.sql]
2014-09-04 16:58:58.178  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [schema-all.sql] in 20 ms.
2014-09-04 16:58:58.178  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2014-09-04 16:58:58.189  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 11 ms.
2014-09-04 16:58:58.198  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSource' of type [class org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseFactory$EmbeddedDataSourceProxy] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.206  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration$DataSourceInitializerConfiguration' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration$DataSourceInitializerConfiguration$$EnhancerBySpringCGLIB$$c0608242] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.257  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.datasource.CONFIGURATION_PROPERTIES' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.260  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from URL [file:/home/jbeneito/Documents/git/spring-batch-sample/target/classes/schema-all.sql]
2014-09-04 16:58:58.262  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from URL [file:/home/jbeneito/Documents/git/spring-batch-sample/target/classes/schema-all.sql] in 2 ms.
2014-09-04 16:58:58.263  WARN 15646 --- [           main] o.s.b.a.jdbc.DataSourceInitializer       : Could not send event to complete DataSource initialization (ApplicationEventMulticaster not initialized - call 'refresh' before multicasting events via the context: org.springframework.context.annotation.AnnotationConfigApplicationContext@6366ebe0: startup date [Thu Sep 04 16:58:55 CEST 2014]; root of context hierarchy)
2014-09-04 16:58:58.263  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSourceInitializer' of type [class org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.277  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration' of type [class org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$$EnhancerBySpringCGLIB$$85a27e41] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.320  INFO 15646 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jobRegistry' of type [class com.sun.proxy.$Proxy25] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2014-09-04 16:58:58.795  INFO 15646 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: HSQL
2014-09-04 16:58:59.056  INFO 15646 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2014-09-04 16:58:59.361  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2014-09-04 16:58:59.381  INFO 15646 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 20 ms.
2014-09-04 16:58:59.890  INFO 15646 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2014-09-04 16:58:59.926  INFO 15646 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: []
2014-09-04 16:59:00.023  INFO 15646 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=importUserJob]] launched with the following parameters: [{run.id=1}]
2014-09-04 16:59:00.060  INFO 15646 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [step1]
Converting (firstName: Jill, lastName: Doe) into (firstName: JILL, lastName: DOE)
Converting (firstName: Joe, lastName: Doe) into (firstName: JOE, lastName: DOE)
Converting (firstName: Justin, lastName: Doe) into (firstName: JUSTIN, lastName: DOE)
Converting (firstName: Jane, lastName: Doe) into (firstName: JANE, lastName: DOE)
Converting (firstName: John, lastName: Doe) into (firstName: JOHN, lastName: DOE)
2014-09-04 16:59:00.162  INFO 15646 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=importUserJob]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED]
2014-09-04 16:59:00.164  INFO 15646 --- [           main] c.m.s.batch.sample.MainBoot              : Started MainBoot in 5.963 seconds (JVM running for 8.019)
***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************
2014-09-04 16:59:00.258  INFO 15646 --- [       Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@6366ebe0: startup date [Thu Sep 04 16:58:55 CEST 2014]; root of context hierarchy
2014-09-04 16:59:00.262  INFO 15646 --- [       Thread-1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown

Process finished with exit code 0

As a side note, this class already scans for compontents, so we don’t need to make an additional scan for components, so we exclude the ApplicationConfiguration class with a filter.

Finally, we use the MainHelper class to show a summary of the results.

That’s all for now, because one more time this entry is growing really fast. Thus, we’ll see in the next post the last topic of Spring Batch that I want to talk about: JSR 352.

Resources

• Webinar: Spring Batch 3.0.0 by Michael Minella. Published on Jun 18, 2014.
• JSR-352, Spring Batch and You by Michael Minella. Published on Feb 3, 2014.
• Integrating Spring Batch and Spring Integration by Gunnar Hillert, Michael Minella. Published on Jul 9, 2014.
• Pro Spring Batch (Expert’s Voice in Spring) by Michael Minella. Published on July 12, 2011 by Apress.
• Spring Batch in Action by Arnaud Cogoluegnes, Thierry Templier, Gary Gregory, Olivier Bazoud. Published on October 10, 2011 by Manning Publications.
• Spring.io GETTING STARTED GUIDE: Creating a Batch Service.
• Spring Batch – Reference Documentation
• Spring Batch – API specification

Michael T. Minella, project lead of Spring Batch and also a member of the JSR 352 (Batch Applications for the Java Platform) expert group, wrote in his book Pro Spring Batch the next definition “Batch processing […] is defined as the processing of data without interaction or interruption. Once started, a batch process runs to some form of completion without any intervention“.

Typically Batch Jobs are long-running, non-interactive and process large volumes of data, more than fits in memory or a single transaction. Thus they usually run outside office hours and include logic for handling errors and restarting if necessary.

Spring Batch provides, among others, the next features:

• Transaction management, to allow you to focus on business processing.
• Chunk based processing, to process a large value of data by dividing it in small pieces.
• Start/Stop/Restart/Skip/Retry capabilities, to handle non-interactive management of the process.
• Web based administration interface (Spring Batch Admin), it provides an API for administering tasks.
• Based on Spring framework, so it includes all the configuration options, including Dependency Injection.
• Compliance with JSR 352: Batch Applications for the Java Platform.

Spring Batch concepts

Batch Stereotypes

• Job: an entity that encapsulates an entire batch process. It is composed of one or more ordered Steps and it has some properties such as restartability.
• Step: a domain object that encapsulates an independent, sequential phase of a batch job.
• Item: the individual piece of data that it’s been processed.
• Chunk: the processing style used by Spring Batch: read and process the item and then aggregate until reach a number of items, called “chunk” that will be finally written.


Chunk-Oriented Processing

• JobLauncher: the entry point to launch Spring Batch jobs with a given set of JobParameters.
• JobRepository: maintains all metadata related to job executions and provides CRUD operations for JobLauncher, Job, and Step implementations.

Running a Job

The JobLauncher interface has a basic implementation SimpleJobLauncher whose only required dependency is a JobRepository, in order to obtain an execution, so that you can use it for executing the Job.

JobLauncher

You can also launch a Job asynchronously by configuring a TaskExecutor. You can also this configuration for running Jobs from within a Web Container.

Job launcher sequence async

A JobLauncher uses the JobRepository to create new JobExecution objects and run them.

Job Repository

Running Jobs: concepts

The main concepts related with Job execution are

Job hierarchy with steps

• JobInstance: a logical run of a Job.
• JobParameters: a set of parameters used to start a batch job. It categorizes each JobInstance.
• JobExecution: physical runs of Jobs, in order to know what happens with the execution.
• StepExecution: a single attempt to execute a Step, that is created each time a Step is run and it also provides information regarding the result of the processing.
• ExecutionContext: a collection of key/value pairs that are persisted and controlled by the framework in order to allow developers a place to store persistent state that is scoped to a StepExecution or JobExecution.

Sample application

Now we are going to see a simple sample application that reads a POJO that represents a Person from a file containing People data and after processing each of them, that is just uppercase its attributes, saving them in a database.

All the code is available at GitHub.

Let’s begin with the basic domain class: Person, just a POJO.

package com.malsolo.springframework.batch.sample;

public class Person {
    private String lastName;
    private String firstName;
    //...
}

Then, let’s see the simple processor, PersonItemProcessor. It implements an ItemProcessor, with a Person both as Input and Output.

It provides a method to be overwritten, process, that allows you to write the custom transformation.

package com.malsolo.springframework.batch.sample;

import org.springframework.batch.item.ItemProcessor;

public class PersonItemProcessor implements ItemProcessor<Person, Person> {

    @Override
    public Person process(final Person person) throws Exception {
        final String firstName = person.getFirstName().toUpperCase();
        final String lastName = person.getLastName().toUpperCase();

        final Person transformedPerson = new Person(firstName, lastName);

        System.out.println("Converting (" + person + ") into (" + transformedPerson + ")");

        return transformedPerson;
    }

}

Once done this, we can proceed to configure the Spring Batch Application, for doing so, we’ll use Java Annotations in a BatchConfiguration file:

// Imports and package omitted
@Configuration
@EnableBatchProcessing
@Import(AdditionalBatchConfiguration.class)
public class BatchConfiguration {

	// Input, processor, and output definition
	
	@Bean
    public ItemReader<Person> reader() {
		FlatFileItemReader<Person> reader = new FlatFileItemReader<Person>();
		reader.setResource(new ClassPathResource("sample-data.csv"));
		reader.setLineMapper(new DefaultLineMapper<Person>() {{
			setLineTokenizer(new DelimitedLineTokenizer() {{
				setNames(new String[] {"firstName", "lastName"});
			}});
			setFieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
				setTargetType(Person.class);
			}});
			
		}});
		return reader;
	}
	
	@Bean
    public ItemProcessor<Person, Person> processor() {
        return new PersonItemProcessor();
    }
	
	@Bean
    public ItemWriter<Person> writer(DataSource dataSource) {
		JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<Person>();
		writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
		writer.setSql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)");
		writer.setDataSource(dataSource);
		return writer;
	}
	
	//  Actual job configuration
	
	@Bean
    public Job importUserJob(JobBuilderFactory jobs, Step s1) {
		return jobs.get("importUserJob")
				.incrementer(new RunIdIncrementer())
				.flow(s1)
				.end()
				.build();
	}
	
	@Bean
    public Step step1(StepBuilderFactory stepBuilderFactory, ItemReader<Person> reader,
            ItemWriter<Person> writer, ItemProcessor<Person, Person> processor) {
		return stepBuilderFactory.get("step1")
				.<Person, Person> chunk(10)
				.reader(reader)
				.processor(processor)
				.writer(writer)
				.build();
	}
	
	@Bean
    public JdbcTemplate jdbcTemplate(DataSource dataSource) {
        return new JdbcTemplate(dataSource);
    }
	
}

Highlights for this class are:

• Line 2: @Configuration, this class will be processed by the Spring container to generate bean definitions.
• Line 3: @EnableBatchProcessing, provides a base configuration for building batch jobs by creating the next beans beans available to be autowired:
  • JobRepository – bean name “jobRepository”
  • JobLauncher – bean name “jobLauncher”
  • JobRegistry – bean name “jobRegistry”
  • PlatformTransactionManager – bean name “transactionManager”
  • JobBuilderFactory – bean name “jobBuilders”
  • StepBuilderFactory – bean name “stepBuilders”

  We’ll see shortly how it works.

• Line 10: the reader bean, an instance of a FlatFileItemReader, that implements the ItemReader interface to read each Person from the file containing people. Spring Batch provides several implementations for this interface, being this implementation that read lines from one Resource one of them.You know, no need of custom code.
• Line 26: the processor bean, an instance of the previously defined PersonItemProcessor. See above.
• Line 31: the writer bean, an instance of a JdbcBatchItemWriter, that implements the ItemWriter interface to write the People already processed to the database. It’s also an implementation provided by Spring Batch, so no need of custom code again. In this case, you only have to provide an SQL, and a callback for the parameters. Since we are using named parameters, we’ve chosen a BeanPropertyItemSqlParameterSourceProvider. This bean also needs a DataSource, so we provided it by passing one as a method parameter in order to Spring inject the instance that it has registered.
• Line 42: a Job bean, that it’s built using the JobBuilderFactory that is autowired by passing it as method parameter for this @Bean method. When you call its get method, Spring Batch will create a job builder and will initialize its job repository, the JobBuilder is the convenience class for building jobs of various kinds as you can see in the code above. We also use a Step that is configured as the next Spring bean.
• Line 51: a Step bean, that it’s built using the StepBuilderFactory that is autowired by passing it as method parameter for this @Bean method, as well as the other dependencies: the reader, the processor and the writer previously defined. When calling the get method from the StepBuilderFactory, Spring Batch will create a step builder and will initialize its job repository and transaction manager, the StepBuilder is an entry point for building all kinds of steps as you can see in the code above.

This configuration is almost everything needed to configure a Batch process as defined in the concepts above.

Actually, only one configuration class needs to have the @EnableBatchProcessing annotation in order to have the base configuration for building batch jobs. Then you can define the job with their steps and the readers/processors/writers that they need.

But an additional data source is needed to be used by the JobRepository. For this sample we’ll use an in-memory one:

@Configuration
public class DataSourceConfiguration {

    @Bean
    public DataSource dataSource() {
        EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
        return builder
                .setType(HSQL)
                .addScript("schema-all.sql")
                .addScript("org/springframework/batch/core/schema-hsqldb.sql")
                .build();
    }

}

In this case we’ll use the same in-memory database, HSQL, with the schema for the application (line 9) and the schema for the job repository (line 10). The former is available as a resource of the application, the file called schema-all.sql, and the latter in the spring-batch-core jar (spring-batch-core-3.0.1.RELEASE.jar at the time of this writing)

Alternate Configuration

The official documentation shows an slightly different configuration by using the @Autowired annotation for the beans that @EnableBatchProcessing will create. Use the one that you like most. In this case they also imports the data base configuration.

@Configuration
@EnableBatchProcessing
@Import(DataSourceConfiguration.class)
public class AppConfig {

    @Autowired
    private JobBuilderFactory jobs;

    @Autowired
    private StepBuilderFactory steps;

    // Input, processor, and output definition omitted

    @Bean
    public Job importUserJob() {
        return jobs.get("importUserJob").incrementer(new RunIdIncrementer()).flow(step1()).end().build();
    }

    @Bean
    protected Step step1(ItemReader<Person> reader, ItemProcessor<Person, Person> processor, ItemWriter<Person> writer) {
        return steps.get("step1")
            .<Person, Person> chunk(10)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .build();
    }

}

We chose another approach: we load it when configuring the application in the main method as you’ll see shortly. besides, we imported an additional batch configuration (see line 28 at BatchConfiguration.java) to provide an alternate way to launch the application.

Enable Batch Processing: how it works

As we said before, we will go a little deeper in how the annotation @EnableBatchProcessing works.

To remind its goal, this annotation provides a base configuration for building batch jobs by creating a list of beans available to be autowired. An extract of the source code gives us a lot of information:

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(BatchConfigurationSelector.class)
public @interface EnableBatchProcessing {

	/**
	 * Indicate whether the configuration is going to be modularized into multiple application contexts. If true then
	 * you should not create any @Bean Job definitions in this context, but rather supply them in separate (child)
	 * contexts through an {@link ApplicationContextFactory}.
	 */
	boolean modular() default false;

}

As you can see at line 4, this annotation imports an implementation of an ImportSelector, one of the options to import beans in a configuration class, in particular, to selective import beans according to certain criteria.

This particular implementation, BatchConfigurationSelector, instantiates the expected beans for providing common structure for enabling and using Spring Batch based in the EnableBatchProcessing’s attribute modular.

There are two implementations depending on whether you want the configuration to be modularized into multiple application contexts so that they don’t interfere with each other with the naming and the uniqueness of beans (for instance, beans named reader) or not. They are ModularBatchConfiguration and SimpleBatchConfiguration respectively. Mainly they both do the same, but the former using an AutomaticJobRegistrar which is responsible for creating separate ApplicationContexts for register isolated jobs that are later accesible via the JobRegistry, and the latter just creates the main components as lazy proxies that only initialize when a method is called (in order to prevent configuration cycles)

The key concept here is that both extends AbstractBatchConfiguration that uses the core interface for this configuration: BatchConfigurer.

The default implementation, DefaultBatchConfigurer, provides the beans mentioned above (jobRepository, jobLauncher, jobRegistry, transactionManager, jobBuilders and stepBuilders), for doing so it doesn’t require a dataSource, it’s Autowired with required to false, so it will use a Map based JobRepository if its dataSource is null, but you have take care if you have a dataSource eligible for autowiring that doesn’t contain the expected database schema for the job repository: the batch process will fail in this case.

Spring Boot provides another implementation, BasicBatchConfigurer, but this is out of the scope of this entry.

With all this information, we already have a Spring Batch application configured, and we more or less know how this configuration is achieved using Java.

Now it’s time to run the application.

Running the sample: JobLauncher

We have all we need to launch a batch job, the Job to be launched and a JobLauncher, so wait no more and execute this main class: MainJobLauncher.

@Component
public class MainJobLauncher {

    @Autowired
    JobLauncher jobLauncher;

    @Autowired
    Job importUserJob;

    public static void main(String... args) throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException {

        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(ApplicationConfiguration.class);

        MainJobLauncher main = context.getBean(MainJobLauncher.class);

        JobExecution jobExecution = main.jobLauncher.run(main.importUserJob, new JobParameters());

        MainHelper.reportResults(jobExecution);
        MainHelper.reportPeople(context.getBean(JdbcTemplate.class));

        context.close();

    }

}

First things first. This is the way I like to write main classes. Some people from Spring are used to writing main classes annotated with @Configuration, but I’d rather to annotate them as @Components in order to separate the actual application and its configuration from the classes that test the functionality.

As Spring component (line 1), it only needs to have the dependencies @Autowired.

That’s the reason for the ApplicationConfiguration class. It’s a @Configuration class that also performs a @ComponentScan from its own package, that will find the very MainJobLauncher and the remain @Configuration classes, because they are also @Components: BatchConfiguration and DataSourceConfiguration.

As a main class, it creates the Spring Application Context (line 12), it gets the component as a Spring bean (line 14) and then it uses its methods (or attributes in this example. Line 16)

Let’s back to the Batch application: the line 16 is the call to the JobLauncher that will run the Spring Batch process.

The remaining lines are intended to show the results, both from the job execution and the results in the database.

It will be something like this:

***********************************************************
importUserJob finished with a status of  (COMPLETED).
* Steps executed:
	step1 : exitCode=COMPLETED;exitDescription=
StepExecution: id=0, version=3, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=5, filterCount=0, writeCount=5 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0
***********************************************************

***********************************************************
* People found:

* Found firstName: JILL, lastName: DOE in the database

* Found firstName: JOE, lastName: DOE in the database

* Found firstName: JUSTIN, lastName: DOE in the database

* Found firstName: JANE, lastName: DOE in the database

* Found firstName: JOHN, lastName: DOE in the database
***********************************************************

Running the sample: CommandLineJobRunner

CommandLineJobRunner is a main class provided by Spring Batch as the primary entry point to launch a Spring Batch Job.

It requires at least two arguments: JobConfigurationXmlPath/JobConfigurationClassName and jobName. With the first, it will create an ApplicationContext by loading a Java Configuration from a class with the same name or by loading an XML Configuration file with the same name.

It has a JobLauncher attribute that is autowired with the application context via its AutowireCapableBeanFactory exposed, that is used to autowire the bean properties by type.

It accepts some options (“-restart”, “-next”, “-stop”, “-abandon”) as well as parameters for the JobLauncher that are converted with the DefaultJobParametersConverter as JobParametersConverter that expects a ‘name=value’ format.

You can declare this main class in the manifest file, directly or using some maven plugin as maven-jar-plugin, maven-shade-plugin or even exec-maven-plugin.

That is, you can invoke from your command line something like this:

$ java CommandLineJobRunner job.xml jobName parameter=value

Well, the sample code is a maven project that you can install (it’s enough if you package the application) and it allows to manage the dependencies (the mvn dependency:copy-dependencies command copies all the dependencies in the target/dependency directory)

To simplify, I’ll also copy the generated jar to the same directory of the dependencies in order to invoke the java command more easily:

~/Documents/git/spring-batch-sample$mvn clean install
[INFO] Scanning for projects...
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
...
~/Documents/git/spring-batch-sample$ mvn dependency:copy-dependencies
[INFO] Scanning for projects...
...
[INFO] Copying spring-batch-core-3.0.1.RELEASE.jar to ~/Documents/git/spring-batch-sample/target/dependency/spring-batch-core-3.0.1.RELEASE.jar
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
...
~/Documents/git/spring-batch-sample$ cp target/spring-batch-sample-0.0.1-SNAPSHOT.jar ./target/dependency/
~/Documents/git/spring-batch-sample$ java -classpath "./target/dependency/*" org.springframework.batch.core.launch.support.CommandLineJobRunner com.malsolo.springframework.batch.sample.ApplicationConfiguration importUserJob
...
12:32:17.039 [main] INFO  o.s.b.c.l.support.SimpleJobLauncher 
- Job: [FlowJob: [name=importUserJob]] 
completed with the following parameters: [{}] 
and the following status: [COMPLETED]
...

That’s all for now.

Since this entry is becoming very large, I’ll explain other ways to run Spring Batch Jobs in a next post.