Spring Boot apps with autoconfiguration for Spring Batch currently do not initialize the data source with BatchDataSourceInitializer if spring.main.lazy-initialization is set to true independent of what the property spring.batch.jdbc.initialize-schema is set to.
The issue can be worked around by users with a LazyInitializationExcludeFilter but a consistent behavior for spring.batch.jdbc.initialize-schema independent of lazy or eager initialization would be nice.
Comment From: wilkinsona
Thanks for the proposal, but this change doesn't feel quite right to me.
I think it should be possible for the initialiser to be lazy and for the schema to then be initialized prior to first use of the DataSource. If the initializer needs to be eagerly initialized for a reason that I have overlooked then I think it should be done with a LazyInitializationExcludeFilter that Spring Boot auto-configures. We should also add a test that replicates the scenario that requires eager initialization and verify that it now works.
Comment From: hpoettker
@wilkinsona Thanks for the quick and insightful feedback! I'd be happy to re-work the PR according to your suggestion to instantiate the DataSourceInitializer if and only if the DataSource gets initialized.
Would it be possible to auto-configure a BeanPostProcessor that applies the DataSourceInitializer to the DataSource in postProcessAfterInitialization? If yes, how could I replicate the logic to pick the data source that is annotated with @BatchDataSource in the case of more than one DataSource? Or would you suggest a different approach?
Regarding the suggested auto-configured LazyInitializationExcludeFilter: If we auto-configured it on the same conditions as the BatchDataSourceInitializer itself, I think this would be pretty much equivalent to annotating the initializer directly as eager. Would it make sense to auto-configure the LazyInitializationExcludeFilter on some stricter conditions like e.g. the property spring.batch.jdbc.initialize-schema being explicitly set?
Comment From: wilkinsona
Sorry, I didn't explain myself clearly earlier.
I think it should be possible for the initialiser to be lazy and for the schema to then be initialized prior to first use of the DataSource
When I said this above, I meant that it should already work like this. I've just tried with our Batch smoke test with lazy init enabled and it doesn't work. In Boot 2.4, it fails with the following:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot ::
2021-07-08 18:13:30.450 INFO 69778 --- [ main] smoketest.batch.SampleBatchApplication : Starting SampleBatchApplication using Java 1.8.0_252 on wilkinsona-a01.vmware.com with PID 69778 (/Users/awilkinson/dev/spring-projects/spring-boot/2.4.x/spring-boot-tests/spring-boot-smoke-tests/spring-boot-smoke-test-batch/bin/main started by awilkinson in /Users/awilkinson/dev/spring-projects/spring-boot/2.4.x/spring-boot-tests/spring-boot-smoke-tests/spring-boot-smoke-test-batch)
2021-07-08 18:13:30.454 INFO 69778 --- [ main] smoketest.batch.SampleBatchApplication : No active profile set, falling back to default profiles: default
2021-07-08 18:13:31.540 INFO 69778 --- [ main] smoketest.batch.SampleBatchApplication : Started SampleBatchApplication in 1.792 seconds (JVM running for 2.307)
2021-07-08 18:13:31.703 INFO 69778 --- [ main] o.s.b.a.b.JobLauncherApplicationRunner : Running default command line with: []
2021-07-08 18:13:31.774 INFO 69778 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-07-08 18:13:32.073 INFO 69778 --- [ main] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
2021-07-08 18:13:32.076 INFO 69778 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-07-08 18:13:32.086 INFO 69778 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: HSQL
2021-07-08 18:13:32.250 INFO 69778 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2021-07-08 18:13:32.361 INFO 69778 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-07-08 18:13:32.393 ERROR 69778 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:802) [main/:na]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:789) [main/:na]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:346) [main/:na]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) [main/:na]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) [main/:na]
at smoketest.batch.SampleBatchApplication.main(SampleBatchApplication.java:69) [main/:na]
Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = ? and JOB_KEY = ?]; nested exception is java.sql.SQLSyntaxErrorException: user lacks privilege or object not found: BATCH_JOB_INSTANCE in statement [SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = ? and JOB_KEY = ?]
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:93) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:815) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.getJobInstance(JdbcJobInstanceDao.java:151) ~[spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.batch.core.repository.support.SimpleJobRepository.isJobInstanceExists(SimpleJobRepository.java:93) ~[spring-batch-core-4.3.3.jar:4.3.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.8.jar:5.3.8]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.8.jar:5.3.8]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.8.jar:5.3.8]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.8.jar:5.3.8]
at com.sun.proxy.$Proxy43.isJobInstanceExists(Unknown Source) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.3.jar:4.3.3]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.8.jar:5.3.8]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.8.jar:5.3.8]
at com.sun.proxy.$Proxy43.isJobInstanceExists(Unknown Source) ~[na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.getNextJobParameters(JobLauncherApplicationRunner.java:206) ~[main/:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:198) ~[main/:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[main/:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[main/:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[main/:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[main/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:799) [main/:na]
... 5 common frames omitted
Caused by: java.sql.SQLSyntaxErrorException: user lacks privilege or object not found: BATCH_JOB_INSTANCE in statement [SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = ? and JOB_KEY = ?]
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.jdbc.JDBCPreparedStatement.<init>(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.jdbc.JDBCConnection.prepareStatement(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337) ~[HikariCP-3.4.5.jar:na]
at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java) ~[HikariCP-3.4.5.jar:na]
at org.springframework.jdbc.core.JdbcTemplate$SimplePreparedStatementCreator.createPreparedStatement(JdbcTemplate.java:1645) ~[spring-jdbc-5.3.8.jar:5.3.8]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:649) ~[spring-jdbc-5.3.8.jar:5.3.8]
... 42 common frames omitted
Caused by: org.hsqldb.HsqlException: user lacks privilege or object not found: BATCH_JOB_INSTANCE
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.error.Error.error(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.readTableName(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.readTableOrSubquery(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadTableReference(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadFromClause(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadTableExpression(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadQuerySpecification(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadSimpleTable(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadQueryPrimary(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadQueryTerm(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadQueryExpressionBody(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.XreadQueryExpression(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserDQL.compileCursorSpecification(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserCommand.compilePart(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.ParserCommand.compileStatement(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.Session.compileStatement(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.StatementManager.compile(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
at org.hsqldb.Session.execute(Unknown Source) ~[hsqldb-2.5.2.jar:2.5.2]
... 48 common frames omitted
2021-07-08 18:13:32.396 INFO 69778 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2021-07-08 18:13:32.397 INFO 69778 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
This tells me that we are missing some dependency relationships between beans. It only works at the moment in the non-lazy case as the Batch database tables aren't being accessed until after the application context has refreshed. If something tried to use those tables during refresh, it may fail as there's no guarantee that the initializer will have run.
Thanks for bringing the problem to our attention. I don't think that changing the eagerness with which the relevant beans are initialized is the right way to fix this. I've opened https://github.com/spring-projects/spring-boot/issues/27193 so that we can take a different approach.