Spring Incomplete cleanup after configuration class creation failure

I'm using Spring Boot 2.1.5.RELEASE, which i recently upgraded two, and the corresponding versions of Spring Security, Spring JPA and Hibernate. One of the main reasons was the option to use @Autowired to inject code into a JPA Attribute Converter.

Doing so resulted in a strange behaviour when running my application. All works perfectly fine (at least it seems so) in tests and when running the application via bootRun. But if I run the same application using the jar-file I get from bootJar I will get the following exception:

org.springframework.security.oauth2.provider.endpoint.TokenEndpoint.handleException:169 Handling error: IllegalStateException, This object has not been built
java.lang.IllegalStateException: This object has not been built
    at org.springframework.security.config.annotation.AbstractSecurityBuilder.getObject(AbstractSecurityBuilder.java:55) ~[spring-security-config-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter$AuthenticationManagerDelegator.authenticate(WebSecurityConfigurerAdapter.java:508) ~[spring-security-config-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.security.oauth2.provider.password.ResourceOwnerPasswordTokenGranter.getOAuth2Authentication(ResourceOwnerPasswordTokenGranter.java:71) ~[spring-security-oauth2-2.3.6.RELEASE.jar!/:?]
    ...

I opened a stackoverflow question for this, but with my own investigation going on, I think the issue is better suited to be filed here. I may be wrong though.

I also created a stripped down sample project and publish it in Github. The readme-file includes instructions on how to reproduce the problem and how to identify possible causes.

Please let me know, if I should provide more information.

Comment From: wilkinsona

Thanks for the sample. Unfortunately, I have been unable to reproduce the problem with it. I tried on macOS using OpenJDK 1.8.0_202. What OS and JDK are you using?

Comment From: wilkinsona

I've reproduced the problem by building the jar on Windows. Once built, this jar exhibits the problem when used on macOS or Windows. Comparing the contents of the two jars, the BOOT-INF/classes entries are ordered differently:

macOS

        0  07-23-2019 09:55   BOOT-INF/classes/
        0  07-23-2019 09:49   BOOT-INF/classes/com/
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/service/
     5492  07-23-2019 09:49   BOOT-INF/classes/com/company/service/ServiceApplication.class
     1908  07-23-2019 09:49   BOOT-INF/classes/com/company/service/JerseyConfig.class
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/service/util/
      610  07-23-2019 09:49   BOOT-INF/classes/com/company/service/util/OperationNotSuccessfulException.class
     1724  07-23-2019 09:49   BOOT-INF/classes/com/company/service/util/JsonUtils.class
     1169  07-23-2019 09:49   BOOT-INF/classes/com/company/service/util/MyStringUtils.class
     1065  07-23-2019 09:49   BOOT-INF/classes/com/company/service/util/HttpHeaders.class
     2347  07-23-2019 09:49   BOOT-INF/classes/com/company/service/util/ApplicationPropertyAccess.class
      625  07-23-2019 09:49   BOOT-INF/classes/com/company/service/util/JsonUtils$1.class
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/
     1294  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/ApplicationRoleGrantedAuthority.class
     3475  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/CustomTokenEnhancer.class
     1220  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/ApplicationRole.class
     3879  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/AuthManagerImpl.class
      233  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/AuthenticationFacade.class
     4340  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/CustomJwtAccessTokenConverter.class
     1316  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/UserPrincipal.class
      481  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/AuthManager.class
      866  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/AuthenticationFacadeImpl.class
      600  07-23-2019 09:49   BOOT-INF/classes/com/company/service/security/NotAuthenticatedException.class
    17390  07-23-2019 09:49   BOOT-INF/classes/com/company/service/SecurityConfiguration.class
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/service/business/
      703  07-23-2019 09:49   BOOT-INF/classes/com/company/service/business/UserManager.class
     8501  07-23-2019 09:49   BOOT-INF/classes/com/company/service/business/PasswordResetManagerImpl.class
     3586  07-23-2019 09:49   BOOT-INF/classes/com/company/service/business/UserDetailsServiceImpl.class
     1278  07-23-2019 09:49   BOOT-INF/classes/com/company/service/business/PasswordResetManager.class
     2828  07-23-2019 09:49   BOOT-INF/classes/com/company/service/business/UserManagerImpl.class
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/service/persistence/
      577  07-23-2019 09:49   BOOT-INF/classes/com/company/service/persistence/UserRepository.class
      217  07-23-2019 09:49   BOOT-INF/classes/com/company/service/persistence/package-info.class
      352  07-23-2019 09:49   BOOT-INF/classes/com/company/service/persistence/JpaMarkerRepositories.class
      745  07-23-2019 09:49   BOOT-INF/classes/com/company/service/persistence/BaseRepository.class
     1883  07-23-2019 09:49   BOOT-INF/classes/com/company/service/persistence/EncryptionConverter.class
      764  07-23-2019 09:49   BOOT-INF/classes/com/company/service/persistence/PasswordResetTokenRepository.class
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/
     1095  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/AbstractModelObject.class
     1574  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/UserDetailsDTO.class
     1243  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/TokenOperationResult.class
      649  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/SecureInformationDTO.class
     1221  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/UserStatus.class
     3286  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/PasswordResetToken.class
     4135  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/User.class
      319  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/JpaMarkerModel.class
     1832  07-23-2019 09:49   BOOT-INF/classes/com/company/service/model/SecureInformation.class
      488  07-23-2019 09:49   BOOT-INF/classes/com/company/service/Profiles.class
        0  07-23-2019 09:49   BOOT-INF/classes/com/company/service/rest/
     3803  07-23-2019 09:49   BOOT-INF/classes/com/company/service/rest/UserDetailsResource.class
      559  07-23-2019 09:49   BOOT-INF/classes/hibernate.properties
     1326  07-23-2019 09:49   BOOT-INF/classes/application-development.properties
     2154  07-23-2019 09:49   BOOT-INF/classes/log4j2-production.xml
      241  07-23-2019 09:49   BOOT-INF/classes/data.sql
     2717  07-23-2019 09:49   BOOT-INF/classes/log4j2-development.xml
       78  07-23-2019 09:49   BOOT-INF/classes/banner.txt
     2583  07-23-2019 09:49   BOOT-INF/classes/keystore.p12
     2243  07-23-2019 09:49   BOOT-INF/classes/jwt.jks
      452  07-23-2019 09:49   BOOT-INF/classes/jwtpublic.cert
     1255  07-23-2019 09:49   BOOT-INF/classes/application-test.properties
     3135  07-23-2019 09:49   BOOT-INF/classes/application.properties

Windows

        0  07-23-2019 10:07   BOOT-INF/classes/
        0  07-23-2019 10:06   BOOT-INF/classes/com/
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/service/
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/service/business/
     1278  07-23-2019 10:06   BOOT-INF/classes/com/company/service/business/PasswordResetManager.class
     8501  07-23-2019 10:06   BOOT-INF/classes/com/company/service/business/PasswordResetManagerImpl.class
     3586  07-23-2019 10:06   BOOT-INF/classes/com/company/service/business/UserDetailsServiceImpl.class
      703  07-23-2019 10:06   BOOT-INF/classes/com/company/service/business/UserManager.class
     2828  07-23-2019 10:06   BOOT-INF/classes/com/company/service/business/UserManagerImpl.class
     1908  07-23-2019 10:06   BOOT-INF/classes/com/company/service/JerseyConfig.class
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/
     1095  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/AbstractModelObject.class
      319  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/JpaMarkerModel.class
     3286  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/PasswordResetToken.class
     1832  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/SecureInformation.class
      649  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/SecureInformationDTO.class
     1243  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/TokenOperationResult.class
     4135  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/User.class
     1574  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/UserDetailsDTO.class
     1221  07-23-2019 10:06   BOOT-INF/classes/com/company/service/model/UserStatus.class
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/service/persistence/
      745  07-23-2019 10:06   BOOT-INF/classes/com/company/service/persistence/BaseRepository.class
     1883  07-23-2019 10:06   BOOT-INF/classes/com/company/service/persistence/EncryptionConverter.class
      352  07-23-2019 10:06   BOOT-INF/classes/com/company/service/persistence/JpaMarkerRepositories.class
      217  07-23-2019 10:06   BOOT-INF/classes/com/company/service/persistence/package-info.class
      764  07-23-2019 10:06   BOOT-INF/classes/com/company/service/persistence/PasswordResetTokenRepository.class
      577  07-23-2019 10:06   BOOT-INF/classes/com/company/service/persistence/UserRepository.class
      488  07-23-2019 10:06   BOOT-INF/classes/com/company/service/Profiles.class
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/service/rest/
     3803  07-23-2019 10:06   BOOT-INF/classes/com/company/service/rest/UserDetailsResource.class
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/
     1220  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/ApplicationRole.class
     1294  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/ApplicationRoleGrantedAuthority.class
      233  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/AuthenticationFacade.class
      866  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/AuthenticationFacadeImpl.class
      481  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/AuthManager.class
     3879  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/AuthManagerImpl.class
     4340  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/CustomJwtAccessTokenConverter.class
     3475  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/CustomTokenEnhancer.class
      600  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/NotAuthenticatedException.class
     1316  07-23-2019 10:06   BOOT-INF/classes/com/company/service/security/UserPrincipal.class
    17390  07-23-2019 10:06   BOOT-INF/classes/com/company/service/SecurityConfiguration.class
     5492  07-23-2019 10:06   BOOT-INF/classes/com/company/service/ServiceApplication.class
        0  07-23-2019 10:06   BOOT-INF/classes/com/company/service/util/
     2347  07-23-2019 10:06   BOOT-INF/classes/com/company/service/util/ApplicationPropertyAccess.class
     1065  07-23-2019 10:06   BOOT-INF/classes/com/company/service/util/HttpHeaders.class
      625  07-23-2019 10:06   BOOT-INF/classes/com/company/service/util/JsonUtils$1.class
     1724  07-23-2019 10:06   BOOT-INF/classes/com/company/service/util/JsonUtils.class
     1169  07-23-2019 10:06   BOOT-INF/classes/com/company/service/util/MyStringUtils.class
      610  07-23-2019 10:06   BOOT-INF/classes/com/company/service/util/OperationNotSuccessfulException.class
     1364  07-23-2019 10:06   BOOT-INF/classes/application-development.properties
     1288  07-23-2019 10:06   BOOT-INF/classes/application-test.properties
     3195  07-23-2019 10:06   BOOT-INF/classes/application.properties
       79  07-23-2019 10:06   BOOT-INF/classes/banner.txt
      245  07-23-2019 10:06   BOOT-INF/classes/data.sql
      569  07-23-2019 10:06   BOOT-INF/classes/hibernate.properties
     2243  07-23-2019 10:06   BOOT-INF/classes/jwt.jks
      462  07-23-2019 10:06   BOOT-INF/classes/jwtpublic.cert
     2583  07-23-2019 10:06   BOOT-INF/classes/keystore.p12
     2783  07-23-2019 10:06   BOOT-INF/classes/log4j2-development.xml
     2201  07-23-2019 10:06   BOOT-INF/classes/log4j2-production.xml

There's something, although I don't yet know what, in the application that is overly sensitive to the ordering of the classpath.

In the meantime, a workaround is to enable reproducible file ordering on bootJar:

bootJar {
    // …
   reproducibleFileOrder = true
}

With this setting, the problem does not occur with a jar built on Windows.

Comment From: szauner

Thank you very much for the quick reply and help, Andy! The workaround circumvents the issue in the full project, too. However, I guess, with every added class that uses injection, the problem could pop up again. I'd be glad to help in any possible way, but probably I would need more knowledge about spring code itself to be of any real help.

Comment From: wilkinsona

When the failure occurs, two instances of SecurityConfiguration are created. The first is created when trying to get an EncryptionConverter from the Hibernate SpringBeanContainer. This attempt fails due to a BeanCurrentlyInCreationException but leaves some pollution behind. This pollution includes a DefaultPasswordEncoderAuthenticationManagerBuilder that is used by the AuthenticationManager passed into AuthorizationServerEndpointsConfigurer.authenticationManager but that is never built. There are a couple of problems here:

The BeanCurrentlyInCreationException is swallowed by SpringBeanContainer, inhibiting problem diagnosis. It is logged at debug level, however.
When the creation of SecurityConfiguration fails, the resulting clean up is insufficient and results in a DefaultPasswordEncoderAuthenticationManagerBuilder that will never be built being used.

The BeanCurrentlyInCreationException is due to a dependency cycle:

entityManagerFactory depends on encryptionConverter due to its use on the SecureInformation entity
encryptionConverter depends on a bean of type PooledPBEStringEncryptor
strongEncryptor, the bean of type PooledPBEStringEncryptor, depends on securityConfiguration which defines it
SecurityConfiguration depends on UserDetailsServiceImpl
UserDetailsServiceImpl depends on UserRepository, a Spring Data JPA repository
UserRepository depends on entityManagerFactory

I don't yet understand how Spring Framework is sometimes able to break this cycle, but it is able to do so. That's quite impressive even if it is dependent on the ordering of the classpath.

@szauner rather than the workaround that I described above, you should update your configuration to break this cycle rather than relying on Spring Framework sometimes being able to do so for you. Moving the strongEncryptor bean definition to a @Configuration class that does not depend on UserDetailsService should be sufficient.

I think there are a couple of areas that warrant investigation by the Framework team:

Should SpringBeanContainer have logged the BeanCurrentlyInCreationException to make the problem easier to diagnose?
Why did the clean up following the failed attempt to create SecurityConfiguration leave things in a state where the DefaultPasswordEncoderAuthenticationManagerBuilder created during the attempt was still used by other beans.

Comment From: wilkinsona

Here are some more details on how the pollution occurs:

SpringBeanCreator requests an EncryptionConverter bean
Through the chain of dependencies describe above, this results in SecurityConfiguration bean created
SecurityConfiguration uses field injection for an AuthenticationManager bean, which it provides itself by overriding authenticationManagerBean
The AuthenticationManager bean is an AuthenticationManagerDelegator which delegates to a DefaultPasswordEncoderAuthenticationManagerBuilder

When the creation of SecurityConfiguration fails, the AuthenticationManager bean that it defined is, I believe, left in the context. This bean has a reference to a DefaultPasswordEncoderAuthenticationManagerBuilder which is then never built. This causes the "This object has not been built" failure described in the opening comment of this issue.

Comment From: szauner

Thank you so much for the work you already put into this. I moved the configuration of the strong encryptor to another file, which solves the problem. I think that's enough information for an answer on SO. If you wrote an answer, I'd be happy to accept it. It's a solution to the problem at hand, regardless of the future work that might or might not be put into this issue.

It appeared to me to be a good idea, to split the configuration into more separate files to decrease the risk of another circular dependency like this. So I created a file PersistenceConfiguration and moved all related stuff into it including the corresponding class annotations.

Comment From: jhoeller

This turns out to have been fixed through #26167, registering @Bean definitions as dependent on the containing configuration class and therefore also enforcing cleanup of any such early-produced singletons when the configuration class itself gets removed.