Mark Davies opened SPR-8471 and commented

A less significant version of this problem has already been raised under #10033 - a performance bottleneck affecting Wicket. However, the same issue causes a serious thread deadlock in our application, occasionally preventing application startup.

The basic issue seems to be that DefaultSingletonBeanRegistry takes a global synchronization lock when creating a singleton bean. Here is the code in that class:

public Object getSingleton(String beanName, ObjectFactory singletonFactory) {
     Assert.notNull(beanName, "'beanName' must not be null");
     synchronized (this.singletonObjects) {
          Object singletonObject = this.singletonObjects.get(beanName);
          if (singletonObject == null) {
          ... stuff ...
                        singletonObject = singletonFactory.getObject();
          ... stuff ...
}

The synchronized block (the same this.singletonObjects reference is used by all threads entering the method) means that Spring can only create singleton beans one at a time, regardless of their type (beanName). This clearly introduces a performance penalty if an application has a large number of singleton beans to construct, e.g. at startup.

That is not the issue affecting our code, though. We see a deadlock, caused by the following two sets of behaviour:

  1. We have Spring-managed singleton beans which perform database access in their constructor (basically loading and caching configuration sets from the database). In order to do this they obtain database connections, which are pooled, with relatively small pool sizes. If a pooled connection is not available, the calling thread blocks and waits until one becomes free. This is usually not a problem since queries are small and rapid, so pool wait times are low, and the maximum pool size is sufficient to work the databasea at full capacity anyway.

  2. We also have non-Spring code doing database access. Such code obtains a database connection from the same pool, purely for the lifetime of running a query and processing a result set, so again very quick for almost cases in our system. But sometimes, whilst processing the result set, we need to use a Spring-managed bean, which may have singleton scope.

You now have a deadlock - thread number one is trying to get a Spring-managed singleton bean, which is waiting for a JDBC connection in its constructor; thread number two is running database code which has the JDBC connection and is waiting to create a Spring-managed singleton of a completely different type. Both thread own the resource needed by the other, so will wait forever. (OK, if the database pool size is two or higher you need two or more threads in the second state, but this has happened to us in customer environments, in production).

Obviously our application code is almost certainly less than ideal in how we use Spring, but it seems to me that we ought to be able to use Spring-managed beans which do database access in the manner I've described, without encountering unpleasant deadlocks such as this. Note that the problem is actually much more general than this particular example of database connections: if your singleton beans require any kind of global monitor, which is also used outside of the context of Spring, you have a deadlock condition.

Both the originally-reported performance problem, and this deadlock, could be solved by a simple improvement to DefaultSingletonBeanRegistry. The getSingleton() method should not synchronize on a global monitor, but instead a monitor specific to the beanName you are instantiating. This is the correct level at which to lock - what you are trying to do is prevent a bean of a certain type being created more than once, but allowing two beans of different types to construct at the same time is perfectly reasonable. Of course you have to synchronize access to the underlying collections with a global lock, but the construction of the singleton bean does not need to be protected with the same global monitor. For instance I believe this kind of thing would do it:

public Object getSingleton(String beanName, ObjectFactory singletonFactory) {
     Object bean;
     Object perBeanMonitor;
     synchronized (GLOBAL_MONITOR) {
          perBeanMonitor = getPerBeanMonitor(beanName);
     }
     synchronized (perBeanMonitor) {
          synchronized (GLOBAL_MONITOR) {
               bean = getBeanFromRegistry(beanName);
          }
          if (bean==null) {
               Object bean = doConstructBean(beanName, singletonFactory);
               synchronized (GLOBAL_MONITOR) {
                        addToRegistry(beanName, bean);
               }
          }
     }
     return bean;
}

It would be extremely helpful if this approach could be implemented in this core part of the Spring framework. We rely on singleton construction to be performant and thread-safe; the fact that DefaultSingletonBeanRegistry behaves as it currently does is causing us serious production problems at client sites, and, whilst we can modify our application code to work around the issue, I do see this as a fundamental flaw in Spring - would be nice to get a quick fix please.

Affects: 2.5.6

Attachments: - test-case.zip (3.09 kB)

Issue Links: - #13428 Deadlock during application context creation ("is duplicated by") - #14470 A Java Thread deadlock has occured ("is duplicated by") - #13837 Deadlock on the beanDefinitionMap and singletonObjects ("is duplicated by") - #12374 Deadlock between DefaultListableBeanFactory and DefaultSingletonBeanRegistry, perhaps due to lazily-instantiated aspect - #14452 Non-singleton beans performance issue - #14654 Deadlock in DefaultListableBeanFactory/DefaultSingletonBeanRegistry - #14985 AbstractApplicationEventMulticaster#getApplicationListeners() should have double check for 'this.retrieverCache.get(cacheKey)' within synchronized block

Referenced from: commits https://github.com/spring-projects/spring-framework/commit/dd68fecbf2509faa189234807f30bbf364a691f1, https://github.com/spring-projects/spring-framework/commit/52124fa31b17fee7b6ffdfc92ec9fe14ad9a0561

19 votes, 26 watchers

Comment From: spring-projects-issues

ken lu commented

We have the similiar issue when we are using Spring JMS MessageListener to specify \

When Spring reads the context file, it starts 10 threads for the MessageListener. This creates a deadlock immediately when these threads get to the singleton Service or DAO.

The problem is the Service and DAO singleton should not be instanced by these thread.

The work around is to specify the MessageListener as lazy-init="true". Make sure all the singleton bean was created, then start the MessageListener.

Really hope Spring framework should sort this type of multi-threaded issue in a better way!

Comment From: spring-projects-issues

Jean-Pierre Bergamin commented

Maybe the issue #13428 is related.

Comment From: spring-projects-issues

John Cotter commented

This appears to be root cause of #12374 and OSGI-816 also. DefaultSingletonBeanRegistry could be refined so as not to hold a global lock on all singletons. As it is now, a second thread calling into DefaultSingletonBeanRegistry originating from call to DefaultListableBeanFactory can yield a deadlock. There is a wide array of use cases where this can occur.

Comment From: spring-projects-issues

John Cotter commented

Similar deadlock pattern involving DefaultSingletonBeanRegistry and DefaultListableBeanFactory observed with Spring 3.0.5.

Comment From: spring-projects-issues

Chris Beams commented

Linking as possibly related to #12374 per John's comment.

Comment From: spring-projects-issues

Piotr Bzdyl commented

I encountered a similar issue but it involved:

  • DefaultSingletonBeanRegistry.getSingleton (trying to lock ConcurrentHashMap)
  • AbstractApplicationEventMulticaster.addApplicationListener(trying to lock AbstractApplicationEventMulticaster$ListenerRetriever)

Deadlock report:

Found one Java-level deadlock:
=============================
"RMI TCP Connection(4)-10.14.5.84":
  waiting to lock monitor 0x0000000061013108 (object 0x00000000c654bdd0, a java.util.concurrent.ConcurrentHashMap),
  which is held by "main"
"main":
  waiting to lock monitor 0x0000000061010c48 (object 0x00000000ca6227b0, a org.springframework.context.event.AbstractApplicationEventMulticaster$ListenerRetriever),
  which is held by "RMI TCP Connection(4)-10.14.5.84"

Java stack information for the threads listed above:
===================================================
"RMI TCP Connection(4)-10.14.5.84":
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
    - waiting to lock <0x00000000c654bdd0> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:166)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:148)
    - locked <0x00000000ca6227b0> (a org.springframework.context.event.AbstractApplicationEventMulticaster$ListenerRetriever)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:86)
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303)
    at org.springframework.security.authentication.DefaultAuthenticationEventPublisher.publishAuthenticationSuccess(DefaultAuthenticationEventPublisher.java:78)
    at org.springframework.security.authentication.ProviderManager.doAuthentication(ProviderManager.java:163)
    at org.springframework.security.authentication.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:48)
    at com.example.JmxSecurityAuthenticator.authenticate(JmxSecurityAuthenticator.java:24)
    at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:213)
    at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:180)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
    at sun.rmi.transport.Transport$1.run(Transport.java:159)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
"main":
    at org.springframework.context.event.AbstractApplicationEventMulticaster.addApplicationListener(AbstractApplicationEventMulticaster.java:63)
    - waiting to lock <0x00000000ca6227b0> (a org.springframework.context.event.AbstractApplicationEventMulticaster$ListenerRetriever)
    at org.springframework.context.support.AbstractApplicationContext.addApplicationListener(AbstractApplicationContext.java:376)
    at org.springframework.context.support.AbstractApplicationContext$ApplicationListenerDetector.postProcessAfterInitialization(AbstractApplicationContext.java:1366)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:407)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1426)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    - locked <0x00000000c654bdd0> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:281)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
    - locked <0x00000000c6573458> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
    - locked <0x00000000c64112a0> (a java.lang.Object)
    at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:140)
    at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:94)
    at org.apache.camel.spring.Main.createDefaultApplicationContext(Main.java:175)
    at org.apache.camel.spring.Main.doStart(Main.java:139)
    at org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:67)
    at org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:54)
    at org.apache.camel.impl.MainSupport.run(MainSupport.java:136)
    at org.apache.camel.impl.MainSupport.run(MainSupport.java:322)
    at org.apache.camel.spring.Main.main(Main.java:72)
    at com.example.RouteBuilder.main(OutboundRouteBuilder.java:71)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.classworlds.Launcher.launchStandard(Launcher.java:410)
    at org.codehaus.classworlds.Launcher.launch(Launcher.java:344)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:461)

Found 1 deadlock.

Comment From: spring-projects-issues

Piotr Bzdyl commented

I forgot to mention the deadlock occurred in 3.0.5.RELEASE.

Comment From: spring-projects-issues

Alex Rau commented

Same here with 3.1.0.RELEASE

Comment From: spring-projects-issues

Nicholas DiPiazza commented

This test case reproduces the issue.

Comment From: spring-projects-issues

Chris Beams commented

Thanks, Nicholas.

Comment From: spring-projects-issues

Juergen Hoeller commented

Note that along with the changes for #14452, several locking improvements made it into Spring Framework 3.2, including changes that avoid the deadlock potential outlined in the comments on this issue.

The deadlock-related changes have also been backported to Spring Framework 3.1.4.

Juergen

Comment From: spring-projects-issues

Hendy Irawan commented

I'm still getting this issue using 3.2.8.RELEASE :

Thread [CPU-00] (Suspended) 
    waiting for: ConcurrentHashMap  (id=494)    
    DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory<?>) line: 207   
    DefaultListableBeanFactory(AbstractBeanFactory).doGetBean(String, Class<T>, Object[], boolean) line: 293    
    DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 194   
    ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(Object, Method, Object[], MethodProxy) line: 305 
    AppConfig$$EnhancerBySpringCGLIB$$4d90c488.sysConfigMap() line: not available   
    AppConfig$SecurityConfig$2.create(String, AppManifest) line: 1017   
    AppConfig$SecurityConfig$2.create(String, AppManifest) line: 1  
    TenantBeans$CreateAndPut.call() line: 177   
    TenantBeans$CreateAndPut.call() line: 1 
    ListenableFutureTask(FutureTask).run() line: 266    
    ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142  
    ThreadPoolExecutor$Worker.run() line: 617   
    Thread.run() line: 745

Comment From: spring-projects-issues

Hendy Irawan commented

Blocks https://github.com/soluvas/soluvas-framework/issues/66

Comment From: spring-projects-issues

Yvonnick Esnault commented

Hi,

This issue is not resolved (version 3.2.5 here)

"HTTP-20"   prio=10 tid=0x00007f673408b000 nid=0x7f0f waiting for monitor entry [0x00007f68c9761000] 
java.lang.Thread.State: BLOCKED (on object monitor) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181) 
- waiting to lock <0x00000007683f4fc8> (a java.util.concurrent.ConcurrentHashMap) 
at org.springframework.beans.factory.support.AbstractBeanFactory.isSingleton(AbstractBeanFactory.java:386) 
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:356) 
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:326) 
at org.springframework.context.support.AbstractApplicationContext.getBeanNamesForType(AbstractApplicationContext.java:1178)

Comment From: spring-projects-issues

Juergen Hoeller commented

Henry, Yvonnick,

Could you please clarify: Are we talking about thread contention here? Specifically, it doesn't seem like you're running into a deadlock but rather 'just' into wait times for a lock?

Juergen

Comment From: spring-projects-issues

Yvonnick Esnault commented

It's a deadlock :

"schedulerFactory_Worker-7":
  waiting to lock monitor 0x00007f6830034ca8 (object 0x000000076a3a02f0, a org.apache.cxf.bus.extension.Extension),
  which is held by "HTTP-20"
"HTTP-20":
  waiting to lock monitor 0x00007f688c001ca8 (object 0x00000007683f4fc8, a java.util.concurrent.ConcurrentHashMap),
  which is held by "schedulerFactory_Worker-7"

with schedulerFactory_Worker-7 like that :

"schedulerFactory_Worker-7":
  at org.apache.cxf.bus.extension.ExtensionManagerImpl.loadBeansOfType(ExtensionManagerImpl.java:346)
  - waiting to lock <0x000000076a3a02f0> (a org.apache.cxf.bus.extension.Extension)
  at org.apache.cxf.bus.spring.SpringBeanLocator.loadBeansOfType(SpringBeanLocator.java:235)
  at org.apache.cxf.transport.TransportFinder.loadDefaultURIs(TransportFinder.java:212)
  at org.apache.cxf.transport.TransportFinder.findTransportForURI(TransportFinder.java:81)
  at org.apache.cxf.bus.managers.ConduitInitiatorManagerImpl.getConduitInitiatorForUri(ConduitInitiatorManagerImpl.java:149)
  at org.apache.cxf.transport.TransportURIResolver.resolve(TransportURIResolver.java:102)
  at org.apache.cxf.catalog.CatalogWSDLLocator.getImportInputSource(CatalogWSDLLocator.java:112)
  at org.apache.cxf.wsdl11.AbstractWrapperWSDLLocator.getImportInputSource(AbstractWrapperWSDLLocator.java:85)
  at com.ibm.wsdl.xml.WSDLReaderImpl.parseImport(WSDLReaderImpl.java:388)
  at com.ibm.wsdl.xml.WSDLReaderImpl.parseDefinitions(WSDLReaderImpl.java:312)
  at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(WSDLReaderImpl.java:2352)
  at com.ibm.wsdl.xml.WSDLReaderImpl.readWSDL(WSDLReaderImpl.java:2338)
  at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:261)
  at org.apache.cxf.wsdl11.WSDLManagerImpl.getDefinition(WSDLManagerImpl.java:206)
  at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.isEmptywsdl(ReflectionServiceFactoryBean.java:2607)
  at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.isFromWsdl(ReflectionServiceFactoryBean.java:543)
  at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:547)
  at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:265)
  - locked <0x0000000760d89360> (a org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean)
  at org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:215)
  at org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:102)
  at org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:91)
  at org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean.java:157)
  - locked <0x00000007652d5470> (a org.apache.cxf.jaxws.spring.JaxWsProxyFactoryBeanDefinitionParser$JAXWSSpringClientProxyFactoryBean)
  at org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBean.java:142)
  - locked <0x00000007652d5470> (a org.apache.cxf.jaxws.spring.JaxWsProxyFactoryBeanDefinitionParser$JAXWSSpringClientProxyFactoryBean)
  at org.apache.cxf.jaxws.spring.JaxWsProxyFactoryBeanDefinitionParser$JAXWSSpringClientProxyFactoryBean.create(JaxWsProxyFactoryBeanDefinitionParser.java:79)
  at org.apache.cxf.jaxws.spring.JaxWsProxyFactoryBeanDefinitionParser$JAXWSSpringClientProxyFactoryBean.getObject(JaxWsProxyFactoryBeanDefinitionParser.java:83)
  - locked <0x00000007652d5470> (a org.apache.cxf.jaxws.spring.JaxWsProxyFactoryBeanDefinitionParser$JAXWSSpringClientProxyFactoryBean)
  at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
  at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
  - locked <0x00000007683f4fc8> (a java.util.concurrent.ConcurrentHashMap)
  at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1468)
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:249)
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:198)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:444)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:418)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:546)
  at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:150)
  at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:303)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1146)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  - locked <0x00000007683f4fc8> (a java.util.concurrent.ConcurrentHashMap)
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:198)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:444)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:418)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:546)
  at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:150)
  at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
  at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:303)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1146)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  - locked <0x00000007627c2780> (a java.util.concurrent.ConcurrentHashMap)
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
  at org.springframework.aop.target.LazyInitTargetSource.getTarget(LazyInitTargetSource.java:67)
  - locked <0x000000076713cf48> (a org.springframework.aop.target.LazyInitTargetSource)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.getTarget(CglibAopProxy.java:663)
  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:614)
[...]

and HTTP-20 :

"HTTP-20":
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:181)
    - waiting to lock <0x00000007683f4fc8> (a java.util.concurrent.ConcurrentHashMap)
    at org.springframework.beans.factory.support.AbstractBeanFactory.isSingleton(AbstractBeanFactory.java:386)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:356)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:326)
    at org.springframework.context.support.AbstractApplicationContext.getBeanNamesForType(AbstractApplicationContext.java:1178)
    at org.apache.cxf.bus.spring.SpringBeanLocator.getBeansOfType(SpringBeanLocator.java:155)
    at org.apache.cxf.transports.http.internal.QueryHandlerRegistryImpl.setBus(QueryHandlerRegistryImpl.java:68)
    at org.apache.cxf.transports.http.internal.QueryHandlerRegistryImpl.<init>(QueryHandlerRegistryImpl.java:43)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.cxf.bus.extension.Extension.load(Extension.java:208)
    at org.apache.cxf.bus.extension.ExtensionManagerImpl.loadAndRegister(ExtensionManagerImpl.java:214)
    - locked <0x000000076a3a02f0> (a org.apache.cxf.bus.extension.Extension)
    at org.apache.cxf.bus.extension.ExtensionManagerImpl.getBeansOfType(ExtensionManagerImpl.java:335)
    - locked <0x000000076a3a02f0> (a org.apache.cxf.bus.extension.Extension)
    at org.apache.cxf.bus.spring.SpringBeanLocator.getBeansOfType(SpringBeanLocator.java:163)
    at org.apache.cxf.bus.CXFBusImpl.getExtension(CXFBusImpl.java:108)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:211)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:180)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
[...]

Comment From: spring-projects-issues

Juergen Hoeller commented

Hmm that seems to be a deadlock between Spring's singleton lock in the bean factory and some related lock behind a CXF FactoryBean... I'm afraid it's no good idea to wrap Spring bean lookup calls in a custom lock there in CXF, when CXF is currently within a Spring-driven bean creation attempt itself!

Juergen

Comment From: spring-projects-issues

Yvonnick Esnault commented

Some information from the CXF mailing list here http://mail-archives.apache.org/mod_mbox/cxf-users/201311.mbox/%3CA0BA72E3-9696-45EE-BA2D-916F76420769@apache.org%3E

Comment From: spring-projects-issues

Juergen Hoeller commented

That sounds like they'll relax their custom lock in CXF which is definitely worthwhile.

I'm afraid we can't relax the singleton lock on our side, otherwise we'd immediately expose ourselves to a deadlock potential during the creation of lazy singletons with dependencies on other lazy singletons...

So I wonder: What specifically could we do on our side still, while keeping our factory-wide singleton lock?

Juergen

Comment From: spring-projects-issues

Juergen Hoeller commented

I'll keep this marked as resolved for 3.2 then, since I don't see any further changes that we could make without sacrificing our singleton guarantees...

Juergen

Comment From: JasonMacKeigan

This is still an issue in 2019.

Comment From: inad9300

I've encountered the same issue with Spring Boot 2.3.1.RELEASE / Spring 5.2.7.RELEASE...

"Thread-6" #48 prio=5 os_prio=0 cpu=22.27ms elapsed=500.34s tid=0x00007f8642ffa000 nid=0x402e waiting for monitor entry  [0x00007f8528559000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:179)
        - waiting to lock <0x00000000c01f1320> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:517)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:527)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:499)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:485)
        at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:228)
        at org.springframework.beans.factory.annotation.BeanFactoryAnnotationUtils.qualifiedBeanOfType(BeanFactoryAnnotationUtils.java:118)
        at org.springframework.beans.factory.annotation.BeanFactoryAnnotationUtils.qualifiedBeanOfType(BeanFactoryAnnotationUtils.java:95)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.determineQualifiedTransactionManager(TransactionAspectSupport.java:492)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.determineTransactionManager(TransactionAspectSupport.java:473)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:335)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:99)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:178)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy165.findAll(Unknown Source)
        at my.package.Abc.lambda$initialize$0(Abc.java:34)
        at my.package.Abc$$Lambda$919/0x00000008409c0440.run(Unknown Source)
        at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

Comment From: CODESTHN

@jhoeller -- I have encountered same issue with Spring release 5.2.1. Please suggest what to do to. Also, please reopen this issue if possible. Thanks

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
- waiting to lock <0x000008001202c048> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.isSingleton(AbstractBeanFactory.java:399)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:431)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:395)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:515)
at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1197)

Comment From: jhoeller

@CODESTHN Are you seeing specific thread contention for getBeansOfType there, while some other thread is creating singleton beans? Is this a startup problem or happening later on for short-lived bean creation attempts? In any case, please create a separate GitHub issue specific to your scenario, we can try to look at it for 5.3 then.

Comment From: jhoeller

Reviewing the recent stacktraces above, we seem to have a specific case about DefaultSingletonBeanRegistry.getSingleton(beanName, false) here, as called from isSingleton and isTypeMatch checks in AbstractBeanFactory. When isSingletonCurrentlyInCreation returns true, we're trying to enter the global lock for an early singleton exposure check... even if we're not allowed to create early references, so only able to check for an existing early reference. By turning earlySingletonObjects into a ConcurrentHashMap and separating that early reference existence check from the creation of early references (which needs to happen within the global lock), we could potentially address a whole range of issues here, including this latest report, for lookup attempts in concurrent singleton creation scenarios.

I'll create a GitHub issue myself to that effect, even for 5.2.9 since such a change seems straightforward enough.