Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-44072

Request threads blocked on spring-beans synchronization (Spring Update needed)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: RESOLVED
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 19.5, 19.6 [Tentative], 20.1, 20.2 [Tentative]
    • Fix Version/s: 20.2 [Tentative]
    • Component/s: None
    • Labels:
      None
    • 19 status:
      Please Merge
    • Test Plan:
      Hide

      No regressions in anything using spring-beans.

      It would be nice to merge this back to 19, but that would probably require a regression test on 20. 

      If anyone does some brief regression test on 20 that should be enough. This was just a minor update of Spring (an internal library) so if all the tools minimally work that should be enough. 

      Show
      No regressions in anything using spring-beans. It would be nice to merge this back to 19, but that would probably require a regression test on 20.  If anyone does some brief regression test on 20 that should be enough. This was just a minor update of Spring (an internal library) so if all the tools minimally work that should be enough. 

      Description

      We observed a large number of request threads blocked like this:

      "ajp-nio-0.0.0.0-8009-exec-240" #2602 daemon prio=5 os_prio=0 tid=0x00007ff1781cb000 nid=0x1d23 waiting for monitor entry [0x00007ff08b09c000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.springframework.beans.factory.support.MethodOverrides.getOverride(MethodOverrides.java:102)
      	- waiting to lock <0x0000000680638250> (a java.util.Collections$SynchronizedSet)
      	at org.springframework.beans.factory.support.CglibSubclassingInstantiationStrategy$LookupOverrideMethodInterceptor.intercept(CglibSubclassingInstantiationStrategy.java:282)
      	at org.sakaiproject.tool.impl.SessionComponent$$EnhancerBySpringCGLIB$$3e5c29b8.threadLocalManager(<generated>)
      	at org.sakaiproject.tool.impl.SessionComponent.getCurrentSession(SessionComponent.java:398)
      

      This is waiting for a synchronized structure inside spring-beans-4.3.25.RELEASE.jar:

      https://github.com/spring-projects/spring-framework/blob/4.3.x/spring-beans/src/main/java/org/springframework/beans/factory/support/MethodOverrides.java#L102

      The synchronization was removed in https://github.com/spring-projects/spring-framework/commit/da44a247cb08013c42f9bde70ee6a99e8eb7c62b for Spring 5.x for this issue:

      https://github.com/spring-projects/spring-framework/issues/23448

      While this is unusual (and in the case above triggered by a large volume of presence requests and some custom code in the call stack), it could show up under heavy load or possibly certain race conditions.

      The fix above is not yet available in Spring 4.x, but it is possible to build a custom jar from source including this fix, a backported version of which is attached.

      Spring issue for backporting this to 4.x:

      https://github.com/spring-projects/spring-framework/issues/25549

        Gliffy Diagrams

          Zeplin

            Attachments

              Activity

                People

                Assignee:
                jonespm Matthew Jones
                Reporter:
                smarquard Stephen Marquard
                Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    Git Integration