Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.5.2, 2.5.3, 2.6.0, 2.7.1
    • Fix Version/s: 2.9.0, 10.0-qa01
    • Component/s: Content
    • Labels:
      None
    • 2.9 Status:
      Resolved
    • Previous Issue Keys:

      Description

      Two request threads from Resources have been running for a very long time (86796s and 86794s respectively), and appear to be in an infinite loop. User action which produced this is unknown.

      "TP-Processor172" daemon prio=1 tid=0x00002aaaacc0fcd0 nid=0x7738 runnable [0x0000000052854000..0x0000000052855da0]
              at java.util.TreeMap.successor(TreeMap.java:1214)
              at java.util.TreeMap.access$400(TreeMap.java:81)
              at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1033)
              at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
              at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6548)
              at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
              at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
              at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
              at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

      "TP-Processor223" daemon prio=1 tid=0x00002aaaad6f7090 nid=0x3398 runnable [0x00000000488b5000..0x00000000488b6e20]
              at java.lang.String.startsWith(String.java:1260)
              at java.lang.String.startsWith(String.java:1281)
              at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6549)
              at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
              at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
              at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
              at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

        Issue Links

          Activity

          Hide
          Stephen Marquard added a comment -
          Both threads seem to be stuck inside this loop:

            Iterator it = currentMap.iterator();
                                  while(it.hasNext())
                                  {
                                          String id = (String) it.next();
                                          if(id.startsWith(collectionId))
                                          {
                                                  it.remove();
                                                  sortMap.remove(id);
                                                  removeObservingPattern(id, state);
                                          }
                                  }

          Show
          Stephen Marquard added a comment - Both threads seem to be stuck inside this loop:   Iterator it = currentMap.iterator();                         while(it.hasNext())                         {                                 String id = (String) it.next();                                 if(id.startsWith(collectionId))                                 {                                         it.remove();                                         sortMap.remove(id);                                         removeObservingPattern(id, state);                                 }                         }
          Hide
          David Horwitz added a comment -
          MAINTANCE TEAM: Unassigned so these get reviewed by the Maintance Team
          Show
          David Horwitz added a comment - MAINTANCE TEAM: Unassigned so these get reviewed by the Maintance Team
          Hide
          David Horwitz added a comment -
          havent seen this in a while ...
          Show
          David Horwitz added a comment - havent seen this in a while ...
          Hide
          Zhen Qian added a comment - - edited
          Reopen this ticket. We just found another occurrence of this yesterday on CTools production, which generates high CPU:


          "TP-Processor15" daemon prio=10 tid=0x00002aaac8110800 nid=0x6f76 runnable [0x00000000468d9000]
             java.lang.Thread.State: RUNNABLE
          at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176)
          at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
          at java.util.TreeMap.access$000(TreeMap.java:88)
          at java.util.TreeMap$PrivateEntryIterator.remove(TreeMap.java:1125)
          at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6755)
          at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:625)
          at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:535)
          at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230)
          at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032)
          at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
          at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
          at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
          at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
          at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
          at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512)
          at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350)
          at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
          at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
          at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
          at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1177)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
          at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
          at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
          at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
          at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:662)
          Show
          Zhen Qian added a comment - - edited Reopen this ticket. We just found another occurrence of this yesterday on CTools production, which generates high CPU: "TP-Processor15" daemon prio=10 tid=0x00002aaac8110800 nid=0x6f76 runnable [0x00000000468d9000]    java.lang.Thread.State: RUNNABLE at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176) at java.util.TreeMap.deleteEntry(TreeMap.java:2151) at java.util.TreeMap.access$000(TreeMap.java:88) at java.util.TreeMap$PrivateEntryIterator.remove(TreeMap.java:1125) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6755) at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:625) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:535) at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032) at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512) at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350) at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204) at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73) at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1177) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662)
          Hide
          Zhen Qian added a comment -
          Line 6755 refers to the same code block as pointed before:

                                  Iterator it = currentMap.iterator();
          while(it.hasNext())
          {
          String id = (String) it.next();
          if(id.startsWith(collectionId))
          {
          it.remove();
          sortMap.remove(id); <<<<<<
          removeObservingPattern(id, state);
          }
          }
          Show
          Zhen Qian added a comment - Line 6755 refers to the same code block as pointed before:                         Iterator it = currentMap.iterator(); while(it.hasNext()) { String id = (String) it.next(); if(id.startsWith(collectionId)) { it.remove(); sortMap.remove(id); <<<<<< removeObservingPattern(id, state); } }
          Hide
          Zhen Qian added a comment -
          TreeMap is not synchronized for removing elements.

          I found similar problem also been reported from HBase project: https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs The adopted approach there is to add synchronized block for all tree operations.
          Show
          Zhen Qian added a comment - TreeMap is not synchronized for removing elements. I found similar problem also been reported from HBase project: https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs The adopted approach there is to add synchronized block for all tree operations.
          Hide
          Zhen Qian added a comment -
          Another hit today on CTools production server:

          "TP-Processor40" daemon prio=10 tid=0x00002aaaccb30000 nid=0x1ecc runnable [0x0000000048085000]
             java.lang.Thread.State: RUNNABLE
          at java.util.TreeMap.put(TreeMap.java:552)
          at java.util.TreeSet.add(TreeSet.java:238)
          at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:221)
          at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
          at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
          at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
          at org.sakaiproject.content.tool.ResourcesAction.buildListContext(ResourcesAction.java:4424)
          at org.sakaiproject.content.tool.ResourcesAction.buildMainPanelContext(ResourcesAction.java:4720)
          at sun.reflect.GeneratedMethodAccessor1124.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:397)
          at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236)
          at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
          at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
          at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
          at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
          at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
          at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512)
          at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350)
          at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
          at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
          at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:837)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
          at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
          at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
          at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
          at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
          at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          at java.lang.Thread.run(Thread.java:662)

          Show
          Zhen Qian added a comment - Another hit today on CTools production server: "TP-Processor40" daemon prio=10 tid=0x00002aaaccb30000 nid=0x1ecc runnable [0x0000000048085000]    java.lang.Thread.State: RUNNABLE at java.util.TreeMap.put(TreeMap.java:552) at java.util.TreeSet.add(TreeSet.java:238) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:221) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ResourcesAction.buildListContext(ResourcesAction.java:4424) at org.sakaiproject.content.tool.ResourcesAction.buildMainPanelContext(ResourcesAction.java:4720) at sun.reflect.GeneratedMethodAccessor1124.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:397) at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512) at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350) at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204) at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:837) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662)
          Hide
          Zhen Qian added a comment -
          attached the patch file to address two recent problems of stuck thread in CTools.
          Show
          Zhen Qian added a comment - attached the patch file to address two recent problems of stuck thread in CTools.
          Hide
          Sam Ottenhoff added a comment -
          Zhen, are you recommending this patch be committed to trunk? Is this code in production for CTools?
          Show
          Sam Ottenhoff added a comment - Zhen, are you recommending this patch be committed to trunk? Is this code in production for CTools?
          Hide
          Zhen Qian added a comment -
          Sam:

          I am waiting for code review and load testing results for the patch.

          One other thing to point out is that the patch is very localized, which just handles the recent reported issues. It doesn't address (many) other places of TreeMap usage inside ResourceAction.java code. I think it is good to do the limited change first.

          Show
          Zhen Qian added a comment - Sam: I am waiting for code review and load testing results for the patch. One other thing to point out is that the patch is very localized, which just handles the recent reported issues. It doesn't address (many) other places of TreeMap usage inside ResourceAction.java code. I think it is good to do the limited change first.
          Hide
          Zhen Qian added a comment -
          I am uploading a new patch file generated by "svn diff" instead of "svn diff -x -w". The new patch file "SAK-13386_withIndentationAdjustment.diff" only differs from the first patch file with indentation.
          Show
          Zhen Qian added a comment - I am uploading a new patch file generated by "svn diff" instead of "svn diff -x -w". The new patch file " SAK-13386 _withIndentationAdjustment.diff" only differs from the first patch file with indentation.
          Hide
          Beth Kirschner added a comment -
          Patch reviewed, tested and applied to trunk. UM will be deploying this to production in the summer.
          Show
          Beth Kirschner added a comment - Patch reviewed, tested and applied to trunk. UM will be deploying this to production in the summer.
          Hide
          Aaron Zeckoski added a comment -
          Looks safe enough. There are a lot of sync blocks which will probably result in some JVM pauses under high load but at least it won't cause threadlocks.
          Show
          Aaron Zeckoski added a comment - Looks safe enough. There are a lot of sync blocks which will probably result in some JVM pauses under high load but at least it won't cause threadlocks.
          Hide
          Sam Ottenhoff added a comment -
          2.9.x: r109815
          Show
          Sam Ottenhoff added a comment - 2.9.x: r109815
          Hide
          Matthew Buckett added a comment -
          I think we've hit this in 2.8.x as well with a stack trace of:

          @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000]
          @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE
          @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
          @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616)
          @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679)
          @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555)
          @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230)
          @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086)
          @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154)
          @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
          @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
          @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608)
          @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
          @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
          @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
          @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
          @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
          @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479)
          @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137)
          @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
          @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
          @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270)
          @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
          @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669)
          @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
          @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
          @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
          @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
          @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
          @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
          @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
          @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)
          Show
          Matthew Buckett added a comment - I think we've hit this in 2.8.x as well with a stack trace of: @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000] @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600) @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616) @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230) @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086) @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154) @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608) @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511) @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479) @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73) @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669) @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)
          Hide
          Zhen Qian added a comment -
          Matthew:

          Can you attach the code for this line?

          org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)

          Show
          Zhen Qian added a comment - Matthew: Can you attach the code for this line? org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
          Hide
          Matthew Buckett added a comment -
          So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the
          iterator for the treeset has become broken through concurrent access.

          // This is where expandedItems gets set.
          SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS);


          // Here is the itteration
          SortedSet newSet = new TreeSet();
          Iterator l = expandedItems.iterator();
          while (l.hasNext ())
          {
          String id = (String) l.next();


          if (id.startsWith(collectionId))
          {
          }
          else
          {
          // newSet.put(id,collection);
          newSet.add(id); // 5600
          }
          Show
          Matthew Buckett added a comment - So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the iterator for the treeset has become broken through concurrent access. // This is where expandedItems gets set. SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS); // Here is the itteration SortedSet newSet = new TreeSet(); Iterator l = expandedItems.iterator(); while (l.hasNext ()) { String id = (String) l.next(); if (id.startsWith(collectionId)) { } else { // newSet.put(id,collection); newSet.add(id); // 5600 }
          Hide
          Matthew Buckett added a comment -
          We're still seeing this from the content tool so I've opened a new issue SAK-23514
          Show
          Matthew Buckett added a comment - We're still seeing this from the content tool so I've opened a new issue SAK-23514
          Hide
          Zhen Qian added a comment -
          We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java:

          java.lang.Thread.State: RUNNABLE
          at java.util.HashMap.hash(HashMap.java:351)
          at java.util.HashMap.removeEntryForKey(HashMap.java:645)
          at java.util.HashMap.remove(HashMap.java:635)
          at java.util.Collections$SynchronizedMap.remove(Collections.java:2044)
          - locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap)
          at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694)
          ....

             java.lang.Thread.State: RUNNABLE
          at org.apache.log4j.Category.getEffectiveLevel(Category.java:442)
          at org.apache.log4j.Category.log(Category.java:855)
          at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)
          at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424)
          at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695)
          .....

          I will open a new jira for this.


          Show
          Zhen Qian added a comment - We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java: java.lang.Thread.State: RUNNABLE at java.util.HashMap.hash(HashMap.java:351) at java.util.HashMap.removeEntryForKey(HashMap.java:645) at java.util.HashMap.remove(HashMap.java:635) at java.util.Collections$SynchronizedMap.remove(Collections.java:2044) - locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694) ....    java.lang.Thread.State: RUNNABLE at org.apache.log4j.Category.getEffectiveLevel(Category.java:442) at org.apache.log4j.Category.log(Category.java:855) at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177) at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695) ..... I will open a new jira for this.

            People

            • Assignee:
              Zhen Qian
              Reporter:
              Stephen Marquard
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: