Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.5.2, 2.5.3, 2.6.0, 2.7.1
    • Fix Version/s: 2.9.0, 10.0
    • Component/s: Content
    • Labels:
      None
    • 2.9 Status:
      Resolved

      Description

      Two request threads from Resources have been running for a very long time (86796s and 86794s respectively), and appear to be in an infinite loop. User action which produced this is unknown.

      "TP-Processor172" daemon prio=1 tid=0x00002aaaacc0fcd0 nid=0x7738 runnable [0x0000000052854000..0x0000000052855da0]
      at java.util.TreeMap.successor(TreeMap.java:1214)
      at java.util.TreeMap.access$400(TreeMap.java:81)
      at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1033)
      at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
      at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6548)
      at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
      at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
      at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

      "TP-Processor223" daemon prio=1 tid=0x00002aaaad6f7090 nid=0x3398 runnable [0x00000000488b5000..0x00000000488b6e20]
      at java.lang.String.startsWith(String.java:1260)
      at java.lang.String.startsWith(String.java:1281)
      at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6549)
      at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
      at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
      at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

              Hide
              smarquard Stephen Marquard added a comment -

              Both threads seem to be stuck inside this loop:

              Iterator it = currentMap.iterator();
              while(it.hasNext())
              {
              String id = (String) it.next();
              if(id.startsWith(collectionId))

              { it.remove(); sortMap.remove(id); removeObservingPattern(id, state); }

              }

              Show
              smarquard Stephen Marquard added a comment - Both threads seem to be stuck inside this loop: Iterator it = currentMap.iterator(); while(it.hasNext()) { String id = (String) it.next(); if(id.startsWith(collectionId)) { it.remove(); sortMap.remove(id); removeObservingPattern(id, state); } }
              Hide
              dhorwitz David Horwitz added a comment -

              MAINTANCE TEAM: Unassigned so these get reviewed by the Maintance Team

              Show
              dhorwitz David Horwitz added a comment - MAINTANCE TEAM: Unassigned so these get reviewed by the Maintance Team
              Hide
              dhorwitz David Horwitz added a comment -

              havent seen this in a while ...

              Show
              dhorwitz David Horwitz added a comment - havent seen this in a while ...
              Hide
              zqian Zhen Qian added a comment - - edited

              Reopen this ticket. We just found another occurrence of this yesterday on CTools production, which generates high CPU:

              "TP-Processor15" daemon prio=10 tid=0x00002aaac8110800 nid=0x6f76 runnable [0x00000000468d9000]
              java.lang.Thread.State: RUNNABLE
              at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176)
              at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
              at java.util.TreeMap.access$000(TreeMap.java:88)
              at java.util.TreeMap$PrivateEntryIterator.remove(TreeMap.java:1125)
              at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6755)
              at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:625)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:535)
              at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032)
              at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
              at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
              at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
              at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
              at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
              at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512)
              at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350)
              at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
              at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
              at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
              at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1177)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
              at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
              at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
              at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
              at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
              at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
              at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
              at java.lang.Thread.run(Thread.java:662)

              Show
              zqian Zhen Qian added a comment - - edited Reopen this ticket. We just found another occurrence of this yesterday on CTools production, which generates high CPU: "TP-Processor15" daemon prio=10 tid=0x00002aaac8110800 nid=0x6f76 runnable [0x00000000468d9000] java.lang.Thread.State: RUNNABLE at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176) at java.util.TreeMap.deleteEntry(TreeMap.java:2151) at java.util.TreeMap.access$000(TreeMap.java:88) at java.util.TreeMap$PrivateEntryIterator.remove(TreeMap.java:1125) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6755) at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:625) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:535) at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032) at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512) at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350) at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204) at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73) at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1177) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662)
              Hide
              zqian Zhen Qian added a comment -

              Line 6755 refers to the same code block as pointed before:

              Iterator it = currentMap.iterator();
              while(it.hasNext())
              {
              String id = (String) it.next();
              if(id.startsWith(collectionId))

              { it.remove(); sortMap.remove(id); <<<<<< removeObservingPattern(id, state); }

              }

              Show
              zqian Zhen Qian added a comment - Line 6755 refers to the same code block as pointed before: Iterator it = currentMap.iterator(); while(it.hasNext()) { String id = (String) it.next(); if(id.startsWith(collectionId)) { it.remove(); sortMap.remove(id); <<<<<< removeObservingPattern(id, state); } }
              Hide
              zqian Zhen Qian added a comment -

              TreeMap is not synchronized for removing elements.

              I found similar problem also been reported from HBase project: https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs The adopted approach there is to add synchronized block for all tree operations.

              Show
              zqian Zhen Qian added a comment - TreeMap is not synchronized for removing elements. I found similar problem also been reported from HBase project: https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs The adopted approach there is to add synchronized block for all tree operations.
              Hide
              zqian Zhen Qian added a comment -

              Another hit today on CTools production server:

              "TP-Processor40" daemon prio=10 tid=0x00002aaaccb30000 nid=0x1ecc runnable [0x0000000048085000]
              java.lang.Thread.State: RUNNABLE
              at java.util.TreeMap.put(TreeMap.java:552)
              at java.util.TreeSet.add(TreeSet.java:238)
              at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:221)
              at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
              at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
              at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
              at org.sakaiproject.content.tool.ResourcesAction.buildListContext(ResourcesAction.java:4424)
              at org.sakaiproject.content.tool.ResourcesAction.buildMainPanelContext(ResourcesAction.java:4720)
              at sun.reflect.GeneratedMethodAccessor1124.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:397)
              at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
              at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
              at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
              at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
              at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
              at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512)
              at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350)
              at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
              at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
              at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:837)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
              at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
              at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
              at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
              at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
              at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
              at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
              at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
              at java.lang.Thread.run(Thread.java:662)

              Show
              zqian Zhen Qian added a comment - Another hit today on CTools production server: "TP-Processor40" daemon prio=10 tid=0x00002aaaccb30000 nid=0x1ecc runnable [0x0000000048085000] java.lang.Thread.State: RUNNABLE at java.util.TreeMap.put(TreeMap.java:552) at java.util.TreeSet.add(TreeSet.java:238) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:221) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ResourcesAction.buildListContext(ResourcesAction.java:4424) at org.sakaiproject.content.tool.ResourcesAction.buildMainPanelContext(ResourcesAction.java:4720) at sun.reflect.GeneratedMethodAccessor1124.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:397) at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512) at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350) at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204) at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:837) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662)
              Hide
              zqian Zhen Qian added a comment -

              attached the patch file to address two recent problems of stuck thread in CTools.

              Show
              zqian Zhen Qian added a comment - attached the patch file to address two recent problems of stuck thread in CTools.
              Hide
              ottenhoff Sam Ottenhoff added a comment -

              Zhen, are you recommending this patch be committed to trunk? Is this code in production for CTools?

              Show
              ottenhoff Sam Ottenhoff added a comment - Zhen, are you recommending this patch be committed to trunk? Is this code in production for CTools?
              Hide
              zqian Zhen Qian added a comment -

              Sam:

              I am waiting for code review and load testing results for the patch.

              One other thing to point out is that the patch is very localized, which just handles the recent reported issues. It doesn't address (many) other places of TreeMap usage inside ResourceAction.java code. I think it is good to do the limited change first.

              Show
              zqian Zhen Qian added a comment - Sam: I am waiting for code review and load testing results for the patch. One other thing to point out is that the patch is very localized, which just handles the recent reported issues. It doesn't address (many) other places of TreeMap usage inside ResourceAction.java code. I think it is good to do the limited change first.
              Hide
              zqian Zhen Qian added a comment -

              I am uploading a new patch file generated by "svn diff" instead of "svn diff -x -w". The new patch file "SAK-13386_withIndentationAdjustment.diff" only differs from the first patch file with indentation.

              Show
              zqian Zhen Qian added a comment - I am uploading a new patch file generated by "svn diff" instead of "svn diff -x -w". The new patch file " SAK-13386 _withIndentationAdjustment.diff" only differs from the first patch file with indentation.
              Hide
              bkirschn Beth Kirschner added a comment -

              Patch reviewed, tested and applied to trunk. UM will be deploying this to production in the summer.

              Show
              bkirschn Beth Kirschner added a comment - Patch reviewed, tested and applied to trunk. UM will be deploying this to production in the summer.
              Hide
              aaronz Aaron Zeckoski added a comment -

              Looks safe enough. There are a lot of sync blocks which will probably result in some JVM pauses under high load but at least it won't cause threadlocks.

              Show
              aaronz Aaron Zeckoski added a comment - Looks safe enough. There are a lot of sync blocks which will probably result in some JVM pauses under high load but at least it won't cause threadlocks.
              Hide
              ottenhoff Sam Ottenhoff added a comment -

              2.9.x: r109815

              Show
              ottenhoff Sam Ottenhoff added a comment - 2.9.x: r109815
              Hide
              buckett Matthew Buckett added a comment -

              I think we've hit this in 2.8.x as well with a stack trace of:

              @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000]
              @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE
              @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
              @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616)
              @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679)
              @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555)
              @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230)
              @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086)
              @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154)
              @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
              @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
              @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
              @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608)
              @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
              @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
              @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
              @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
              @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
              @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
              @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479)
              @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137)
              @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
              @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
              @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270)
              @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
              @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
              @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669)
              @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
              @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
              @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
              @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
              @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
              @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
              @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
              @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
              @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
              @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
              @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
              @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
              @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
              @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)

              Show
              buckett Matthew Buckett added a comment - I think we've hit this in 2.8.x as well with a stack trace of: @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000] @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600) @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616) @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230) @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086) @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154) @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608) @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511) @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479) @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73) @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669) @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)
              Hide
              zqian Zhen Qian added a comment -

              Matthew:

              Can you attach the code for this line?

              org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)

              Show
              zqian Zhen Qian added a comment - Matthew: Can you attach the code for this line? org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
              Hide
              buckett Matthew Buckett added a comment -

              So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the
              iterator for the treeset has become broken through concurrent access.

              // This is where expandedItems gets set.
              SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS);

              // Here is the itteration
              SortedSet newSet = new TreeSet();
              Iterator l = expandedItems.iterator();
              while (l.hasNext ())
              {
              String id = (String) l.next();

              if (id.startsWith(collectionId))
              {
              }
              else
              {
              // newSet.put(id,collection);
              newSet.add(id); // 5600
              }

              Show
              buckett Matthew Buckett added a comment - So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the iterator for the treeset has become broken through concurrent access. // This is where expandedItems gets set. SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS); // Here is the itteration SortedSet newSet = new TreeSet(); Iterator l = expandedItems.iterator(); while (l.hasNext ()) { String id = (String) l.next(); if (id.startsWith(collectionId)) { } else { // newSet.put(id,collection); newSet.add(id); // 5600 }
              Hide
              buckett Matthew Buckett added a comment -

              We're still seeing this from the content tool so I've opened a new issue SAK-23514

              Show
              buckett Matthew Buckett added a comment - We're still seeing this from the content tool so I've opened a new issue SAK-23514
              Hide
              zqian Zhen Qian added a comment -

              We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java:

              java.lang.Thread.State: RUNNABLE
              at java.util.HashMap.hash(HashMap.java:351)
              at java.util.HashMap.removeEntryForKey(HashMap.java:645)
              at java.util.HashMap.remove(HashMap.java:635)
              at java.util.Collections$SynchronizedMap.remove(Collections.java:2044)

              • locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap)
                at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694)
                ....

              java.lang.Thread.State: RUNNABLE
              at org.apache.log4j.Category.getEffectiveLevel(Category.java:442)
              at org.apache.log4j.Category.log(Category.java:855)
              at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)
              at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424)
              at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695)
              .....

              I will open a new jira for this.

              Show
              zqian Zhen Qian added a comment - We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java: java.lang.Thread.State: RUNNABLE at java.util.HashMap.hash(HashMap.java:351) at java.util.HashMap.removeEntryForKey(HashMap.java:645) at java.util.HashMap.remove(HashMap.java:635) at java.util.Collections$SynchronizedMap.remove(Collections.java:2044) locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694) .... java.lang.Thread.State: RUNNABLE at org.apache.log4j.Category.getEffectiveLevel(Category.java:442) at org.apache.log4j.Category.log(Category.java:855) at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177) at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695) ..... I will open a new jira for this.

                People

                • Assignee:
                  zqian Zhen Qian
                  Reporter:
                  smarquard Stephen Marquard
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  10 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Git Source Code