Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.5.2, 2.5.3, 2.6.0, 2.7.1
    • Fix Version/s: 2.9.0, 10.0
    • Component/s: Content
    • Labels:
      None
    • 2.9 Status:
      Resolved
    • Previous Issue Keys:

      Description

      Two request threads from Resources have been running for a very long time (86796s and 86794s respectively), and appear to be in an infinite loop. User action which produced this is unknown.

      "TP-Processor172" daemon prio=1 tid=0x00002aaaacc0fcd0 nid=0x7738 runnable [0x0000000052854000..0x0000000052855da0]
      at java.util.TreeMap.successor(TreeMap.java:1214)
      at java.util.TreeMap.access$400(TreeMap.java:81)
      at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1033)
      at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
      at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6548)
      at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
      at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
      at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

      "TP-Processor223" daemon prio=1 tid=0x00002aaaad6f7090 nid=0x3398 runnable [0x00000000488b5000..0x00000000488b6e20]
      at java.lang.String.startsWith(String.java:1260)
      at java.lang.String.startsWith(String.java:1281)
      at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6549)
      at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
      at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
      at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
      at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            Stephen Marquard added a comment -

            Both threads seem to be stuck inside this loop:

            Iterator it = currentMap.iterator();
            while(it.hasNext())
            {
            String id = (String) it.next();
            if(id.startsWith(collectionId))

            { it.remove(); sortMap.remove(id); removeObservingPattern(id, state); }

            }

            Show
            Stephen Marquard added a comment - Both threads seem to be stuck inside this loop: Iterator it = currentMap.iterator(); while(it.hasNext()) { String id = (String) it.next(); if(id.startsWith(collectionId)) { it.remove(); sortMap.remove(id); removeObservingPattern(id, state); } }
            Hide
            David Horwitz added a comment -

            MAINTANCE TEAM: Unassigned so these get reviewed by the Maintance Team

            Show
            David Horwitz added a comment - MAINTANCE TEAM: Unassigned so these get reviewed by the Maintance Team
            Hide
            David Horwitz added a comment -

            havent seen this in a while ...

            Show
            David Horwitz added a comment - havent seen this in a while ...
            Hide
            Zhen Qian added a comment - - edited

            Reopen this ticket. We just found another occurrence of this yesterday on CTools production, which generates high CPU:

            "TP-Processor15" daemon prio=10 tid=0x00002aaac8110800 nid=0x6f76 runnable [0x00000000468d9000]
            java.lang.Thread.State: RUNNABLE
            at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176)
            at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
            at java.util.TreeMap.access$000(TreeMap.java:88)
            at java.util.TreeMap$PrivateEntryIterator.remove(TreeMap.java:1125)
            at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6755)
            at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:625)
            at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:535)
            at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230)
            at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032)
            at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
            at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
            at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
            at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
            at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
            at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512)
            at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350)
            at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
            at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
            at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
            at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1177)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
            at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
            at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
            at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
            at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
            at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
            at java.lang.Thread.run(Thread.java:662)

            Show
            Zhen Qian added a comment - - edited Reopen this ticket. We just found another occurrence of this yesterday on CTools production, which generates high CPU: "TP-Processor15" daemon prio=10 tid=0x00002aaac8110800 nid=0x6f76 runnable [0x00000000468d9000] java.lang.Thread.State: RUNNABLE at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176) at java.util.TreeMap.deleteEntry(TreeMap.java:2151) at java.util.TreeMap.access$000(TreeMap.java:88) at java.util.TreeMap$PrivateEntryIterator.remove(TreeMap.java:1125) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6755) at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:625) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:535) at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032) at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512) at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350) at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204) at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73) at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1177) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662)
            Hide
            Zhen Qian added a comment -

            Line 6755 refers to the same code block as pointed before:

            Iterator it = currentMap.iterator();
            while(it.hasNext())
            {
            String id = (String) it.next();
            if(id.startsWith(collectionId))

            { it.remove(); sortMap.remove(id); <<<<<< removeObservingPattern(id, state); }

            }

            Show
            Zhen Qian added a comment - Line 6755 refers to the same code block as pointed before: Iterator it = currentMap.iterator(); while(it.hasNext()) { String id = (String) it.next(); if(id.startsWith(collectionId)) { it.remove(); sortMap.remove(id); <<<<<< removeObservingPattern(id, state); } }
            Hide
            Zhen Qian added a comment -

            TreeMap is not synchronized for removing elements.

            I found similar problem also been reported from HBase project: https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs The adopted approach there is to add synchronized block for all tree operations.

            Show
            Zhen Qian added a comment - TreeMap is not synchronized for removing elements. I found similar problem also been reported from HBase project: https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs The adopted approach there is to add synchronized block for all tree operations.
            Hide
            Zhen Qian added a comment -

            Another hit today on CTools production server:

            "TP-Processor40" daemon prio=10 tid=0x00002aaaccb30000 nid=0x1ecc runnable [0x0000000048085000]
            java.lang.Thread.State: RUNNABLE
            at java.util.TreeMap.put(TreeMap.java:552)
            at java.util.TreeSet.add(TreeSet.java:238)
            at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:221)
            at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
            at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
            at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288)
            at org.sakaiproject.content.tool.ResourcesAction.buildListContext(ResourcesAction.java:4424)
            at org.sakaiproject.content.tool.ResourcesAction.buildMainPanelContext(ResourcesAction.java:4720)
            at sun.reflect.GeneratedMethodAccessor1124.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:397)
            at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236)
            at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
            at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
            at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
            at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
            at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
            at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512)
            at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350)
            at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
            at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
            at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:837)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658)
            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
            at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
            at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
            at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
            at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
            at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
            at java.lang.Thread.run(Thread.java:662)

            Show
            Zhen Qian added a comment - Another hit today on CTools production server: "TP-Processor40" daemon prio=10 tid=0x00002aaaccb30000 nid=0x1ecc runnable [0x0000000048085000] java.lang.Thread.State: RUNNABLE at java.util.TreeMap.put(TreeMap.java:552) at java.util.TreeSet.add(TreeSet.java:238) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:221) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ListItem.getListItem(ListItem.java:288) at org.sakaiproject.content.tool.ResourcesAction.buildListContext(ResourcesAction.java:4424) at org.sakaiproject.content.tool.ResourcesAction.buildMainPanelContext(ResourcesAction.java:4720) at sun.reflect.GeneratedMethodAccessor1124.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:397) at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236) at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1032) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:597) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:512) at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1350) at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204) at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:837) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:662)
            Hide
            Zhen Qian added a comment -

            attached the patch file to address two recent problems of stuck thread in CTools.

            Show
            Zhen Qian added a comment - attached the patch file to address two recent problems of stuck thread in CTools.
            Hide
            Sam Ottenhoff added a comment -

            Zhen, are you recommending this patch be committed to trunk? Is this code in production for CTools?

            Show
            Sam Ottenhoff added a comment - Zhen, are you recommending this patch be committed to trunk? Is this code in production for CTools?
            Hide
            Zhen Qian added a comment -

            Sam:

            I am waiting for code review and load testing results for the patch.

            One other thing to point out is that the patch is very localized, which just handles the recent reported issues. It doesn't address (many) other places of TreeMap usage inside ResourceAction.java code. I think it is good to do the limited change first.

            Show
            Zhen Qian added a comment - Sam: I am waiting for code review and load testing results for the patch. One other thing to point out is that the patch is very localized, which just handles the recent reported issues. It doesn't address (many) other places of TreeMap usage inside ResourceAction.java code. I think it is good to do the limited change first.
            Hide
            Zhen Qian added a comment -

            I am uploading a new patch file generated by "svn diff" instead of "svn diff -x -w". The new patch file "SAK-13386_withIndentationAdjustment.diff" only differs from the first patch file with indentation.

            Show
            Zhen Qian added a comment - I am uploading a new patch file generated by "svn diff" instead of "svn diff -x -w". The new patch file " SAK-13386 _withIndentationAdjustment.diff" only differs from the first patch file with indentation.
            Hide
            Beth Kirschner added a comment -

            Patch reviewed, tested and applied to trunk. UM will be deploying this to production in the summer.

            Show
            Beth Kirschner added a comment - Patch reviewed, tested and applied to trunk. UM will be deploying this to production in the summer.
            Hide
            Aaron Zeckoski added a comment -

            Looks safe enough. There are a lot of sync blocks which will probably result in some JVM pauses under high load but at least it won't cause threadlocks.

            Show
            Aaron Zeckoski added a comment - Looks safe enough. There are a lot of sync blocks which will probably result in some JVM pauses under high load but at least it won't cause threadlocks.
            Hide
            Sam Ottenhoff added a comment -

            2.9.x: r109815

            Show
            Sam Ottenhoff added a comment - 2.9.x: r109815
            Hide
            Matthew Buckett added a comment -

            I think we've hit this in 2.8.x as well with a stack trace of:

            @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000]
            @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE
            @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
            @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616)
            @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679)
            @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555)
            @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230)
            @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086)
            @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154)
            @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
            @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
            @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
            @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
            @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608)
            @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
            @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
            @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
            @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
            @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
            @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
            @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479)
            @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137)
            @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
            @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
            @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270)
            @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
            @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
            @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
            @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669)
            @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
            @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
            @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
            @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
            @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
            @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
            @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
            @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
            @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
            @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
            @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
            @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
            @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
            @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
            @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
            @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)

            Show
            Matthew Buckett added a comment - I think we've hit this in 2.8.x as well with a stack trace of: @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000] @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600) @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616) @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230) @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086) @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154) @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608) @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511) @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479) @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73) @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669) @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)
            Hide
            Zhen Qian added a comment -

            Matthew:

            Can you attach the code for this line?

            org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)

            Show
            Zhen Qian added a comment - Matthew: Can you attach the code for this line? org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
            Hide
            Matthew Buckett added a comment -

            So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the
            iterator for the treeset has become broken through concurrent access.

            // This is where expandedItems gets set.
            SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS);

            // Here is the itteration
            SortedSet newSet = new TreeSet();
            Iterator l = expandedItems.iterator();
            while (l.hasNext ())
            {
            String id = (String) l.next();

            if (id.startsWith(collectionId))
            {
            }
            else
            {
            // newSet.put(id,collection);
            newSet.add(id); // 5600
            }

            Show
            Matthew Buckett added a comment - So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the iterator for the treeset has become broken through concurrent access. // This is where expandedItems gets set. SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS); // Here is the itteration SortedSet newSet = new TreeSet(); Iterator l = expandedItems.iterator(); while (l.hasNext ()) { String id = (String) l.next(); if (id.startsWith(collectionId)) { } else { // newSet.put(id,collection); newSet.add(id); // 5600 }
            Hide
            Matthew Buckett added a comment -

            We're still seeing this from the content tool so I've opened a new issue SAK-23514

            Show
            Matthew Buckett added a comment - We're still seeing this from the content tool so I've opened a new issue SAK-23514
            Hide
            Zhen Qian added a comment -

            We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java:

            java.lang.Thread.State: RUNNABLE
            at java.util.HashMap.hash(HashMap.java:351)
            at java.util.HashMap.removeEntryForKey(HashMap.java:645)
            at java.util.HashMap.remove(HashMap.java:635)
            at java.util.Collections$SynchronizedMap.remove(Collections.java:2044)

            • locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap)
              at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694)
              ....

            java.lang.Thread.State: RUNNABLE
            at org.apache.log4j.Category.getEffectiveLevel(Category.java:442)
            at org.apache.log4j.Category.log(Category.java:855)
            at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)
            at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424)
            at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695)
            .....

            I will open a new jira for this.

            Show
            Zhen Qian added a comment - We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java: java.lang.Thread.State: RUNNABLE at java.util.HashMap.hash(HashMap.java:351) at java.util.HashMap.removeEntryForKey(HashMap.java:645) at java.util.HashMap.remove(HashMap.java:635) at java.util.Collections$SynchronizedMap.remove(Collections.java:2044) locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694) .... java.lang.Thread.State: RUNNABLE at org.apache.log4j.Category.getEffectiveLevel(Category.java:442) at org.apache.log4j.Category.log(Category.java:855) at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177) at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695) ..... I will open a new jira for this.

              People

              • Assignee:
                Zhen Qian
                Reporter:
                Stephen Marquard
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development