Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.5.2, 2.5.3, 2.6.0, 2.7.1
    • Fix Version/s: 2.9.0, 10.0-qa01
    • Component/s: Content
    • Labels:
      None
    • 2.9 Status:
      Resolved
    • Previous Issue Keys:

      Description

      Two request threads from Resources have been running for a very long time (86796s and 86794s respectively), and appear to be in an infinite loop. User action which produced this is unknown.

      "TP-Processor172" daemon prio=1 tid=0x00002aaaacc0fcd0 nid=0x7738 runnable [0x0000000052854000..0x0000000052855da0]
              at java.util.TreeMap.successor(TreeMap.java:1214)
              at java.util.TreeMap.access$400(TreeMap.java:81)
              at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1033)
              at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
              at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6548)
              at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
              at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
              at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
              at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

      "TP-Processor223" daemon prio=1 tid=0x00002aaaad6f7090 nid=0x3398 runnable [0x00000000488b5000..0x00000000488b6e20]
              at java.lang.String.startsWith(String.java:1260)
              at java.lang.String.startsWith(String.java:1281)
              at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6549)
              at sun.reflect.GeneratedMethodAccessor2209.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:591)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:507)
              at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:221)
              at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1005)
              at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:151)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
              at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
              at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)

        Issue Links

          Activity

          Hide
          Matthew Buckett added a comment -
          I think we've hit this in 2.8.x as well with a stack trace of:

          @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000]
          @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE
          @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
          @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616)
          @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679)
          @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555)
          @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230)
          @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086)
          @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154)
          @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
          @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
          @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608)
          @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
          @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
          @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
          @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
          @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
          @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479)
          @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137)
          @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
          @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
          @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270)
          @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
          @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
          @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
          @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669)
          @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
          @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
          @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
          @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
          @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
          @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
          @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
          @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
          @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
          @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
          @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
          @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
          @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
          @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
          @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)
          Show
          Matthew Buckett added a comment - I think we've hit this in 2.8.x as well with a stack trace of: @40000000510939051b15672c "TP-Processor22" daemon prio=10 tid=0x00007f08a9ecb800 nid=0x5fc0 runnable [0x00007f08961e5000] @40000000510939051b156b14 java.lang.Thread.State: RUNNABLE @40000000510939051b15866c at org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600) @40000000510939051b15960c at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) @40000000510939051b15a1c4 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) @40000000510939051b15b934 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) @40000000510939051b15c8d4 at java.lang.reflect.Method.invoke(Method.java:616) @40000000510939051b15d48c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.actionDispatch(VelocityPortletPaneledAction.java:679) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.VelocityPortletPaneledAction.processAction(VelocityPortletPaneledAction.java:555) @40000000510939051b15f7b4 at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:230) @40000000510939051b15fb9c at org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1086) @40000000510939051b160754 at org.sakaiproject.cheftool.ToolServlet.doPost(ToolServlet.java:154) @40000000510939051b1616f4 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b1622ac at org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56) @40000000510939051b1622ac at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b163e04 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b164da4 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b166514 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:608) @40000000510939051b1670cc at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b16806c at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b168c24 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659) @40000000510939051b1697dc at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359) @40000000510939051b16a77c at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) @40000000510939051b16b71c at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511) @40000000510939051b16c2d4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1479) @40000000510939051b16ce8c at org.sakaiproject.portal.charon.handlers.HierarchyToolHandler.doTool(HierarchyToolHandler.java:137) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96) @40000000510939051b16de2c at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73) @40000000510939051b16e9e4 at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1270) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) @40000000510939051b16f984 at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) @40000000510939051b17053c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) @40000000510939051b1714dc at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b172094 at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:669) @40000000510939051b172c4c at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) @40000000510939051b173034 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) @40000000510939051b1747a4 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) @40000000510939051b17535c at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) @40000000510939051b1762fc at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) @40000000510939051b176eb4 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) @40000000510939051b177a6c at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581) @40000000510939051b178624 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) @40000000510939051b1791dc at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) @40000000510939051b17b504 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) @40000000510939051b17b8ec at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) @40000000510939051b17b8ec at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) @40000000510939051b17c4a4 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) @40000000510939051b17c88c at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) @40000000510939051b17d444 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) @40000000510939051b17e3e4 at java.lang.Thread.run(Thread.java:636)
          Hide
          Zhen Qian added a comment -
          Matthew:

          Can you attach the code for this line?

          org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)

          Show
          Zhen Qian added a comment - Matthew: Can you attach the code for this line? org.sakaiproject.content.tool.ResourcesAction.doCollapse_collection(ResourcesAction.java:5600)
          Hide
          Matthew Buckett added a comment -
          So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the
          iterator for the treeset has become broken through concurrent access.

          // This is where expandedItems gets set.
          SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS);


          // Here is the itteration
          SortedSet newSet = new TreeSet();
          Iterator l = expandedItems.iterator();
          while (l.hasNext ())
          {
          String id = (String) l.next();


          if (id.startsWith(collectionId))
          {
          }
          else
          {
          // newSet.put(id,collection);
          newSet.add(id); // 5600
          }
          Show
          Matthew Buckett added a comment - So it gets a set from the session state and then adds them to a another set. I suspect a threading issue where the iterator for the treeset has become broken through concurrent access. // This is where expandedItems gets set. SortedSet expandedItems = (SortedSet) state.getAttribute(STATE_EXPANDED_COLLECTIONS); // Here is the itteration SortedSet newSet = new TreeSet(); Iterator l = expandedItems.iterator(); while (l.hasNext ()) { String id = (String) l.next(); if (id.startsWith(collectionId)) { } else { // newSet.put(id,collection); newSet.add(id); // 5600 }
          Hide
          Matthew Buckett added a comment -
          We're still seeing this from the content tool so I've opened a new issue SAK-23514
          Show
          Matthew Buckett added a comment - We're still seeing this from the content tool so I've opened a new issue SAK-23514
          Hide
          Zhen Qian added a comment -
          We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java:

          java.lang.Thread.State: RUNNABLE
          at java.util.HashMap.hash(HashMap.java:351)
          at java.util.HashMap.removeEntryForKey(HashMap.java:645)
          at java.util.HashMap.remove(HashMap.java:635)
          at java.util.Collections$SynchronizedMap.remove(Collections.java:2044)
          - locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap)
          at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694)
          ....

             java.lang.Thread.State: RUNNABLE
          at org.apache.log4j.Category.getEffectiveLevel(Category.java:442)
          at org.apache.log4j.Category.log(Category.java:855)
          at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)
          at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424)
          at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695)
          .....

          I will open a new jira for this.


          Show
          Zhen Qian added a comment - We are running 2.9.2 code base now. Just had a CPU instance with the doNavigate call in SiteAction.java: java.lang.Thread.State: RUNNABLE at java.util.HashMap.hash(HashMap.java:351) at java.util.HashMap.removeEntryForKey(HashMap.java:645) at java.util.HashMap.remove(HashMap.java:635) at java.util.Collections$SynchronizedMap.remove(Collections.java:2044) - locked <0x00000005d30192e8> (a java.util.Collections$SynchronizedMap) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6694) ....    java.lang.Thread.State: RUNNABLE at org.apache.log4j.Category.getEffectiveLevel(Category.java:442) at org.apache.log4j.Category.log(Category.java:855) at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177) at org.sakaiproject.content.tool.ResourcesAction.removeObservingPattern(ResourcesAction.java:3424) at org.sakaiproject.content.tool.ResourcesAction.doNavigate(ResourcesAction.java:6695) ..... I will open a new jira for this.

            People

            • Assignee:
              Zhen Qian
              Reporter:
              Stephen Marquard
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: