Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-11098

Stuck threads use excessive CPU

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: RESOLVED
    • Priority: Critical
    • Resolution: Incorporated
    • Affects Version/s: 2.4.0, 2.4.1, 2.5.0, 2.5.2, 2.5.3
    • Fix Version/s: 2.9.0
    • Component/s: JSF
    • Labels:
      None

      Description

      This is similar to SAK-8932. As it's not clear whether this is a generic JSF issue or tool-related, I'm creating a separate JIRA for each tool that it shows up in.

      A running app server shows 2 threads that get into some sort of loop, using up lots of CPU time (affecting response time for other threads) and producing excessive GC activity (several GCs per second).

      Stack traces of the 2 threads concerned are below (from kill -3 and a ps to show thread execution time, see SAK-8932 for details).

      This is a 64bit JVM, jdk1.5.0_12 running on Linux. Full running process looks like:

      /usr/java/jdk1.5.0_12/jre/bin/java -server -Xmx1500m -XX:MaxPermSize=512m -XX:+UseAdaptiveSizePolicy -XX:GCTimeRatio=49 -XX:ThreadStackSize=256 -Dhttp.proxyHost=cache.uct.ac.za -Dhttp.proxyPort=8080 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dsakai.security=/usr/local/sakaiconfig/a -Duser.language=en -Duser.region=ZA -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/sakai/conf/logging.properties -Djava.endorsed.dirs=/usr/local/sakai/common/endorsed -classpath :/usr/local/sakai/bin/bootstrap.jar:/usr/local/sakai/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/sakai -Dcatalina.home=/usr/local/sakai -Djava.io.tmpdir=/usr/local/sakai/temp org.apache.catalina.startup.Bootstrap start

      "TP-Processor135" daemon prio=1 tid=0x0000002b23e46b50 nid=0x4c8a runnable [0x000000004474a000..0x000000004474cc90]
      at com.sun.faces.context.FacesContextImpl.getMessages(FacesContextImpl.java:201)
      at javax.faces.component.UIData.keepSaved(UIData.java:1016)
      at javax.faces.component.UIData.processDecodes(UIData.java:733)
      at javax.faces.component.UIForm.processDecodes(UIForm.java:144)
      at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:880)
      at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:306)
      at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:79)
      at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
      at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:90)
      at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:403)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:230)
      at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:364)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:441)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1099)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:163)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:86)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:63)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:942)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

      "TP-Processor255" daemon prio=1 tid=0x0000002b1d7e3680 nid=0x4d9e runnable [0x000000004606e000..0x000000004606fe90]
      at com.sun.faces.context.FacesContextImpl.getMessages(FacesContextImpl.java:201)
      at javax.faces.component.UIData.keepSaved(UIData.java:1016)
      at javax.faces.component.UIData.processDecodes(UIData.java:733)
      at javax.faces.component.UIForm.processDecodes(UIForm.java:144)
      at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:880)
      at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:306)
      at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:79)
      at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
      at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:90)
      at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:403)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:230)
      at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:364)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:441)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1099)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:163)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:86)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:63)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:942)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

        Gliffy Diagrams

          Attachments

          1. catalina.1
            382 kB
          2. screenshot-1.jpg
            screenshot-1.jpg
            59 kB
          3. slowthreads.txt
            4 kB
          4. stuck-analysis.xls
            232 kB
          5. tq-long-running-threads-uct-30oct07.txt
            24 kB
          6. UCBBSP-1506_2_threads_031208.txt
            21 kB

            Issue Links

              Activity

                People

                • Assignee:
                  maintenanceteam Core Team
                  Reporter:
                  smarquard Stephen Marquard
                • Votes:
                  2 Vote for this issue
                  Watchers:
                  13 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Git Source Code