Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-11098

Stuck threads use excessive CPU

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: RESOLVED
    • Priority: Critical
    • Resolution: Incorporated
    • Affects Version/s: 2.4.0, 2.4.1, 2.5.0, 2.5.2, 2.5.3
    • Fix Version/s: 2.9.0
    • Component/s: JSF
    • Labels:
      None

      Description

      This is similar to SAK-8932. As it's not clear whether this is a generic JSF issue or tool-related, I'm creating a separate JIRA for each tool that it shows up in.

      A running app server shows 2 threads that get into some sort of loop, using up lots of CPU time (affecting response time for other threads) and producing excessive GC activity (several GCs per second).

      Stack traces of the 2 threads concerned are below (from kill -3 and a ps to show thread execution time, see SAK-8932 for details).

      This is a 64bit JVM, jdk1.5.0_12 running on Linux. Full running process looks like:

      /usr/java/jdk1.5.0_12/jre/bin/java -server -Xmx1500m -XX:MaxPermSize=512m -XX:+UseAdaptiveSizePolicy -XX:GCTimeRatio=49 -XX:ThreadStackSize=256 -Dhttp.proxyHost=cache.uct.ac.za -Dhttp.proxyPort=8080 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dsakai.security=/usr/local/sakaiconfig/a -Duser.language=en -Duser.region=ZA -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/sakai/conf/logging.properties -Djava.endorsed.dirs=/usr/local/sakai/common/endorsed -classpath :/usr/local/sakai/bin/bootstrap.jar:/usr/local/sakai/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/sakai -Dcatalina.home=/usr/local/sakai -Djava.io.tmpdir=/usr/local/sakai/temp org.apache.catalina.startup.Bootstrap start

      "TP-Processor135" daemon prio=1 tid=0x0000002b23e46b50 nid=0x4c8a runnable [0x000000004474a000..0x000000004474cc90]
      at com.sun.faces.context.FacesContextImpl.getMessages(FacesContextImpl.java:201)
      at javax.faces.component.UIData.keepSaved(UIData.java:1016)
      at javax.faces.component.UIData.processDecodes(UIData.java:733)
      at javax.faces.component.UIForm.processDecodes(UIForm.java:144)
      at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:880)
      at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:306)
      at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:79)
      at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
      at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:90)
      at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:403)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:230)
      at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:364)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:441)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1099)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:163)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:86)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:63)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:942)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

      "TP-Processor255" daemon prio=1 tid=0x0000002b1d7e3680 nid=0x4d9e runnable [0x000000004606e000..0x000000004606fe90]
      at com.sun.faces.context.FacesContextImpl.getMessages(FacesContextImpl.java:201)
      at javax.faces.component.UIData.keepSaved(UIData.java:1016)
      at javax.faces.component.UIData.processDecodes(UIData.java:733)
      at javax.faces.component.UIForm.processDecodes(UIForm.java:144)
      at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:880)
      at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:306)
      at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:79)
      at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
      at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:90)
      at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:403)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:230)
      at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
      at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:364)
      at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
      at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:441)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1099)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:163)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:86)
      at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:63)
      at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:942)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

        Gliffy Diagrams

          Zeplin

            Attachments

            1. catalina.1
              382 kB
            2. screenshot-1.jpg
              screenshot-1.jpg
              59 kB
            3. slowthreads.txt
              4 kB
            4. stuck-analysis.xls
              232 kB
            5. tq-long-running-threads-uct-30oct07.txt
              24 kB
            6. UCBBSP-1506_2_threads_031208.txt
              21 kB

              Issue Links

                Activity

                  People

                  Assignee:
                  maintenanceteam Core Team
                  Reporter:
                  smarquard Stephen Marquard
                  Votes:
                  2 Vote for this issue
                  Watchers:
                  13 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved:

                      Git Integration