Issue Details (XML | Word | Printable)

Key: SAK-11875
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Critical Critical
Assignee: Stephen Marquard
Reporter: Chris Maurer
Votes: 1
Watchers: 10
Operations

If you were logged in you would be able to see more operations.
Sakai

Chat messages aren't always delivered

Created: 09-Oct-2007 13:18   Updated: 29-Dec-2008 09:30
Component/s: Chat Room, Courier (SVN module)
Affects Version/s: 2.4.0, 2.4.1, 2.5.0, 2.5.2
Fix Version/s: 2.5.4, 2.5.x, 2.6.0-beta03

Time Tracking:
Not Specified

File Attachments: 1. Java Source File BasicCourierService.java (10 kB)
2. HTML File chat1000.html (150 kB)
3. File diff (8 kB)
4. File mktest.pl (0.9 kB)

Issue Links:
Incorporate
 
Relate
 

2.4.x Status: None
2.5.x Status: Resolved
2.6.x Status: None


 Description  « Hide
Sometimes, a message won't be delivered to a client in chat. Here's an error from the log:

WARN: (2007-10-09 14:09:14,268 http-8080-Processor24_org.sakaiproject.util.RequestFilter)
java.util.ConcurrentModificationException
        at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:449)
        at java.util.AbstractList$Itr.next(AbstractList.java:420)
        at org.sakaiproject.courier.impl.BasicCourierService.getDeliveries(BasicCourierService.java:208)
        at org.sakaiproject.courier.cover.CourierService.getDeliveries(CourierService.java:69)
        at org.sakaiproject.courier.tool.CourierTool.doGet(CourierTool.java:102)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:592)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
        at java.lang.Thread.run(Thread.java:613)



 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Chris Maurer added a comment - 10-Oct-2007 12:05
r36652
Adding a synchronized block around the delivery iteration.

Chris Maurer added a comment - 10-Oct-2007 12:18
r36653
Added to 2.4.x

Megan May added a comment - 15-Oct-2007 05:48
Testing has shown this doesn't resolve the problem

Megan May added a comment - 13-Nov-2007 11:47
Assigning to xingtang since he's working through these.

Matthew Buckett added a comment - 29-Nov-2007 06:01
If this problem was re-opened because it was noticed on one of the QA servers then the reason for this could well be because it was never merged into the 2.5 branch (2.5 branch created 3rd Oct and fix was commited on the 10th Oct).

Doesn't someone just need to merge the fix onto the 2.5 branch?

Lance Speelmon added a comment - 13-Dec-2007 21:12
The current thinking on this issue is that it may be the EventServive impl that is the root of the problem. We are planning to test a JMS based impl to see if the problem goes way. We will update the group as soon as we know more. L

Matthew Rubinstein added a comment - 20-Dec-2007 11:00 - edited
Testing this fix on our test instance when compared to production shows that the fix did not resolve this issue.

This was tested at U of M on our 2.4.x sakai test instance which included the fix.

Messages will always sync up if a participant submits test, but if a participant idles they will not reliably receive updates for up to 20 seconds in some cases.

This became an issue with instructors who use the tool for required chats where participation is assessed. Students are unable to follow the conversation in real time due to this issue.

Matthew Buckett added a comment - 21-Dec-2007 04:45
Is there a reliable way to reproduce the bug?

Michael Lockett added a comment - 14-Jan-2008 10:48
Messages in the tool are able to be sent and recieved as they should be.

Peter A. Knoop added a comment - 14-Jan-2008 17:15
Michale, why do you think this issue is fixed? There has been no work on it or code check-ins since other folks reported they were experiencing the problem?

xingtang added a comment - 15-Jan-2008 15:38 - edited
Hi, Matthew Buckett , are you sure you have solved this problem?

Cause i have done a long time on this bug, but have not got progress on it. I use JMeter to test the bug, it appears some messages lost.


Matthew Buckett added a comment - 16-Jan-2008 02:12 - edited
I think you are confusing people xingtang. Micheal Lockett closed the issue saying it was fixed.

I've checked again and the fix that Chris Maurer made still isn't in the 2.5 branch! BaseCourierService (around line 206) is missing the synchronization on the 2.5 branch.

If people are seeing this issue still on the 2.4 branch or trunk then the fix Chris made doesn't work.

If there are some JMeter scripts to reproduce this issue, are they available somewhere so that others can check.

I still don't think this issue should be fixed as I believe it still exists on the 2.5 branch.

Stephen Marquard added a comment - 16-Jan-2008 02:21
Reopening this as there's doubt that it's fixed.

xintang, can you add the JMeter test scripts you're using to this issue, and describe how to demonstrate the bug ?

Matthew Buckett added a comment - 16-Jan-2008 03:06
Just a side note the indentation on this fix is spaces in a file that uses tabs everywhere else. Better to catch this now and associate it with the original ticket. If someone could convert the spaces to tabs on trunk that would be nice.

xingtang added a comment - 24-Jan-2008 14:47
I have simulated 4 users to chat in the Chatroom by JMeter. There are some messages disappeared according to following log.

Here is the results:


# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 1
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 3
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 2
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 10
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 7
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 4
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 5
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 6
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 8
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 9
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 11
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 12
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 15
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 13
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 14
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 20
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 19
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 16
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 17
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 18
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 21
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 22
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 23
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 25
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 24
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 26
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 29
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 27
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 30
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 28
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 32
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 31
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 37
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 36
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 38
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 35
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 33
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 34
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 39
# test Delete this message (Jan 24, 2008 4:38 PM EST) test - 40
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 42
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 41
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 43
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 45
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 44
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 47
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 49
# teacher Delete this message (Jan 24, 2008 4:38 PM EST) teacher - 46
# test1 Delete this message (Jan 24, 2008 4:38 PM EST) test1 - 48
# student Delete this message (Jan 24, 2008 4:38 PM EST) student - 50
# teacher Delete this message (Jan 24, 2008 4:39 PM EST) teacher - 52
# test Delete this message (Jan 24, 2008 4:39 PM EST) test - 51
# test Delete this message (Jan 24, 2008 4:39 PM EST) test - 53
# test1 Delete this message (Jan 24, 2008 4:39 PM EST) test1 - 55
# test Delete this message (Jan 24, 2008 4:39 PM EST) test - 54
# student Delete this message (Jan 24, 2008 4:39 PM EST) student - 56
# student Delete this message (Jan 24, 2008 4:39 PM EST) student - 58
# teacher Delete this message (Jan 24, 2008 4:39 PM EST) teacher - 57
# test1 Delete this message (Jan 24, 2008 4:39 PM EST) test1 - 59
# teacher Delete this message (Jan 24, 2008 4:39 PM EST) teacher - 60
# test1 Delete this message (Jan 24, 2008 4:39 PM EST) test1 - 63
# student Delete this message (Jan 24, 2008 4:39 PM EST) student - 65
# test Delete this message (Jan 24, 2008 4:39 PM EST) test - 61
# teacher Delete this message (Jan 24, 2008 4:39 PM EST) teacher - 62
# student Delete this message (Jan 24, 2008 4:39 PM EST) student - 66
# test1 Delete this message (Jan 24, 2008 4:39 PM EST) test1 - 68
# test Delete this message (Jan 24, 2008 4:39 PM EST) test - 64



And the JMeter script is:


<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="1.8">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <stringProp name="TestPlan.comments"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <longProp name="ThreadGroup.start_time">1174401372000</longProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.num_threads">10</stringProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">1</stringProp>
        </elementProp>
        <longProp name="ThreadGroup.end_time">1174401372000</longProp>
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
      </ThreadGroup>
      <hashTree>
        <CookieManager guiclass="CookiePanel" testclass="CookieManager" testname="HTTP Cookie Manager" enabled="true">
          <collectionProp name="CookieManager.cookies"/>
          <boolProp name="CookieManager.clearEachIteration">false</boolProp>
        </CookieManager>
        <hashTree/>
        <Arguments guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
          <collectionProp name="Arguments.arguments">
            <elementProp name="server" elementType="Argument">
              <stringProp name="Argument.metadata">=</stringProp>
              <stringProp name="Argument.value">localhost</stringProp>
              <stringProp name="Argument.name">server</stringProp>
            </elementProp>
            <elementProp name="tool_id" elementType="Argument">
              <stringProp name="Argument.metadata">=</stringProp>
              <stringProp name="Argument.value">2d92fea7-7edf-4070-a824-284957b91f90</stringProp>
              <stringProp name="Argument.name">tool_id</stringProp>
            </elementProp>
            <elementProp name="courier_path" elementType="Argument">
              <stringProp name="Argument.metadata">=</stringProp>
              <stringProp name="Argument.value">/courier/f9605c79-3121-49a1-9e50-424576c7cdc2/2d92fea7-7edf-4070-a824-284957b91f90</stringProp>
              <stringProp name="Argument.name">courier_path</stringProp>
            </elementProp>
            <elementProp name="site_id" elementType="Argument">
              <stringProp name="Argument.name">site_id</stringProp>
              <stringProp name="Argument.value">5b94cab3-2057-4ab0-a994-5534c0366764</stringProp>
              <stringProp name="Argument.metadata">=</stringProp>
            </elementProp>
            <elementProp name="page_id" elementType="Argument">
              <stringProp name="Argument.name">page_id</stringProp>
              <stringProp name="Argument.value">508a6a39-936a-452f-8e8e-b74b233f26db</stringProp>
              <stringProp name="Argument.metadata">=</stringProp>
            </elementProp>
            <elementProp name="count" elementType="Argument">
              <stringProp name="Argument.name">count</stringProp>
              <stringProp name="Argument.value">1</stringProp>
              <stringProp name="Argument.metadata">=</stringProp>
            </elementProp>
          </collectionProp>
        </Arguments>
        <hashTree/>
        <UserParameters guiclass="UserParametersGui" testclass="UserParameters" testname="User Parameters" enabled="true">
          <collectionProp name="UserParameters.names">
            <stringProp name="-265713450">username</stringProp>
            <stringProp name="1216985755">password</stringProp>
          </collectionProp>
          <collectionProp name="UserParameters.thread_values">
            <collectionProp name="-435648223">
              <stringProp name="3556498">test</stringProp>
              <stringProp name="3556498">test</stringProp>
            </collectionProp>
            <collectionProp name="1941467329">
              <stringProp name="-1439577118">teacher</stringProp>
              <stringProp name="-1439577118">teacher</stringProp>
            </collectionProp>
            <collectionProp name="-1942206399">
              <stringProp name="-1879145925">student</stringProp>
              <stringProp name="-1879145925">student</stringProp>
            </collectionProp>
            <collectionProp name="-898666015">
              <stringProp name="110251487">test1</stringProp>
              <stringProp name="110251487">test1</stringProp>
            </collectionProp>
          </collectionProp>
          <boolProp name="UserParameters.per_iteration">true</boolProp>
          <stringProp name="TestPlan.comments">huxt
</stringProp>
        </UserParameters>
        <hashTree/>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="/portal/xlogin" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">localhost</stringProp>
          <stringProp name="HTTPSampler.port">8080</stringProp>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/portal/xlogin</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.mimetype"></stringProp>
          <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
          <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
          <stringProp name="HTTPSampler.monitor">false</stringProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Referer</stringProp>
                <stringProp name="Header.value">http://${server}:8080/portal/xlogin</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Host</stringProp>
                <stringProp name="Header.value">${server}:8080</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Content-Type</stringProp>
                <stringProp name="Header.value">application/x-www-form-urlencoded</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">300</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="/portal/xlogin" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments">
              <elementProp name="" elementType="HTTPArgument">
                <stringProp name="Argument.metadata">=</stringProp>
                <stringProp name="Argument.value">${username}</stringProp>
                <boolProp name="HTTPArgument.use_equals">true</boolProp>
                <stringProp name="Argument.name">eid</stringProp>
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
              </elementProp>
              <elementProp name="" elementType="HTTPArgument">
                <stringProp name="Argument.metadata">=</stringProp>
                <stringProp name="Argument.value">${password}</stringProp>
                <boolProp name="HTTPArgument.use_equals">true</boolProp>
                <stringProp name="Argument.name">pw</stringProp>
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
              </elementProp>
              <elementProp name="" elementType="HTTPArgument">
                <stringProp name="Argument.metadata">=</stringProp>
                <stringProp name="Argument.value">Login</stringProp>
                <boolProp name="HTTPArgument.use_equals">true</boolProp>
                <stringProp name="Argument.name">submit</stringProp>
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="HTTPSampler.domain">localhost</stringProp>
          <stringProp name="HTTPSampler.port">8080</stringProp>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/portal/xlogin</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.mimetype"></stringProp>
          <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
          <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
          <stringProp name="HTTPSampler.monitor">false</stringProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Referer</stringProp>
                <stringProp name="Header.value">http://${server}:8080/portal/xlogin</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Host</stringProp>
                <stringProp name="Header.value">${server}:8080</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Content-Type</stringProp>
                <stringProp name="Header.value">application/x-www-form-urlencoded</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">300</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="Room" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">${server}</stringProp>
          <stringProp name="HTTPSampler.port">8080</stringProp>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/portal/tool/${tool_id}/main</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">false</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.mimetype"></stringProp>
          <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
          <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
          <stringProp name="HTTPSampler.monitor">false</stringProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Referer</stringProp>
                <stringProp name="Header.value">http://${server}:8080/portal/site/${site_id}/page/${page_id}</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Host</stringProp>
                <stringProp name="Header.value">${server}:8080</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">300</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
        <LoopController guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">true</boolProp>
          <stringProp name="LoopController.loops">10</stringProp>
        </LoopController>
        <hashTree>
          <CounterConfig guiclass="CounterConfigGui" testclass="CounterConfig" testname="Counter" enabled="true">
            <stringProp name="CounterConfig.start">1</stringProp>
            <stringProp name="CounterConfig.end"></stringProp>
            <stringProp name="CounterConfig.incr">1</stringProp>
            <stringProp name="CounterConfig.name">Counter</stringProp>
            <stringProp name="CounterConfig.format"></stringProp>
            <boolProp name="CounterConfig.per_user">false</boolProp>
          </CounterConfig>
          <hashTree/>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="Room" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
              <collectionProp name="Arguments.arguments"/>
            </elementProp>
            <stringProp name="HTTPSampler.domain">${server}</stringProp>
            <stringProp name="HTTPSampler.port">8080</stringProp>
            <stringProp name="HTTPSampler.protocol">http</stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/portal/tool/${tool_id}/room</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">false</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.monitor">false</stringProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree>
            <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
              <collectionProp name="HeaderManager.headers">
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Encoding</stringProp>
                  <stringProp name="Header.value">gzip,deflate</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Referer</stringProp>
                  <stringProp name="Header.value">http://${server}:8080/portal/site/${site_id}/page/${page_id}</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Language</stringProp>
                  <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Host</stringProp>
                  <stringProp name="Header.value">${server}:8080</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Charset</stringProp>
                  <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">User-Agent</stringProp>
                  <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept</stringProp>
                  <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Keep-Alive</stringProp>
                  <stringProp name="Header.value">300</stringProp>
                </elementProp>
              </collectionProp>
            </HeaderManager>
            <hashTree/>
          </hashTree>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="Courier" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
              <collectionProp name="Arguments.arguments">
                <elementProp name="" elementType="HTTPArgument">
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.value">true</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                  <stringProp name="Argument.name">auto</stringProp>
                  <boolProp name="HTTPArgument.always_encode">false</boolProp>
                </elementProp>
              </collectionProp>
            </elementProp>
            <stringProp name="HTTPSampler.domain">${server}</stringProp>
            <stringProp name="HTTPSampler.port">8080</stringProp>
            <stringProp name="HTTPSampler.protocol">http</stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">${courier_path}</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">false</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.monitor">false</stringProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree>
            <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
              <collectionProp name="HeaderManager.headers">
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Encoding</stringProp>
                  <stringProp name="Header.value">gzip,deflate</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Referer</stringProp>
                  <stringProp name="Header.value">http://${server}:8080/portal/tool/${tool_id}/room</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Language</stringProp>
                  <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Host</stringProp>
                  <stringProp name="Header.value">${server}:8080</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Charset</stringProp>
                  <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">User-Agent</stringProp>
                  <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept</stringProp>
                  <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Keep-Alive</stringProp>
                  <stringProp name="Header.value">300</stringProp>
                </elementProp>
              </collectionProp>
            </HeaderManager>
            <hashTree/>
          </hashTree>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="Post Form Post" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
              <collectionProp name="Arguments.arguments"/>
            </elementProp>
            <stringProp name="HTTPSampler.domain">${server}</stringProp>
            <stringProp name="HTTPSampler.port">8080</stringProp>
            <stringProp name="HTTPSampler.protocol">http</stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/portal/tool/${tool_id}/roomControl</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">false</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.monitor">false</stringProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree>
            <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
              <collectionProp name="HeaderManager.headers">
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Encoding</stringProp>
                  <stringProp name="Header.value">gzip,deflate</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Referer</stringProp>
                  <stringProp name="Header.value">http://${server}:8080/portal/tool/${tool_id}/roomControl</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Language</stringProp>
                  <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Host</stringProp>
                  <stringProp name="Header.value">${server}:8080</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Charset</stringProp>
                  <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">User-Agent</stringProp>
                  <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Content-Type</stringProp>
                  <stringProp name="Header.value">application/x-www-form-urlencoded</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept</stringProp>
                  <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Keep-Alive</stringProp>
                  <stringProp name="Header.value">300</stringProp>
                </elementProp>
              </collectionProp>
            </HeaderManager>
            <hashTree/>
          </hashTree>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="Post Form Post" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
              <collectionProp name="Arguments.arguments">
                <elementProp name="" elementType="HTTPArgument">
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.value">some+value</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                  <stringProp name="Argument.name">_idOfAction</stringProp>
                  <boolProp name="HTTPArgument.always_encode">false</boolProp>
                </elementProp>
                <elementProp name="" elementType="HTTPArgument">
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.value">${username} - ${Counter}</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                  <stringProp name="Argument.name">mainForm%3Amessage</stringProp>
                  <boolProp name="HTTPArgument.always_encode">false</boolProp>
                </elementProp>
                <elementProp name="" elementType="HTTPArgument">
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.value">Add+message</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                  <stringProp name="Argument.name">mainForm%3Asubmit</stringProp>
                  <boolProp name="HTTPArgument.always_encode">false</boolProp>
                </elementProp>
                <elementProp name="" elementType="HTTPArgument">
                  <stringProp name="Argument.metadata">=</stringProp>
                  <stringProp name="Argument.value">mainForm</stringProp>
                  <boolProp name="HTTPArgument.use_equals">true</boolProp>
                  <stringProp name="Argument.name">mainForm</stringProp>
                  <boolProp name="HTTPArgument.always_encode">false</boolProp>
                </elementProp>
              </collectionProp>
            </elementProp>
            <stringProp name="HTTPSampler.domain">${server}</stringProp>
            <stringProp name="HTTPSampler.port">8080</stringProp>
            <stringProp name="HTTPSampler.protocol">http</stringProp>
            <stringProp name="HTTPSampler.method">POST</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/portal/tool/${tool_id}/roomControl</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.monitor">false</stringProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree>
            <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
              <collectionProp name="HeaderManager.headers">
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Encoding</stringProp>
                  <stringProp name="Header.value">gzip,deflate</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Referer</stringProp>
                  <stringProp name="Header.value">http://${server}:8080/portal/tool/${tool_id}/roomControl</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Language</stringProp>
                  <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Host</stringProp>
                  <stringProp name="Header.value">${server}:8080</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Charset</stringProp>
                  <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">User-Agent</stringProp>
                  <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Content-Type</stringProp>
                  <stringProp name="Header.value">application/x-www-form-urlencoded</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept</stringProp>
                  <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Keep-Alive</stringProp>
                  <stringProp name="Header.value">300</stringProp>
                </elementProp>
              </collectionProp>
            </HeaderManager>
            <hashTree/>
          </hashTree>
          <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="Room" enabled="true">
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
              <collectionProp name="Arguments.arguments"/>
            </elementProp>
            <stringProp name="HTTPSampler.domain">${server}</stringProp>
            <stringProp name="HTTPSampler.port">8080</stringProp>
            <stringProp name="HTTPSampler.protocol">http</stringProp>
            <stringProp name="HTTPSampler.method">GET</stringProp>
            <stringProp name="HTTPSampler.contentEncoding"></stringProp>
            <stringProp name="HTTPSampler.path">/portal/tool/${tool_id}/room</stringProp>
            <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
            <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
            <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
            <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
            <stringProp name="HTTPSampler.mimetype"></stringProp>
            <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
            <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
            <stringProp name="HTTPSampler.monitor">false</stringProp>
            <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
          </HTTPSampler>
          <hashTree>
            <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
              <collectionProp name="HeaderManager.headers">
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Encoding</stringProp>
                  <stringProp name="Header.value">gzip,deflate</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Referer</stringProp>
                  <stringProp name="Header.value">http://${server}:8080/portal/site/${site_id}/page/${page_id}</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Language</stringProp>
                  <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Host</stringProp>
                  <stringProp name="Header.value">${server}:8080</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept-Charset</stringProp>
                  <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">User-Agent</stringProp>
                  <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Accept</stringProp>
                  <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
                </elementProp>
                <elementProp name="" elementType="Header">
                  <stringProp name="Header.name">Keep-Alive</stringProp>
                  <stringProp name="Header.value">300</stringProp>
                </elementProp>
              </collectionProp>
            </HeaderManager>
            <hashTree/>
          </hashTree>
        </hashTree>
        <ResultCollector guiclass="ViewResultsFullVisualizer" testclass="ResultCollector" testname="View Results Tree" enabled="true">
          <boolProp name="ResultCollector.error_logging">false</boolProp>
          <objProp>
            <value class="SampleSaveConfiguration">
              <time>true</time>
              <latency>true</latency>
              <timestamp>true</timestamp>
              <success>true</success>
              <label>true</label>
              <code>true</code>
              <message>true</message>
              <threadName>true</threadName>
              <dataType>true</dataType>
              <encoding>false</encoding>
              <assertions>true</assertions>
              <subresults>true</subresults>
              <responseData>false</responseData>
              <samplerData>false</samplerData>
              <xml>false</xml>
              <fieldNames>false</fieldNames>
              <responseHeaders>false</responseHeaders>
              <requestHeaders>false</requestHeaders>
              <responseDataOnError>false</responseDataOnError>
              <saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
              <assertionsResultsToSave>0</assertionsResultsToSave>
            </value>
            <name>saveConfig</name>
          </objProp>
          <stringProp name="filename"></stringProp>
        </ResultCollector>
        <hashTree/>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="/portal/logout" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">localhost</stringProp>
          <stringProp name="HTTPSampler.port">8080</stringProp>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/portal/logout</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.mimetype"></stringProp>
          <stringProp name="HTTPSampler.FILE_NAME"></stringProp>
          <stringProp name="HTTPSampler.FILE_FIELD"></stringProp>
          <stringProp name="HTTPSampler.monitor">false</stringProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="Browser-derived headers" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Referer</stringProp>
                <stringProp name="Header.value">http://${server}:8080/portal/logout</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Host</stringProp>
                <stringProp name="Header.value">${server}:8080</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Content-Type</stringProp>
                <stringProp name="Header.value">application/x-www-form-urlencoded</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5</stringProp>
              </elementProp>
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">300</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>


xingtang added a comment - 28-Jan-2008 14:20
    * t1f t1l (Jan 28, 2008 4:12 PM EST) this is xingtang
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 4
    * test (Jan 28, 2008 4:15 PM EST) test - 10
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 9
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 6
    * test (Jan 28, 2008 4:15 PM EST) test - 8
    * student (Jan 28, 2008 4:15 PM EST) student - 1
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 7
    * test (Jan 28, 2008 4:15 PM EST) test - 5
    * student (Jan 28, 2008 4:15 PM EST) student - 2
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 3
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 13
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 11
    * test (Jan 28, 2008 4:15 PM EST) test - 14
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 12
    * test (Jan 28, 2008 4:15 PM EST) test - 15
    * test (Jan 28, 2008 4:15 PM EST) test - 16
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 17
    * student (Jan 28, 2008 4:15 PM EST) student - 18
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 19
    * student (Jan 28, 2008 4:15 PM EST) student - 20
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 21
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 24
    * test (Jan 28, 2008 4:15 PM EST) test - 23
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 22
    * test (Jan 28, 2008 4:15 PM EST) test - 25
    * test (Jan 28, 2008 4:15 PM EST) test - 27
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 26
    * student (Jan 28, 2008 4:15 PM EST) student - 28
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 29
    * student (Jan 28, 2008 4:15 PM EST) student - 30
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 31
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 33
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 32
    * test (Jan 28, 2008 4:15 PM EST) test - 34
    * test (Jan 28, 2008 4:15 PM EST) test - 35
    * student (Jan 28, 2008 4:15 PM EST) student - 37
    * student (Jan 28, 2008 4:15 PM EST) student - 40
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 38
    * test (Jan 28, 2008 4:15 PM EST) test - 36
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 39
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 41
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 43
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 42
    * student (Jan 28, 2008 4:15 PM EST) student - 46
    * student (Jan 28, 2008 4:15 PM EST) student - 48
    * test (Jan 28, 2008 4:15 PM EST) test - 45
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 47
    * test (Jan 28, 2008 4:15 PM EST) test - 44
    * test (Jan 28, 2008 4:15 PM EST) test - 49
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 50
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 51
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 53
    * teacher (Jan 28, 2008 4:15 PM EST) teacher - 52
    * student (Jan 28, 2008 4:15 PM EST) student - 54
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 55
    * student (Jan 28, 2008 4:15 PM EST) student - 57
    * test1 (Jan 28, 2008 4:15 PM EST) test1 - 60
    * test (Jan 28, 2008 4:15 PM EST) test - 56
    * test (Jan 28, 2008 4:15 PM EST) test - 58
    * test (Jan 28, 2008 4:16 PM EST) test - 59
    * test1 (Jan 28, 2008 4:16 PM EST) test1 - 62
    * student (Jan 28, 2008 4:16 PM EST) student - 61
    * student (Jan 28, 2008 4:16 PM EST) student - 64
    * test1 (Jan 28, 2008 4:16 PM EST) test1 - 66
    * test1 (Jan 28, 2008 4:16 PM EST) test1 - 69
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 63
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 67
    * student (Jan 28, 2008 4:16 PM EST) student - 70
    * student (Jan 28, 2008 4:16 PM EST) student - 71
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 65
    * test1 (Jan 28, 2008 4:16 PM EST) test1 - 74
    * test (Jan 28, 2008 4:16 PM EST) test - 68
    * test (Jan 28, 2008 4:16 PM EST) test - 73
    * student (Jan 28, 2008 4:16 PM EST) student - 76
    * student (Jan 28, 2008 4:16 PM EST) student - 77
    * test1 (Jan 28, 2008 4:16 PM EST) test1 - 75
    * test (Jan 28, 2008 4:16 PM EST) test - 72
    * test1 (Jan 28, 2008 4:16 PM EST) test1 - 78
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 79
    * student (Jan 28, 2008 4:16 PM EST) student - 81
    * student (Jan 28, 2008 4:16 PM EST) student - 82
    * test1 (Jan 28, 2008 4:16 PM EST) test1 - 85
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 80
    * test (Jan 28, 2008 4:16 PM EST) test - 84
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 83
    * test (Jan 28, 2008 4:16 PM EST) test - 86
    * test (Jan 28, 2008 4:16 PM EST) test - 87
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 88
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 89
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 90
    * test (Jan 28, 2008 4:16 PM EST) test - 91
    * test (Jan 28, 2008 4:16 PM EST) test - 92
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 94
    * test (Jan 28, 2008 4:16 PM EST) test - 93
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 95
    * test (Jan 28, 2008 4:16 PM EST) test - 98
    * test (Jan 28, 2008 4:16 PM EST) test - 97
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 96
    * test (Jan 28, 2008 4:16 PM EST) test - 100
    * teacher (Jan 28, 2008 4:16 PM EST) teacher - 99

xingtang added a comment - 31-Jan-2008 09:44
Hi Lance,
I have test this bug using JMeter. But I cannot reproduce this problem. It seems every message has been delivered and received successfully.

By the way, I have replaced the old event system using ActiveMQ system. The test result using JMeter appears that it runs good.

Should I test in on more machines?

xingtang added a comment - 31-Jan-2008 12:45
I can reproduce this problem now.

You need open at least two clients (I use firefox and safari on mac os x) at the same time by different accounts.

Then, you can send message from one client and watch on the other client.

It seems the messages have been sent successfully but do not show correctly on the second client. When you refresh the second client(click refresh button). The lost messages will come out because they have been stored in database.

xingtang added a comment - 04-Feb-2008 09:23
I got the reason why messages have not been delivered successfully.

Actually, messages have been saved to database successfully. But they are not displayed successfully on the clients(Firefox, IE, Safari etc).

These code is the wrong logic:

<script type="text/javascript" language="JavaScript">
updateTime = 1000;
updateUrl = "<h:outputText value="#{ChatTool.courierString}" />";

scheduleUpdate();
/*checkForUpdate();*/

</script>


That means, we need using AJAX to construct the client site elements.

The Ajax has a auto refresh logic which can provide the logic we needed.


xingtang added a comment - 13-Feb-2008 13:57 - edited
I have fixed this problem.

Where is the problem?

cluster-impl-->org.sakairoject.cluster.impl-->SakaiClusterService-->Maintenance()->Run()


You need forbidden the thread to sleep like this: //Thread.sleep(m_refresh * 1000L);


When the thread sleep, some messages will disappear.

I have test is using JMeter, It runs good now. But I need Lance to approve this fix. There could be better way that we can fix this bug for example we could replace the cluster service.



========
// cycle every REFRESH seconds
if (!m_maintenanceCheckerStop)
{
try
{
//Thread.sleep(m_refresh * 1000L);
}
catch (Exception ignore)
{
}
}

Stephen Marquard added a comment - 14-Feb-2008 00:43
xintang,

That Thread.sleep is in a loop which just checks for expired servers in the cluster. It's hard to see how it could be affecting chat. Do you have an analysis of why it works like this?

You might also consider that removing the sleep increases the cpu load of the app server which could have other side-effects that explain what you see (especially in timing-sensitive concurrency issues).

xingtang added a comment - 14-Feb-2008 08:04
Dear Stephen Marquard ,


Based on my test, the cluser service provide event system for message broadcasting. When the thread go to sleep, the event cannot be broadcast successfully. When it wake up, the event can be send and receive successfully. I have test it using JMeter.

I agree with your point that if we open the thread and keep it alive, we need a little more CPU load. So I am not sure whether if we must use cluster system for timing-sensitive concurrency tools like ChatRoom TOOL. It is better we replace this service using other packages.

yours.

Stephen Marquard added a comment - 14-Feb-2008 08:31
xintang,

The code that you're looking it (where the Thread.sleep is) is not involved in event propagation - it's a maintenance thread that removes expired servers from the cluster.

So I have to doubt that your test results mean what you think they mean. It's possible that you are seeing a side-effect of just increasing cpu load on the server (by effectively creating a continuous loop), rather than the actual cause of the bug.

Lance Speelmon added a comment - 14-Feb-2008 12:46
I concur with Stephen, I do not think removing the Thread.sleep is the root of the problem. L

xingtang added a comment - 22-Feb-2008 09:39
I have chat with Chen and Chris, I am going to use JMeter simulate huge users to reproduce this problem.

I cannot reproduce this problem on my machine. I need more information about this bug.

According to my test, the nightly server works fine when i use JMeter testing that server.

On my point of view, I found all messages have been saved to the database successfully everytime.

So, we have a selection that we can search the database by time and get the new messages and push them to the clients. That meas, we need build a new JSP page and using AJAX to serach database CHAT2_MESSAGE.

Sql for Mysql database:

select * from CHAT2_MESSAGE where MESSAGE_DATE > '2008-02-21 04:08:36.0' order by body ASC, MESSAGE_DATE Desc ;

Charles Hedrick added a comment - 23-Mar-2008 10:18
I am seeing this problem in production. I tested with 5 clients and things were fine. But in production with 42 users in a chat room, clients aren't getting updated. the database is fine, because when you manually refresh the data is there.

xingtang added a comment - 25-Mar-2008 11:43
I am trying to replace the old data fetching way by using my ajax function.

That means i will acquire chatroom messages directly form the database.

Charles Hedrick added a comment - 26-Mar-2008 10:32
That would certainly simplify the code enormously. However with several hundred users in a room, this might generate a lot of database traffic. Perhaps a cache of recent data for each chat room?

Peter A. Knoop added a comment - 01-Jun-2008 10:04
Chris/Lance, do you think some of the recent patches Chuck H. has provided helped here, or is there still work needed to address this issue?

Stephen Marquard added a comment - 11-Aug-2008 07:01
With chat 2-5-x (r50548), we are getting reports of problems like:

---
During the chats what happens is that most of the posts will come up, but some of them, about a tenth don't show up on the screen. In order to see them you need to refresh and then re-enter the chat room. If one post doesn't appear, it doesn't affect the next one. This results in disjointed chats as not all the posts come up. I think, although i can't confirm, that not all the computers are always affected. But most of them usually are. Also posts that you yourself may post do suffer the same problem.
---

As noted above, the change in r36652 to courier hasn't ever been merged to 2-5-x. We are trying that as a patch (as we see similar ConcurrentModificationException errors in our app server logs), but are wondering whether this issue is still being experienced at IU, Rutgers or elsewhere?

Lance Speelmon added a comment - 12-Aug-2008 05:44
Stephen - We are still experiencing the issue at IU. Thanks, L

Stephen Marquard added a comment - 12-Aug-2008 07:07

Here's a Selenium script to reproduce the issue:

- Install the Selenium IDE addon for Firefox (tested with FF3 and Selenium IDE 1.0b2).
- In a Sakai site, add the chat room
- Open the 'submit' section in a new tab (so the URL is /portal/tool/.../roomControl)
- In FF3, open Selenium (Tools / Selenium IDE), open the attached script, and run it.
- It should paste in 1000 chat messages. The speed can be varied in selenium.

- In another browser (IE7) log in and go to the same chat room
- As long as you don't post messages, you should see messages 1 to 1000 appearing normally
- As soon as you try and post a message, you'll see dropped messages from the sequence, and possibly your own message also not appearing
- If you reset the tool, you'll see all the messages reappear correctly (i.e. they are in the db)

This was tested on a cluster where the sending selenium-driven browser was on a different app server to the listening IE7 client (not sure if that's relevant yet).


Stephen Marquard added a comment - 12-Aug-2008 07:09
mktest.pl is to create a selenium test script for N iterations.

Charles Hedrick added a comment - 12-Aug-2008 08:47
There are at least 3 separate problems here:

1) If you are seeing ConcurrentModificationException, you are missing a patch that makes m_addresses a ConcurrentHashMap
2) even with that patch, there are serious synchronization issues in BasicCourierService. It needs a thorough review.
3) I believe the problem as described in Stephen's 12-Aug message is different. However BasicCourierService is easy enough to fix that I recommend doing that first.


Stephen Marquard added a comment - 12-Aug-2008 08:55
We've applied the courier patch which gets rid of the ConcurrentModificationException, but the "lost messages" problem is still there.

I couldn't reproduce it on an unloaded 2-app server test cluster though - so far it's only showed up on our production system.

Stephen Marquard added a comment - 12-Aug-2008 09:50
Merged courier change to 2-5-x branch in r50600. This change has been tested on UCT production server, and does appear to eliminate the ConcurrentModificationException from courier.

The overall issue (missing chat messages) remains unresolved.

Megan May added a comment - 13-Aug-2008 08:05
Per Lance, changing to unassigned as there are not resources at Indiana to work on this currently. If there is anyone in the community that can work on this, please get contact lance

Charles Hedrick added a comment - 13-Aug-2008 09:55
I've started looking at BasicCourierService. The first thing I find is that there's a method that can't be useful: hasDeliveries. I don't see any way this could be used safely. By the time you look at the result, it may no longer be true, so it's hard to see how it could be safely used for anything. Fortunately it doesn't seem to be used anywhere.


Charles Hedrick added a comment - 13-Aug-2008 12:36
The current code tries to lock the list of items for delivery at a given list. That can't work, because there's code that adds and removes that list, and locking the list doesn't block those operations. It would be safe to lock the whole hash table, but that would reduce concurrency greatly. What we really need to do is lock the slot in the hash table, but of course when there's no data there's nothing to lock.

What I propose to do is to have an array of objects. We'll hash the address string, take the hash mod the size of the array, and lock that element of the array. That approximates locking the slot, though not perfectly. The size of the array will control the amount of concurrency allowed.


Stephen Marquard added a comment - 13-Aug-2008 13:17
Chuck,

To clarify, you are identifying the underlying cause of the issue as a locking/concurrency problem in deliver(...) and getDeliveries(...) methods in BasicCourierService.java ?

Charles Hedrick added a comment - 13-Aug-2008 14:09
There may be more than one cause. But it seems silly to debug chat when BasicCourierService has obvious bugs that would lose updates.

I've uploaded a patch. I give you diff against 2.5.0 release, and the patched version of BasicCourierService.java.

Since you seem to be able to duplicate the problem, I'd appreciate it if you'd see whether this improves things.


Charles Hedrick added a comment - 13-Aug-2008 14:24
It's worth noting that if there is a very high rate of messages, the "act" method could be called out of order. I can prevent that, but at the code of holding a lock during execution of the method.


Stephen Marquard added a comment - 14-Aug-2008 04:32
I set up a test case to only test courier behaviour: multiple threads calling deliver() and one thread calling getDeliveries().

So far I am able to show that the change in r36652 above fixes the ConcurrentModificationException (i.e. the same result as seen from production logs), so running the test against 2.5.0 BasicCourierService.java shows the exception, whereas under 2-5-x BasicCourierService.java it works fine.

I am unable to show any failures in message delivery through courier in the 2-5-x code, i.e. the 2-5-x code and Chuck H's updated BasicCourierService are functionally the same and neither exhibits behaviour that would lead to the chat symptoms reported.

The test environment is slightly complex but uses:

http://source.cet.uct.ac.za/svn/sakai/scripts/trunk/loadtest/courier.pl
http://source.cet.uct.ac.za/svn/sakai/scripts/trunk/loadtest/SakaiQA.jws

and TestDelivery.java in the SAK-11875 branch for this issue. (The webservices axis project also needs courier-util added as a dependency in its pom).


Stephen Marquard added a comment - 14-Aug-2008 05:06
Running 2-5-x BasicCourierService with 40 insertion threads and 1 listener thread did eventually produce this error:

WARN: error (2008-08-14 13:41:38,608 http-8080-Processor41_org.sakaiproject.axis.SakaiQA)
java.util.ConcurrentModificationException
        at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:449)
        at java.util.AbstractList$Itr.next(AbstractList.java:420)
        at SakaiQA.getDeliveries(SakaiQA.java:1141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
        at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
        at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453)
        at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
        at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)

which is not evident with the SAK-11875 branch version. We are going to try the branch in production and see what happens.

Charles Hedrick added a comment - 14-Aug-2008 06:46
I've updated BasicCourierService.java, but not the diff. I've clarified the comments explaining what I"m doing. There are no changes to the code.

Stephen Marquard added a comment - 20-Aug-2008 23:51
We have been running the SAK-11875 branch in production for about a week, and the previously reported problems appear to have cleared up. It also passes the stress test as detailed above (courier.pl and SakaiQA.jws) without problems.

I am proposing to merge this to trunk and then to 2-5-x.



Megan May added a comment - 22-Aug-2008 04:30
Stephen, the ticket is assigned to you. Are you able to commit this change or do you need someone else to do it?

Stephen Marquard added a comment - 22-Aug-2008 05:11
I can commit it.

Stephen Marquard added a comment - 29-Aug-2008 05:14
I'm not going to get to this until 15 Sep, as I'm away on leave until then. The merge into trunk is slightly complex because the changes are based on 2-5-x rather than trunk code.

If anyone else with commit access to courier would like to tackle this in the meantime, please feel free.

Sites experiencing this problem in production with 2-5-x can use this branch of courier in .externals, which is running successfully in production at UCT:

courier -r50691 https://source.sakaiproject.org/svn/courier/branches/SAK-11875

Stephen Marquard added a comment - 19-Sep-2008 02:36
r52498 restores the use of StringUtil.different for comparison, otherwise you get NPEs as below in the Admin Workspace / Online tool switching from Manual to Auto Refresh to Manual.

org.sakaiproject.portal.api.PortalHandlerException: org.sakaiproject.tool.api.ToolException
    at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:891)
caused by: org.sakaiproject.tool.api.ToolException
    at org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:227)
caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
caused by: java.lang.NullPointerException
    at org.sakaiproject.courier.impl.BasicCourierService.clear(BasicCourierService.java:168)
    at org.sakaiproject.util.ObservingCourier.justDelivered(ObservingCourier.java:110)
    at org.sakaiproject.presence.tool.PresenceToolAction.buildMainPanelContext(PresenceToolAction.java:121)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)

Stephen Marquard added a comment - 19-Sep-2008 03:33
Changes from branch merged to trunk and 2-5-x.