[SAK-37058] Removing a timed assessment during take may disable timed assessment server thread. Timed assessments get stuck in progress. Created: 03-Mar-2017  Updated: 28-Sep-2018  Resolved: 01-Jun-2017

Status: Verified
Project: Sakai
Component/s: Tests & Quizzes (Samigo)
Affects Version/s: 11.2
Fix Version/s: 11.5 [Tentative], 12.0

Type: Bug Priority: Critical
Reporter: David Hutchins Assignee: SAMIGO TEAM (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depend
is depended on by SAK-36506 Events from timer thread should have ... Verified
depends on SAK-34943 Refactor autosubmit to enclose each u... CLOSED
Incorporate
incorporates SAK-34277 Timed Assessment auto-submitted earli... RESOLVED
incorporates SAK-34474 Possible for student to take more tim... RESOLVED
incorporates SAK-37051 Exception when timer submit's an asse... RESOLVED
incorporates SAK-36327 some timed tests don't submit when ti... CLOSED
Relate
is related to SAK-37069 TimedAssessmentRunnable thread stoppe... RESOLVED
11 status: Resolved
Previous Issue Keys: SAM-3170

 Description   

We (UVa) encountered a fault on a production server, that proved difficult to hunt down. After investigation, the root cause of this fault was found to be that exceptions thrown from GradingService may cause SubmitTimedAssessmentThread to die before removing the offending item from the queue. After this has occurred, the server will not timer submit any further assessments from the thread. Restart of tomcat is required after this fault is encountered.

This condition can be reproduced as follows:

  1. As Instructor:
    1. Create a timed assessment with the publishes to GB
  2. As Student:
    1. Begin taking the assessment and see the timer counting down
    2. Leave the assessment
  3. As Instructor:
    1. Delete the published assessment before the timer expires
    2. Wait for the timer to attempt submission

In tomcat logs a GradebookServiceException is seen:

org.sakaiproject.tool.assessment.services.GradebookServiceException: org.sakaiproject.service.gradebook.shared.AssessmentNotFoundException: There is no assessment id=x in gradebook uid=xxx-xxx-xxx

If you attempt another timed assessment then leave it, the assessment it will never submit.

Note: Workarounds for institutions who may be experiencing this issue to get stuck exams submitted:

  1. Instructor could use Autosubmit to retrieve stuck exams.
  2. If the latest submission date for the exam had not passed so the student with the stuck exam could still access the link to it, the student could log out of the affected Sakai server and log into a different server (where the timers for all assessments are still working). Once on a server with functional timers, if the student with the stuck exam clicked the link to it and tried to continue taking the assessment, they’d be told their exam was already submitted and they couldn’t take it. At this point, the timer submit would be triggered and a couple minutes later, the submission would complete.


 Comments   
Comment by Karen McPhaul [ 09-Mar-2017 ]

We experienced this on our 11.2 production instance. About week ago, out of the blue, we began getting daily reports of timed tests not submitting when the timer ran out. We verified that any time a student left a timed test before the timer ran out, the test would not submit when the time limit was reached. On a separate test system running 11.1, we followed the steps in this jira, the exception was thrown, and the problem began on that system. A full system restart did set things right again on both systems.

Comment by David Hutchins [ 09-Mar-2017 ]

I have re-written the TimedAssessmentQueue to use a ScheduledExecutorService with multiple threads, and converted the "thread" code into a Runnable that gets queued for each assessment, instead of the current one thread for all approach. We still need to finish testing, but so far it looks good.

Can't prove it, but I suspect this issue is responsible for a number of other "can't reproduce " timed assessment issues we have encountered.

Comment by David Hutchins [ 17-Apr-2017 ]

We have had this (PR: https://github.com/sakaiproject/sakai/pull/4227) in production since 4/11/2017, and at the time of this comment have processed 4300 timed assessments, 208 of which were submitted by the timer code.

Comment by Tiffany Stull [ 13-Jun-2017 ]

It seems other institutions may be experiencing this issue, so I've modified the title of the jira to hopefully make it easier to find. Also included a note in the description for workarounds to submit stuck exams.

Comment by Matthew Jones [ 14-Jul-2017 ]

Marking tested based on UVA's comments

Generated at Wed Sep 18 14:04:16 CDT 2019 using Jira 8.0.3#800011-sha1:073e8b433c2c0e389c609c14a045ffa7abaca10d.