[SAK-37058] Removing a timed assessment during take may disable timed assessment server thread. Timed assessments get stuck in progress. Created: 03-Mar-2017 Updated: 28-Sep-2018 Resolved: 01-Jun-2017
|Component/s:||Tests & Quizzes (Samigo)|
|Fix Version/s:||11.5 [Tentative], 12.0|
|Reporter:||David Hutchins||Assignee:||SAMIGO TEAM (Inactive)|
|Remaining Estimate:||Not Specified|
|Time Spent:||Not Specified|
|Original Estimate:||Not Specified|
|Previous Issue Keys:||SAM-3170|
We (UVa) encountered a fault on a production server, that proved difficult to hunt down. After investigation, the root cause of this fault was found to be that exceptions thrown from GradingService may cause SubmitTimedAssessmentThread to die before removing the offending item from the queue. After this has occurred, the server will not timer submit any further assessments from the thread. Restart of tomcat is required after this fault is encountered.
This condition can be reproduced as follows:
In tomcat logs a GradebookServiceException is seen:
org.sakaiproject.tool.assessment.services.GradebookServiceException: org.sakaiproject.service.gradebook.shared.AssessmentNotFoundException: There is no assessment id=x in gradebook uid=xxx-xxx-xxx
If you attempt another timed assessment then leave it, the assessment it will never submit.
Note: Workarounds for institutions who may be experiencing this issue to get stuck exams submitted:
|Comment by Karen McPhaul [ 09-Mar-2017 ]|
We experienced this on our 11.2 production instance. About week ago, out of the blue, we began getting daily reports of timed tests not submitting when the timer ran out. We verified that any time a student left a timed test before the timer ran out, the test would not submit when the time limit was reached. On a separate test system running 11.1, we followed the steps in this jira, the exception was thrown, and the problem began on that system. A full system restart did set things right again on both systems.
|Comment by David Hutchins [ 09-Mar-2017 ]|
I have re-written the TimedAssessmentQueue to use a ScheduledExecutorService with multiple threads, and converted the "thread" code into a Runnable that gets queued for each assessment, instead of the current one thread for all approach. We still need to finish testing, but so far it looks good.
Can't prove it, but I suspect this issue is responsible for a number of other "can't reproduce " timed assessment issues we have encountered.
|Comment by David Hutchins [ 17-Apr-2017 ]|
We have had this (PR: https://github.com/sakaiproject/sakai/pull/4227) in production since 4/11/2017, and at the time of this comment have processed 4300 timed assessments, 208 of which were submitted by the timer code.
|Comment by Tiffany Stull [ 13-Jun-2017 ]|
It seems other institutions may be experiencing this issue, so I've modified the title of the jira to hopefully make it easier to find. Also included a note in the description for workarounds to submit stuck exams.
|Comment by Matthew Jones [ 14-Jul-2017 ]|
Marking tested based on UVA's comments