[SAK-37056] Auto-submit always sends last score which can allow zeros to override scores on multiple submissions in various cases Created: 07-Nov-2014  Updated: 10-Oct-2018  Resolved: 06-Mar-2018

Status: Verified
Project: Sakai
Component/s: Tests & Quizzes (Samigo)
Affects Version/s: 10.3, 11.4, 12.0, 19.0
Fix Version/s: 11.5 [Tentative], 12.0, 19.0

Type: Bug Priority: Blocker
Reporter: Sam Ottenhoff Assignee: Matthew Jones
Resolution: Fixed Votes: 1
Labels: LSClient, manifesto!, needstesting
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File screenshot-1.png    
Issue Links:
Depend
depends on SAK-34943 Refactor autosubmit to enclose each u... CLOSED
Duplicate
is duplicated by SAK-40765 Event Log is incorrectly modified whe... Verified
Relate
relates to SAK-36393 Autosubmit should also submit opened ... Verified
relates to SAK-35810 Autosubmit works such that it allows ... CLOSED
relates to SAK-37079 Text change: Autosubmit job submittin... Verified
is related to SAK-34145 Clarify to user what will happen with... CLOSED
12 status: Resolved
11 status: Resolved
Previous Issue Keys: SAM-2435
Test Plan:

Steps to replicate:

There are a few slight variations on settings that should be tested here. Most of the test plan for each is the same with the variation at the end. The variations are

  • Timed verses un-timed
  • Autosubmit or not
  • All 3 grading options in grading and feedback (hightest/last/average)

{Common for all}

Add Tests & Quizzes and Gradebook to a site. This needs to be a server where auto-submit is enabled (most of the nighties should have this setup now)

1) Create assessment with 2 questions each with points assigned to them (10 points is best)
In settings
2) Ensure that Grading and Feedback "Send assessment score to Gradebook immediately, regardless of options below" is checked
3) Ensure that unlimited submissions are allowed

Save this assessment as a working copy. All published copies will be differing variations of this.

3) Click settings on the Working copy

(Test 1 to 3 - No special submission, highest/average/last)

1) In settings set due and retract date in 5 minutes from now
2) Set grading option to highest
3) Change the name to indicate "No special highest" Publish
4) Take the assessment as a student. Get all answers correct the first time scoring 20, submit.
5) Take again get all answers wrong the second time scoring 0, submit.
6) Login as instructor, the highest grade (20) should be recorded in the gradebook.

Repeat all steps using average and last. For the average 10 should be recorded and for the last 0 should be recorded.

(Test 4-6 Auto Submit job, highest/average/last)

This is similar to the previous tests, however autosubmit should be checked.
Complete steps 1-4 in the previous tests
In Step 5 do NOT submit, just save.
After the due date, run the auto submit job as admin.

Check all of the scores as in the previous test. It should be correct (20 for highest, 10 for average 0 for last)

(Test 7-9 Timed assessment, highest/average/last)

Similar to the autosubmit, but this is a timed assessment

  • Set a timer of 5 minutes in addition to the other settings
  • Wait for 5 minutes
  • After the time is up check the grading (There might be some delay?)

These tests are essentially verifying any second (and subsequent submissions) automatically done or done manually provide the correct scores into the gradebook.


 Description   

There's a grading issue where zeroes can overwrite good scores even for highest score. In my research autosubmit will only always submit the last (current) score to the gradebook. There is some logic to have it not submit a score to the gradebook in the highest score case, but no consideration for average score. There are a few ways to replicate this. All involve multiple submissions with auto-submit running at the end. Either on the end of the initial run or any time an instructor modifies the dates and gives students more time.
 
See test plan for full steps to test this.



 Comments   
Comment by Stephen Marquard [ 12-Nov-2014 ]

What are the expected vs actual results here? I would expect two submissions, both auto-submitted.

Comment by Sam Ottenhoff [ 12-Nov-2014 ]

Expected result would be the highest score is sent to the gradebook.

Actual result is the student receives a zero because the step 5 and 6 lead to a blank submission that results in a zero.

Comment by Neal Caidin [ 09-Jan-2018 ]

Please Qa this issue. It might be fixed in 11.x (11.5) and later.

 

Comment by Matthew Jones [ 09-Jan-2018 ]

May be incorporated in 11.5 and 12 by SAM-3128

Comment by Sam Ottenhoff [ 09-Jan-2018 ]

SAM-3128 is a text change only

Comment by Matthew Jones [ 09-Jan-2018 ]

Yeah, I see that now, so there's still some edge case where the autosubmitted job is overwriting the score for a "highest score"? I wouldn't expect this. There's 3 scoring cases based on the SAM-3128 wording of every assessment auto submitted.

When auto submit runs and submits, depending on the scoring option


1) If Highest score and score less than highest don't record, otherwise this is the score (Easy/obvious/default)
2) If Average score and more than one attempt already submitted don't include (Debatable)
3) If last score and more than one attempt already submitted (This could go either way too)

I'd probably for options 2 and 3 always say auto submit will not include it in the score, but I don't know how possible it is to have an submission ignored for scoring currently in Samigo, so that might take some work.

Alternatively for 2 and 3 it could include it but somewhere it should indicate to the instructor that a the score includes an auto-submitted assessment so may not be correct.

Comment by Benjamin Rappleyea [ 09-Jan-2018 ]

Work around that we found to correct the issue and get the correct submission to register in the gradebook was to unassociate from the gradebook and then reassociate. This corrects the issue and the highest score then populates in the gradebook.

In Sakai 10.2 and prior the behavior was reportedly that if a user already had a submission autosubmit would not submit a new score.  Did this logic get removed on purpose?

Comment by Tiffany Stull [ 09-Jan-2018 ]

Benjamin Rappleyea Yes, that logic got removed on purpose.  See comments in SAM-3128.  Also, there was a discussion on the sakai-dev and sakai-users lists explaining why the logic changed.

Comment by Benjamin Rappleyea [ 30-Jan-2018 ]

Added label of LSClient as this is being handled by Duke and ISU through LongSight

Comment by Matthew Jones [ 16-Feb-2018 ]

I feel like there has to be some step missing here in the test plan, I can't get this to reproduce anymore as-is.

I found this issue SAM-1618 (11.0) where this code was touched a little which was a month after this was originally filed through it still seems to be reported.

Here's the problem I'm having with the test case "as-is".

  • What should the original retract date be? (Maybe it doesn't matter)
  • On step 5, after modifying JUST the retract date, the student who previously submitted is unable to submit again. Retract date only allows "one chance to submit after due date IF they haven't already submitted".. Autosubmit run on step 3 submits it for them.
  • Changing both the due date and retract date (instead of just retract) is NOT causing the lower second score of zero to overwrite the higher first score.

I looked back through the issues and I can't see what's missing here and what the problem is.

It sounds like this "might" just be a problem with incorrect grades going to the gradebook while the correct grades are visible in Samigo. This isn't listed as a step.

Comment by Matthew Jones [ 16-Feb-2018 ]

So it looks like prior to SAM-1618 updateCurrentGrade was false by default and set to true. After SAM-1618 it was true by default and set to false. The missing steps here might be that the gradebook is updated with zeros rather than Samigo and the gradebook needs to be checked?

Comment by Matthew Jones [ 19-Feb-2018 ]

Okay, I am seeing this now with Gradebook. I'm not sure who originally filed the description but I cannot reproduce it with those steps. I've updated the steps for a case I can reproduce.

Comment by Matthew Jones [ 22-Feb-2018 ]

Okay so from what I'm seeing here the issue is a little different than the title. The auto submit jobs will always send the current (last) score to the gradebook It never seems to send the average or highest. There was a special case added in SAM-1618 that won't update the score if the current score is lower than the highest score and while this is a pretty good idea this won't work for average score. Ideally this code should be refactored so these special cases aren't dealt with in 2 places but I don't know how possible that is.

Comment by Jolie Tingen [ 22-Feb-2018 ]

Matthew Jones We might be seeing this issue even when the due/retract date isn't modified. I'll update our LS ticket with references to other issues where we've reported this including one within the past week.

Comment by Matthew Jones [ 23-Feb-2018 ]

I agree Jolie. And I think prior to 11 those running autosubmit didn't see this as much of an issue.

However this change on SAM-2729 would submit any attempt whether empty or not if more than one submission was allowed. And the problem is that auto submit never did anything other than send the score for it's assessment to the gradebook, essentially the last score. not considering the scoring options of highest/average. There was some logic added to autosubmit in 11 to try to address the highest case but it's not running for me, I think it's only to catch a special case, not a general case.

So prior to SAM-2729 it seemed much less likely a zero would ever be submitted and overwrite a score.

I think modifying the retract/due dates is only one way to reproduce this, it seems another easier way would just be for a student to start an assessment, start another, leave it without answering anything and have that second auto submitted.

Comment by Benjamin Rappleyea [ 23-Feb-2018 ]

Good morning all!

Some time ago I spoke with Tiffany Stull about this, hoping to hear that it was just minor glitch that she knew a good fix for, but no... she informed me that this is in fact a regression (in UVA's opinion) and they have corrected the issue by reverting to the Sakai 10 logic prior to  SAM-3128 being implemented. See her comment above in this thread:

Benjamin Rappleyea Yes, that logic got removed on purpose.  See comments in -SAM-3128-.  Also, there was a discussion on the sakai-dev and sakai-users lists explaining why the logic changed.

Tiffany Stull also mentioned that UVA does not have this issue any longer.

Comment by Matthew Jones [ 23-Feb-2018 ]

Well, SAM-3128 was just a text change, the actual code change was on SAM-1618.

I think reverting that would probably eliminate this problem, but introduce other problems where a partial submission cannot be submitted. Even if SAM-1618 only auto-submitted submissions that didn't have a score of zero, it could still overwrite the good grade. I kind of feel like the only way forward is just to keep trying to fix this. I have the highest score case fixed, but I don't have a fix yet for the average score.

Comment by Matthew Jones [ 07-Mar-2018 ]

I added a pretty complete test plan that needs to be run though. Let me know if anyone has any questions here or on slack about it or how it goes!

Comment by Neal Caidin [ 07-Mar-2018 ]

I assume the 12 status flag should be set to merge?

 

 

Comment by Matthew Jones [ 07-Mar-2018 ]

Yeah, I set the merge/status flags up.

Comment by Neal Caidin [ 12-Mar-2018 ]

For the Test Plan, what should be the setting for the "Send assessment score directly to Gradebook immediately..." be set to? If unchecked, then after the exam is over should it be checked? Otherwise, if there is no role for the Gradebook, is there any need for it in the Test Plan? 

Thanks.

 

Comment by Neal Caidin [ 12-Mar-2018 ]

Ah, sorry. I see the step "Send grades to gradebook" should be checked. Nevermind.

 

Comment by Matthew Jones [ 12-Mar-2018 ]

Yeah right it should be "Send Grades to the gradebook immediately" for every option for ease of testing. The problem here lies is when various workflows send to the gradebook.

Comment by Jolie Tingen [ 12-Mar-2018 ]

Still testing, working through all the possible combinations. So far, so good. Results are here - https://docs.google.com/spreadsheets/d/1D_QdpZ9bwbcCu4QD3_vo7thKFVl0s1dSpwytgGLOqjk/edit#gid=0

Comment by Jolie Tingen [ 12-Mar-2018 ]

So, on the spreadsheet, I said that the Average Autosubmit passed but maybe not. For each student I submitted two of the three submissions allowed. I expected an average of 62.5 for each, but got 41.67. I assumed that all submissions were required for the average, but for all of these tests, I am just starting the last submission with no edits and closing the browser, trying to replicate the issue of a null submission on the last. The Average option counted that as a 0, but I don't believe that is true of the others I tested. Matthew Jones tell me how it works. Does the average calculate with number of possible submissions as the divisor for submitted scores? And for any submissions not used get the score of 0?

Comment by Matthew Jones [ 12-Mar-2018 ]

It should count the last submission as a zero. This was a change made in SAM-2729. We have an open task on SAM-3359 to warn the user before starting than auto-submit could result in a zero if started and not completed.

So for the auto submit cases.

Average should count the 0 in the average
Last should result in 0
Highest should have a value other than zero that was calculated prior.

Comment by Jolie Tingen [ 12-Mar-2018 ]

Ok, I still think my tests were passing. I'm doing more testing in the morning to cover the remaining options on the spreadsheet and will report back here with an update.

Comment by Neal Caidin [ 12-Mar-2018 ]

I think the tests are passing too. I don't think the category of timed assessment with auto submit is any different than just timed assessment. 

Comment by Matthew Jones [ 12-Mar-2018 ]

Yeah, there's only 3 cases.

Autosubmit
Timed Assessment
Manual Submit

There's no point to autosubmit after a timed assessment as it'll already be submitted by the timer.

Comment by Tiffany Stull [ 13-Mar-2018 ]

Matthew Jones, oddly enough, many instructors enable both a timer and autosubmit.  I have seen one case where the autosubmit job managed to trigger before the timer fully expired on a one-page test, and the student's test autosubmitted with several answers blank (because autosubmit does not save before submitting).

Comment by Neal Caidin [ 13-Mar-2018 ]

Setting to verified based on testing by me, Jolie Tingen and Matthew Jones .

 

Comment by Matthew Jones [ 13-Mar-2018 ]

Tiffany Stull Autosubmit submitting a timed assessment early is a different problem from this one. Autosubmit should not run until after the due and retract dates have passed. If the timer is able to run past those dates that's also a different problem.

Comment by Tiffany Stull [ 13-Mar-2018 ]

Yes, I'm aware it's a different problem, but something of note for testing autosubmit and timers.

I think the trouble in the case of the autosubmit job submitting over the timer might have been related to the timer's grace period of several seconds while it is saving and submitting.  So if it the timer gets truncated by the date, it might be a few seconds after the date before it fully saves/submits.  But I've only seen it once and I could not reproduce.

Comment by Matthew Jones [ 13-Mar-2018 ]

Yeah, auto-submit cannot save because it's not on the client. And currently by default auto-submit isn't even scheduled to run so every school may have a different experience depending on their schedule. Some schools only run auto submit once a day, some every hour some possibly more often. The point is really just to catch anything that didn't get submitted so too frequently running this shouldn't catch anything, but it's not going to submit any answers for the user, that's only in the client.

I don't know if the code for autosubmit checks the timer at all but maybe that could be incorporated. There were those UVA fixes for the timer SAM-3170 to make it more reliable which might also reduce this as a problem.

As long as all methods send the "correct" score to the gradebook, that was what all this issue fixed. Perhaps a better solution in the future could be if auto submit was ever turned on by default, it could handle submission of the timed assessments. The timer would just mark them as complete and ready to autosubmit. But nobody has set this up yet. Maybe for 13.

Comment by Tiffany Stull [ 13-Mar-2018 ]

Dates really need to function more like timers, so that work saves at the latest submission date if the student is still in the process of working on a test at that date (instead of it looking like they can continue working on the test when they can't), but it would require a total rewrite of due/late submission functionality.  Anyway, a discussion for another JIRA .

Comment by Matthew Jones [ 15-Mar-2018 ]

Okay this should be on the 11.x nightly server as of tomorrow (3-16). I'd be cool if someone could check it out there. At least verify the auto submission case is working.

Comment by Derek Ramsey [ 16-Mar-2018 ]

Tested on 11x with due and retract date.  Both were submitted when autosubmit job completed.  Also verified the average and last scores were submitted to gb properly when autosubmitted.

Comment by Sam Ottenhoff [ 10-Oct-2018 ]

Seems like this change causes auto-submit to overwrite good event log data .... SAK-40765

Generated at Sat Oct 19 23:21:47 CDT 2019 using Jira 8.0.3#800011-sha1:073e8b433c2c0e389c609c14a045ffa7abaca10d.