[SAK-34849] 'Minimum value' option for questions Created: 16-Jul-2010  Updated: 01-Feb-2019  Resolved: 09-Feb-2016

Status: RESOLVED
Project: Sakai
Component/s: Tests & Quizzes (Samigo)
Affects Version/s: 2.7.0
Fix Version/s: 11.0

Type: Feature Request Priority: Major
Reporter: Nicola Monat-Jacobs Assignee: Matthew Jones
Resolution: Fixed Votes: 1
Labels: PATCH_ADDED, bigone, defaultfalse

Attachments: Microsoft Word !SAM-948-Spec.doc     Text File SAM-948-4.patch     Text File SAM-948-5.patch     Text File SAM-948-float-double.patch     Text File SAM-948_sak26x.patch     Text File SAM-948_sak28x.patch     PNG File Screen shot 2011-04-12 at 3.04.25 PM.png    
Issue Links:
Depend
is depended on by SAK-34160 samigo: Minimum point value fails for... OPEN
is depended on by SAK-40450 samigo: Minimum point value fails for... OPEN
is depended on by SAK-40460 samigo: accessibility: Minimum point ... OPEN
is depended on by SAK-40464 samigo: Export/Import ignores minimum... OPEN
is depended on by SAK-41293 samigo: Tests & Quizzes - Parent jira... OPEN
is depended on by SAK-34963 Option to Not Display how many points... Verified
Relate
is related to SAK-40473 samigo: Option to add minimum point v... OPEN
is related to SAK-40476 Tests & Quizzes - Parent jira for bug... OPEN
is related to SAK-36597 Cannot add negative points to the 'Mu... Verified
is related to SAK-36657 Negative Points option does not get c... Verified
is related to SAK-37096 samigo: Incorrect behavior of Multipl... Verified
is related to SAK-34651 Assign partial credits for test (More... RESOLVED
is related to SAK-34741 Ability to flag question as Extra Credit RESOLVED
Property addition/change required:
Yes
Previous Issue Keys: SAM-948
Test Plan:

Set sakai property
samigo.allowMinScore=true

  • Create an assessment with some questions, set the minimum score to a value (like 5) set the score to another value (like 10)
  • Publish the assessment
  • Take the assessment as a student
  • Get the questions wrong, see the minimum score is recorded

 Description   

We've had clients request a 'minimum value' option for test questions. The scenario is: the instructor has some optional, extra credit test questions, and if a student attempts the question, they should get some credit for answering the questions, even if they got the questions wrong, just for attempting.

Thoughts or suggestions?



 Comments   
Comment by Bryan Holladay [ 08-Apr-2011 ]

I've attached a patch to this jira... please review and commit

Comment by Bryan Holladay [ 08-Apr-2011 ]

Attached a patch for Sak 2.6.x... doesn't need to be merged, just thought I'd supply it since I have it and in case someone wants to use it for 2.6.x

Comment by Karen Tsao [ 08-Apr-2011 ]

Assign to our Ux designer to review.

Comment by Jacqueline Mai (Inactive) [ 08-Apr-2011 ]

Hi Bryan and Nicola,

Is there a Sakai instance where I could take a look at this patch?

Thanks,
Jackie

Comment by Nicola Monat-Jacobs [ 11-Apr-2011 ]

Jackie - I'll send you some login info where you can test this feature out.

Comment by Bryan Holladay [ 12-Apr-2011 ]

Added a new sakai.property to turn off min value if not wanted (default is true)

samigo.allowMinScore

also added a description for the field

Comment by Jacqueline Mai (Inactive) [ 12-Apr-2011 ]

Hi Nicola,

Thanks for granting me access to a server to review the feature. A couple of observations:
-My first thought is that there should be some explanation on the UI to explain what this new feature is - either provide a succinct description directly on the UI or else as an explanation via a 'What's This?" link, that's prevalent elsewhere in SAMigo. MC question type, in particular, has multiple point value options (answer point value, negative marking value, partial credit (if enabled)...so a new point value option needs to be explained.
-If the intention is to give credit for attempting to answer a question, then the MC single correct question type should not allow users to enter negative marking or partial correct answer point value. During my review, I was able to provide a negative marking value for single correct multiple choice question but the scoring ignored that value; it only took into account the minimum value.

Have you tested this feature with clients who requested it in the first place?

Thanks,
Jackie

Comment by Bryan Holladay [ 12-Apr-2011 ]

attached a new patch that will disable the negative point field when there is a min point value entered

Comment by Bryan Holladay [ 12-Apr-2011 ]

Jackie,

I just added a patch that has a description about 20 minutes before your post (see attached image). Also, I just added a new patch that will disable the negative point value when there is a min value entered. This will be up on our sakai instance for you to look at in about 10 mins.

Thanks,
Bryan

Comment by Jacqueline Mai (Inactive) [ 12-Apr-2011 ]

Hi Bryan,

Does your patch also handle disabling of partial credit if it's enabled? If partial credit is enabled, then under Multiple Choice, Single Correct, "Enable Negative Marking, Enable Partial Credit, Reset to Default Grading Logic" should be disabled. Refer to page 3 of SAM-818 spec for screenshot of what I'm talking about since the Longsight's Sakai instance does not have partial credit enabled.

Also, if you'd like to include this patch in a future release of SAMigo, it might be helpful to have this feature be configurable (turned off by default) since the UI for SAMigo is already so cluttered that we need to be judicious about what to add as part of the default UI. Also, since an instructor could manually adjust a question's score if he wanted to give credit to a question that was answered incorrectly, I consider this new feature a nice-to-have but not a must-have.

Thanks,
Jackie

Comment by Bryan Holladay [ 14-Apr-2011 ]

Jackie,

Why would you want to disable partial credit with min score? The way I am thinking of it, is a user could potentially setup a multiple point single correct question with the following:

Points: 10
Min Score: 3

"Enable Partial Credit" Checked

A) 100%
B) 70%
C) 0%
D) 0%

This way, if they answer C or D they will still get 3 points for attempting the question.

Granted, the user could put a partial credit value that is lower than the min score, so in that case, the student would still get the min score.

Comment by Bryan Holladay [ 14-Apr-2011 ]

Attached a new patch:

Now "Min Value" feature is off by default. To turn on, set samigo.allowMinScore=true in sakai.properties

Comment by Jacqueline Mai (Inactive) [ 05-May-2011 ]

[this message was sent via email on 04/28/11 but didn't get posted as comment here so manually pasting]

Hi Bryan,

Sorry I missed this message from you. I was thinking about this purely in the context of optional extra credit questions, which this feature is supposedly designed for. If that's the case, then it wouldn't make sense to have this feature in a partial credit context. An instructor wouldn't go through the trouble of making a question partial credit, which is meant to reward some level of correctness in a response but is not designed to give credit to an incorrect response. Having a minimum point value here would be very confusing, at least to me. Perhaps I don't fully understand all the contexts that this feature is supposed to support. It would be helpful if you could provide full specs on this feature - what problem(s)/need(s) is it solving and how does it do it. It would help me and anyone wishing to use this feature to understand how it's supposed to work.

Thanks,
Jackie

Comment by Alan Regan [ 05-May-2011 ]

Hi, all.

I originally requested this update based on the need of some of our faculty at Pepperdine University. In our Sakai 2.6 instance, some of our professors have been entering a negative value into the negative value field of Tests & Quizzes. Given the programming in the 2.6 version of Tests & Quizzes, two negatives make a positive and will allow for a "minimum" value for any attempt at a question. While testing Sakai 2.8, I see that this "feature" has been corrected. So that our professors can still accomplish their goals, we requested a minimum point value option for any question in Tests & Quizzes.

USE CASE: A participation quiz at the start of every class period that rewards correct answers and attempts. The professor has a timed test available during the first 10-15 minutes of class. The quiz is short – 5-10 questions. If each question is worth 2 points, the professor does not want to dramatically penalize attempts. Most questions are review questions from the previous class period that may still be unclear to the students. So, the professor wants students that get the answer right to get full points, students that at least attempt the question to get some points (1 point in this example), and only students who fail to take the quiz or fail to answer questions will receive no points. This serves as a way for the professor to "take attendance" and to reinforce class concepts without limited impact on the student's overall participation grade. In this low-stakes assessment, the professors wants to reward both knowing the answer and giving it the "good college try." On full exams, which are high-stakes assessments, the professor would not use the minimum point value option.

Does this help?

Comment by Jacqueline Mai (Inactive) [ 06-May-2011 ]

Hi Alan,

Thanks for explaining your use case. I know Longsight has provided code for this new feature which I looked at briefly and provided feedback on areas that stood out as potentially confusing. My mistake was not to ask upfront for a full spec of how this new feature is supposed to work across all affected pages of SAMigo (authoring and grading will be affected since this feature introduces a new grading logic to the mix). I asked Bryan for a spec but have not heard back. If there isn't one, might you work with Bryan and team to document this new proposed feature? Before the SAMigo can include it in a future release, we need to insure that contributed code does not negatively affect user experience or come in conflict with other contributions.

https://jira.sakaiproject.org/browse/SAM-939 provides an example of what we are looking for in a design spec document.

Thanks,
Jackie

Comment by Alan Regan [ 06-May-2011 ]

Hi, Jackie.

Happy to! I'll work with Bryan and see what else we can provide.

Also, I want to review all of the comments on this thread again and see if I can find a 2.8 instance that has partial credit enabled. The QA servers I have been using do not have this feature enabled, so I wasn't able to see this as a possible solution.

More later... thanks, Jackie!

– Alan

Comment by Jacqueline Mai (Inactive) [ 06-May-2011 ]

Hi Alan,

Great. Thanks! I believe partial credit is disabled by default. You would need to contact the server administrator to enable it via sakai.properties.

Best,
Jackie

Comment by Bryan Holladay [ 09-May-2011 ]

Jackie,

I've attached the requested spec to this jira.

Thanks
Bryan

Comment by Jacqueline Mai (Inactive) [ 09-May-2011 ]

Hi Alan and Bryan,

Bryan, thanks for the spec.

Alan, reviewing this again makes me wonder why partial credit would not serve your need? Granted, it's currently only available for MC, single correct answer but could potentially be built out for other auto-scored question types. To add this minimum value feature on top of partial credit would be very confusing as it changes users' expectation of how partial credit is supposed to work. For instance, if an instructor wants to give a minimum value to a MC, single correct answer question, why not specify a partial credit value for incorrect answer(s)?

Thanks,
Jackie

Comment by Alan Regan [ 09-May-2011 ]

Hi, Jackie.

I am working with Bryan to get access to a server with 2.8 and partial credit enabled. He sent details on a server and I am trying to carve out some time – too many meetings today!

I'm curious, is there a reason that partial credit is disabled by default? I didn't realize it was available since it's off by default on the QA servers. I can see a lot of faculty appreciating the feature. Also, I think you're right. For the use case I described, the partial credit feature would need to be extended to other auto-graded question types.

More later... thanks again to you both, Bryan and Jackie!

– Alan

Comment by Jacqueline Mai (Inactive) [ 09-May-2011 ]

Hi Alan,

Partial credit was contributed by Rice University and released as part of SAMigo 2.7. If I remember correctly, I believe one of the main reasons for turning it off by default had to do with our lack of confidence in the QA that had been done to it prior to its release. Up until 2.7, many of the community contributions had been running locally for a period of time before they were contributed to SAMigo. This allowed time for issues to arise and get fixed locally before they got contributed to the community. Partial credit was a new request and had not been used at Rice University before. Plus, it wasn't a need at Stanford and we didn't know how much of a need it was to other Sakai schools. We thought it best to leave it disabled by default.

Best,
Jackie

Comment by Alan Regan [ 07-Jun-2011 ]

Thanks for the reasoning for disabling partial credit by default, Jackie. If the solution is much more stable, I think you'll find faculty interest in it for some use cases. (If it's not on, though, they won't even know it exists...)

I was finally able to get access to a Longsight test server with Partial Credit enabled. I like the feature and it works as expected.

The benefits of Bryan's development:
1.) Easy. Number one complaint of online systems is "how many clicks it takes to do anything." If a professor wants a minimum value, Bryan's solution offers one form field. The partial credit option provides multiple form fields... more work on the professor in the situation when the exact same partial credit number would be applied to all answers. Not a deal breaker, but not as quick as one field.
2.) Can apply to any auto-graded question type, while partial credit only applies to multiple-choice/single-answer currently.

The value of Jackie's points:
1.) I am 100% behind not making the programming more complicated than necessary. If partial credit could be applied to all reasonable auto-graded question types, I think this would be a reasonable solution.
2.) If the minimum value option is considered (for less clicking and ease of use for faculty), then I agree that if partial credit is selected by the professor, the minimum value option should be disabled – to prevent confusion.

Thoughts?

Comment by Jacqueline Mai (Inactive) [ 24-Jun-2011 ]

Hi Alan,

I was discussing your use case with a colleague. One question she brought up is why an instructor would want to define the minimum point value on a per question basis, especially if the point value is the same across multiple questions? For instance, if an instructor had 5 questions to which he wanted to give the same point value, wouldn't it be better if he could define the point value once and then specify the questions he wanted to apply this point value to?

Comment by Bryan Holladay [ 12-Sep-2012 ]

Ignore the rest of the patches and only use this one (#5)

Comment by Jacqueline Mai (Inactive) [ 13-Sep-2012 ]

Hi Bryan,

Can you explain what patch #5 does?

Thanks,
Jackie

Comment by Bryan Holladay [ 13-Sep-2012 ]

compared to #4:

exactly the same except for GradingServices.java

the original patch assumed the iterator was going through a single question at a time. Instead, it actually iterates through all questions on the page. So the min score is being tracked by a map:

//itemId ->

{user's score, minScore, # of answers}

.
Map<Long, Float[]> minScoreCheck = new HashMap<Long, Float[]>();

Once this map is populated with the values for each question type (only populated with questions that have a minScore value), it verifies that the min score was scored. If not, then it gives each question part a percentage of the minScore in order to equal 100% of the minScore.

Comment by Jacqueline Mai (Inactive) [ 13-Sep-2012 ]

Hi Bryan,

There have been comments back and forth up until June 2011. My question about what the patch does was to find out whether it addressed some items raised such as the ability to provide minimum value that applies to multiple questions (vs. defining min value each question at a time) so I was looking for more of a functional explanation rather than a technical answer . I'm still unclear on the value of this new feature so I'll add it as an item for discussion at the next SAMigo Working Group meeting.

Thanks,
Jackie

Comment by Alan Regan [ 24-Sep-2012 ]

Hi, everyone.

Just following up on this item. This JIRA was created based on a valid teaching use case at our institution. Here is the use case I posted earlier:

USE CASE: A participation quiz at the start of every class period that rewards correct answers and offers partial credit for attempts. The professor has a timed test available during the first 10-15 minutes of class. The quiz is short – 5-10 questions, each question worth 2 points in this example. The professor does not want to dramatically penalize attempts. Most questions are review questions from the previous class period that may still be unclear to the students. So, the professor wants students that get the answer right to get full points, students that at least attempt the question to get some points (1 point in this example), and only students who fail to take the quiz or fail to answer questions will receive no points. This serves as a way for the professor to "take attendance" and to reinforce class concepts with limited impact on the student's overall participation grade. In this low-stakes assessment, the professors wants to reward both knowing the answer and giving it the "good college try." On full exams, which are high-stakes assessments, the professor would not use the minimum point value option.

***

So, in the above, let's say there are 5 questions in a participation quiz. Each question is worth 2 points, so a total of 10 points possible. The goal is not to dramatically penalize attempts and to also recognize the value of this participation and preparation assessment as a learning and reinforcement tool. If there isn't a minimum value and the student missed everything, then the student would receive 0 points. With minimum value, then the student would at least receive 5 points, which is still failing but not nearly as devastating as a 0. In the above example, if there are 10 class sessions each with a participation quiz, receiving minimum credit would earn 50/100 points. Without this, it would be 0/100 points. It's a big difference numerically.

Bear in mind, the professor that asked for this teaches three sections of a GE class with a total of 150 students. His use case helps him in several ways: (1) it helps take attendance, (2) it reinforces concepts from prior classes or class readings, (3) it helps the professor and the students assess what they remember from last class period and the reading (and areas that need focus/attention), and (4) it simplifies the scoring of these quizzes for the professor (manually giving partial credit to these students would be unbearable).

****

There was an earlier question:
"I was discussing your use case with a colleague. One question she brought up is why an instructor would want to define the minimum point value on a per question basis, especially if the point value is the same across multiple questions? For instance, if an instructor had 5 questions to which he wanted to give the same point value, wouldn't it be better if he could define the point value once and then specify the questions he wanted to apply this point value to?"

Good suggestion. If the assessment is uniform, then I could definitely see value in applying the minimum point value option at the "part" level instead of the "question" level. If all questions in the part are worth the same amount (e.g. 2 points each), then it would be very efficient to set the minimum point value in the "part" settings instead of the question settings. Would this address concerns about this feature?

BTW, if you see my post from June 7, 2011 – I just want to be sure that there's a consistent "partial credit" option on all scored question types (M/C, T/F, FITB, etc.) so that we can support the use case mentioned. If this patch isn't the option, that's fine by me as long as there is a solution that makes it easy for the professor.

Thanks, everyone!

Comment by Matthew Jones [ 29-Oct-2014 ]

When/if this gets into 11, it will should probably have the float -> double from SAM-2087 and a conversion for this already running it. Attaching patch.

Comment by Alan Regan [ 03-Feb-2016 ]

Since there's talk of inclusion in Sakai 11, wanted to repeat that this functionality is actively in use at our university in production and has been a great option for our professors. It helps make the Tests & Quizzes tool an effective attendance and participation tool – providing at least a minimum amount for review questions (e.g. each question worth 3 points with minimum value of 2 points would have minor impact on "wrong" answers). This is incredibly helpful in some of our larger classes (80-200 students) – it not only reinforces class content/concepts but also aids in identifying areas of confusion within the class, plus the bonus of taking participation/attendance. I hope this gets added into Sakai 11 as a core feature (even if it's disabled by default!). Thanks!

Comment by Matthew Jones [ 03-Feb-2016 ]

PR is up for review on this

Comment by Neal Caidin [ 13-Jul-2016 ]

This feature is not compatible with partial credit, correct? Working on Sakai 11 release notes.

Comment by Alan Regan [ 13-Jul-2016 ]

We have both partial credit and minimum point value enabled on our production system; they can co-exist. The value of minimum point value is that it applies to more than just multiple choice questions. It works with Fill-in-the-blank, Matching, Multiple Choice, Numeric Response, Short Answer/Essay, and True/False. The feature was created before other types, like Extended Matching, were added to Tests & Quizzes.

Having "minimum point value" available for Short Answer/Essay is a big hit here for "quick write" activities to encourage reflection on a day's learning and take attendance/participation.

Generated at Wed Sep 18 16:25:59 CDT 2019 using Jira 8.0.3#800011-sha1:073e8b433c2c0e389c609c14a045ffa7abaca10d.