Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-36412

Matching questions with distractors - Incorrect scoring and display of answer keys in several areas

    XMLWordPrintable

    Details

    • 12 status:
      Resolved
    • Previous Issue Keys:
      SAM-1579
    • Test Plan:
      Hide

      Testing Steps #1 - Scoring:

      1. In a site with at least 5 students, create a test with Matching questions that have the following parameters (many choices are used to demonstrate correct scoring in each instance, and various combinations of correct answers to confirm correct scoring in the case of each question) - I've attached here an assessment with the parameters listed, but question authoring/editing screens should be observed to make sure None of the Above is correctly displaying and NOT *distractor* matchingquestions-somehavedistractors-somenodistractor.zip - also, exported assessment will not retain GB setting on import:
        1. Include at least 5 Choices total. One Match is None of the Above (aka distractor).
        2. Include at least 6 Choices total. TWO Matches are None of the Above (aka distractor).
        3. Include at least 6 Choices total. THREE Matches are None of the Above.
        4. Include at least 4 Choices. NO Match is None of the Above.
        5. Include at least 5 Choices. At least TWO Choices have the SAME Match - (to create this match, select Existing > # from the Match drop-down)
      2. After creating the questions, review the Answer Key on the Questions editing screen and make sure they look more or less correct (you'll see a problem with Question #5's answer key, and any question where you entered two choices with distractor answers consecutively).
      3. In the test Settings, select the following settings under Grading and Feedback:
        • Send grades to gradebook
        • Feedback Delivery: Feedback on Submission
        • Release Questions and the following, and select ALL feedback options.
      4. Publish the test.
      5. Take the test created as Student 1 and select correct answers for everything (100%). Submit.
      6. After submitting, make sure your score's correct in each of the following screens:
        • samigo main page, under the Submitted Assessments list
        • samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.)
        • Gradebook
      7. Log in as Student 2. Select a combination of correct and incorrect answers. Submit the test.
      8. After submitting, calculate your score and make sure it's correct in each of the following screens:
        • samigo main page, under the Submitted Assessments list
        • samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.)
        • Gradebook
      9. Log in as Student 3. Answer everything, but incorrectly. Submit.
      10. After submitting, make sure you failed in each of the following screens:
        • samigo main page, under the Submitted Assessments list
        • samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.)
        • Gradebook
      11. Log in Student 4. For questions WITH distractors, select correct responses to everything EXCEPT the items for which distractor answers are correct. Get the distractors wrong. Make at least two distractors wrong by having NO RESPONSE to them. For questions WITHOUT distractors, select some right and some wrong answers, getting at least 2 wrong by having NO RESPONSE.
      12. After submitting, calculate your score and make sure it's correct for all questions in each of the following screens:
        • samigo main page, under the Submitted Assessments list
        • samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.)
        • Gradebook
      13. Log in as Student 5. Don't respond to any questions and submit.
      14. After submitting, make sure you failed in each of the following screens:
        • samigo main page, under the Submitted Assessments list
        • samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.)
        • Gradebook
      15. Log in as instructor and go to samigo > Published Copies.
      16. View the Total Scores for the assessment and confirm that all students' scores are correct.
      17. Click on each student's name and confirm their submission is displaying correctly, with correct answer keys (See also Testing Steps #2 and #3 - Feedback for more details about correct display.)
      18. Click Questions.
      19. Go to each question and confirm that the students' scores and answer keys are correct.

      Testing Steps #2 - Feedback:

      Issue 1 screen shot:

      Feedback Issue 1 testing (description below is what it should look like when issue fixed):

      1. Create a test with a Matching question that has at least one None of the Above answer. Include at least 5 Choices total.
      2. In the test Settings, select the following settings under Grading and Feedback:
        • Feedback Delivery: Immediate Feedback
        • Release Questions and the following:
          • Student Response
          • Correct Response
      3. Publish the test.
      4. Log in as student and take the test.
      5. View the distractor question and click Show Feedback to see the Answer Key.
      6. The answer numbers are shown in correct order (i.e. 1, 2, 3, 4, 5), and are correct.
      7. Select some correct and some incorrect answers, then click Show Feedback again. Confirm the Answer Key is correct.

      Issue 2 screen shot:

      Feedback Issue 2 testing (description below is what it should look like when issue fixed):

      1. Take the test created under Issue 1 testing steps above as a student (Student 1) and select answers for all parts of the question (some correct and some incorrect). Submit.
      2. Log in as a second student (Student 2). Select ONLY the distractor answer for the part of the question where it would be a correct answer. Submit the test.
      3. Log in as a third student (Student 3), do NOT select ANY answer options. Submit.
      4. Log in as instructor and go to samigo > Published Copies.
      5. View the Scores for the assessment above.
      6. Click Questions.
      7. Go to the distractor question and review the feedback, confirm all students' answer keys are correct per below:
        • Student 1: All answers entered and display with check marks and red Xs appropriately for correct/incorrect selections. Calculate the score and make sure it's correct too.
        • Student 2: The item for which you selected None of the Above answer says None of the Above and all the items for which you DIDN'T select answers should say No Response. Calculate the score and make sure it's correct too.
        • Student 3: All answers display No Response. Score should be 0.
      8. Take note of what the Answer Key near the top of the page on the individual question screen says, and compare it to the Student Response column for each student.
      9. For each student, click on their name (either on the Questions screen or Total Scores screen), check the Answer Key for the distractor question and make sure it displays correctly.
      10. Also, confirm the following on each student's individual grading screen:
        • In the display of the question text, there's a space in between each letter and its corresponding choice, i.e,:
          • CORRECT:
            A. [answer]
            B. [answer]
            C. None of the Above
          • INCORRECT:
            A. [answer]
            B. [answer]
            C.None of the Above
        • Each answer displays in correct sequential order:
          1:A, 2:B, 3:C, etc.
        • NO item in the Answer Key displays a space in between the colon and the letter, i.e.
          • CORRECT:
            1:A
          • INCORRECT:
            1: A

      Testing Steps #3 - Feedback also:

      1. Take an assessment with a matching question
      2. Don't answer anything
      3. Submit
      4. As instructor access scores for the assessment
      5. go to the questions tab
      6. verify existence of all No Response answers in the Student Response column

      Testing Steps #4 - Export Scores:

      1. After doing Testing Steps #1, you should have a number of students' responses to matching questions; you can use these.
      2. As instructor, go to samigo > Published Copies > Scores for the corresponding assessment.
      3. Click Export.
      4. Click the other Export button to confirm.
      5. You should get an Excel file that lists student ids, scores for the test, and responses to questions.
      6. Compare the results in the Excel file to the results in the Questions screen for each question, and make sure each student's responses match up. Note: The sort order of responses in the spreadsheet will look odd, but I think this has to do with a separate randomization issue - there is an unresolved jira about this at UVA which will need to be copied to a new jira here with dependency.
      Show
      Testing Steps #1 - Scoring: In a site with at least 5 students, create a test with Matching questions that have the following parameters (many choices are used to demonstrate correct scoring in each instance, and various combinations of correct answers to confirm correct scoring in the case of each question) - I've attached here an assessment with the parameters listed, but question authoring/editing screens should be observed to make sure None of the Above is correctly displaying and NOT * distractor * matchingquestions-somehavedistractors-somenodistractor.zip - also, exported assessment will not retain GB setting on import: Include at least 5 Choices total. One Match is None of the Above (aka distractor). Include at least 6 Choices total. TWO Matches are None of the Above (aka distractor). Include at least 6 Choices total. THREE Matches are None of the Above . Include at least 4 Choices . NO Match is None of the Above . Include at least 5 Choices . At least TWO Choices have the SAME Match - (to create this match, select Existing > # from the Match drop-down) After creating the questions, review the Answer Key on the Questions editing screen and make sure they look more or less correct (you'll see a problem with Question #5's answer key, and any question where you entered two choices with distractor answers consecutively). In the test Settings , select the following settings under Grading and Feedback : Send grades to gradebook Feedback Delivery : Feedback on Submission Release Questions and the following , and select ALL feedback options. Publish the test. Take the test created as Student 1 and select correct answers for everything (100%). Submit. After submitting, make sure your score's correct in each of the following screens: samigo main page, under the Submitted Assessments list samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.) Gradebook Log in as Student 2. Select a combination of correct and incorrect answers. Submit the test. After submitting, calculate your score and make sure it's correct in each of the following screens: samigo main page, under the Submitted Assessments list samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.) Gradebook Log in as Student 3. Answer everything, but incorrectly. Submit. After submitting, make sure you failed in each of the following screens: samigo main page, under the Submitted Assessments list samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.) Gradebook Log in Student 4. For questions WITH distractors, select correct responses to everything EXCEPT the items for which distractor answers are correct. Get the distractors wrong. Make at least two distractors wrong by having NO RESPONSE to them. For questions WITHOUT distractors, select some right and some wrong answers, getting at least 2 wrong by having NO RESPONSE. After submitting, calculate your score and make sure it's correct for all questions in each of the following screens: samigo main page, under the Submitted Assessments list samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.) Gradebook Log in as Student 5. Don't respond to any questions and submit. After submitting, make sure you failed in each of the following screens: samigo main page, under the Submitted Assessments list samigo > Feedback for the test (See also Testing Steps #2 and #3 - Feedback for more details about correct display.) Gradebook Log in as instructor and go to samigo > Published Copies . View the Total Scores for the assessment and confirm that all students' scores are correct. Click on each student's name and confirm their submission is displaying correctly, with correct answer keys (See also Testing Steps #2 and #3 - Feedback for more details about correct display.) Click Questions . Go to each question and confirm that the students' scores and answer keys are correct. Testing Steps #2 - Feedback: Issue 1 screen shot: Feedback Issue 1 testing (description below is what it should look like when issue fixed): Create a test with a Matching question that has at least one None of the Above answer. Include at least 5 Choices total. In the test Settings , select the following settings under Grading and Feedback : Feedback Delivery : Immediate Feedback Release Questions and the following : Student Response Correct Response Publish the test. Log in as student and take the test. View the distractor question and click Show Feedback to see the Answer Key . The answer numbers are shown in correct order (i.e. 1, 2, 3, 4, 5), and are correct. Select some correct and some incorrect answers, then click Show Feedback again. Confirm the Answer Key is correct. Issue 2 screen shot: Feedback Issue 2 testing (description below is what it should look like when issue fixed): Take the test created under Issue 1 testing steps above as a student (Student 1) and select answers for all parts of the question (some correct and some incorrect). Submit. Log in as a second student (Student 2). Select ONLY the distractor answer for the part of the question where it would be a correct answer. Submit the test. Log in as a third student (Student 3), do NOT select ANY answer options. Submit. Log in as instructor and go to samigo > Published Copies . View the Scores for the assessment above. Click Questions . Go to the distractor question and review the feedback, confirm all students' answer keys are correct per below: Student 1: All answers entered and display with check marks and red Xs appropriately for correct/incorrect selections. Calculate the score and make sure it's correct too. Student 2: The item for which you selected None of the Above answer says None of the Above and all the items for which you DIDN'T select answers should say No Response . Calculate the score and make sure it's correct too. Student 3: All answers display No Response . Score should be 0. Take note of what the Answer Key near the top of the page on the individual question screen says, and compare it to the Student Response column for each student. For each student, click on their name (either on the Questions screen or Total Scores screen), check the Answer Key for the distractor question and make sure it displays correctly. Also, confirm the following on each student's individual grading screen: In the display of the question text, there's a space in between each letter and its corresponding choice, i.e,: CORRECT: A. [answer] B. [answer] C. None of the Above INCORRECT: A. [answer] B. [answer] C.None of the Above Each answer displays in correct sequential order: 1:A, 2:B, 3:C, etc. NO item in the Answer Key displays a space in between the colon and the letter, i.e. CORRECT: 1:A INCORRECT: 1: A Testing Steps #3 - Feedback also: Take an assessment with a matching question Don't answer anything Submit As instructor access scores for the assessment go to the questions tab verify existence of all No Response answers in the Student Response column Testing Steps #4 - Export Scores: After doing Testing Steps #1, you should have a number of students' responses to matching questions; you can use these. As instructor, go to samigo > Published Copies > Scores for the corresponding assessment. Click Export . Click the other Export button to confirm. You should get an Excel file that lists student ids, scores for the test, and responses to questions. Compare the results in the Excel file to the results in the Questions screen for each question, and make sure each student's responses match up. Note: The sort order of responses in the spreadsheet will look odd, but I think this has to do with a separate randomization issue - there is an unresolved jira about this at UVA which will need to be copied to a new jira here with dependency.

      Description

      • No points taken off if distractors are incorrectly matched. The points are divided by the correct number of matches. ex:
        • 6 matches plus 2 distractors = 6 points
        • If the student got 3 correct matches, but got one of the distractors wrong, the student still receives 3 points.
      • On the questions screen, if the student does not make an entry (left as 'Select'), there is nothing displayed under the student response unless there is a distractor, in which case the distractor alone shows.
      • The answer keys for matching questions with distractors are not displayed consistently between the edit screens and the score screens
        • If you create an assessment that allows Immediate Feedback (i.e. student can click a Show Feedback link while in process of taking the quiz to view feedback), the answer key for the distractor question inside the quiz has the answers out of numerical order. It looks weird, but the answers shown appear to be correct in each case.
        • When the instructor views the individual question screen for a distractor question in Scores > Questions, and a student has NOT selected answer options for each of the parts of the question, the Answer Key is INCORRECT.
        • Some minor typographical issues (spaces displaying inconsistently between letters and matches).
      • The answer keys for matching questions with distractors are not displayed correctly on the instructor's "Edit" UI, when any choice has either 'Existing' as an answer, or if there are two consecutive distractors in any sequence, or if there is a single distractor in the middle of the sequence

        Gliffy Diagrams

          Attachments

          1. 4-E-ShouldNOTbespace.png
            2 kB
            Tiffany Stull
          2. DistractorFeedbackInsideAssessment.png
            16 kB
            Tiffany Stull
          3. Distractors2.png
            66 kB
            Andrea Schmidt
          4. E-Noneoftheabove-needsspace.png
            16 kB
            Tiffany Stull
          5. InstructorAnswerKeyDistractor.png
            22 kB
            Tiffany Stull
          6. NO_RESPONSE.JPG
            158 kB
            Prabhleen Kaur
          7. NO1.JPG
            160 kB
            Prabhleen Kaur
          8. Student3-correctanswersfordistractor.png
            25 kB
            Tiffany Stull

            Issue Links

              Activity

                People

                • Assignee:
                  bjones86 Brian Jones
                  Reporter:
                  agschmidt Andrea Schmidt
                • Votes:
                  2 Vote for this issue
                  Watchers:
                  12 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Git Source Code