|
In revision 19616, I added a special logger to look for grade table data contention instances and causes.
It can be selectively turned on with the following log4j properties: log4j.logger.org.sakaiproject.tool.gradebook.business.impl.GradebookManagerHibernateImpl.GB_DATA=debug log4j.logger.org.sakaiproject.component.gradebook.GradebookServiceHibernateImpl.GB_DATA=debug The messages will also show up if the parent class's logger is at a "debug" level or lower. After comparing the benefits of various strategies, I finally decided to try the most radical approach and completely remove calculated course grades from the database. Instead, they're now calculated on the fly when they're needed.
I moved this change into the Sakai trunk in revision 20001. If there are no bugs, the switch should be invisible to existing users. Behind the scenes, however, this is a major modification of all business logic relating to course grades. Before this check-in, users could receive contention errors (and lose work) in scenarios such as the following: * Grader A is scoring Student X on Quiz 1 at around the same time Grader B is scoring Student X on Quiz 2. * Grader A is scoring Student X on Quiz 1 at around the same time Grader B is scoring Student Y on Quiz 1. * Grader A is scoring Student X on Quiz 1 at around the same time Instructor is overriding Student X's course grade. * Student X is submitting a Gradebook-linked assignment at around the same time Grader A is scoring Student X on a different assignment. After this change, the ONLY time data contention should occur is when two graders are trying to change the same score for the same student at around the same time. By rearranging the business logic, I've tried to minimize the negative side of the change. However, the student's view of automatically calculated course grades (when they've been released to students but have not been explicitly overridden by the instructor), and the display of gradebook-wide mean values in the instructor's Course Grade Details view will both require more reading from the database. Given students' anxiety over discovering final grades as quickly possible, the student view is much more likely to be more of a performance issue, and it would be good to plan for some serious load testing over the next month or so. (And some serious QA testing, too.) (As a little background, the original Gradebook design was influenced by our belief that we'd soon be adding grade curving as a feature, and by our experiences dealing with rabid student curiosity in the UC Berkeley pilot project. However, with the addition of Gradebook programmatic clients such as Samigo, Assignments, and Message Center, and with the addition of support for course sections and GSIs/TAs, reducing data contention when scoring has turned out to be a much higher priority than supporting curved grading.) If we run into performance problems around the grades, the most obvious next step I see is to break the course grades and assignment scores into separate database tables, and to break the course grade definitions and assignment definitions into separate database tables. Leaving them all together might make them too difficult to optimize.
I didn't take that step now, though, because that would require massive changes to existing production databases. The changes I checked in are fully compatible with existing data. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Task
Closed
Critical
Analyze and reduce data contention in Gradebook
* Add more log messages specifically to help in analyzing deadlocks.
* When it's fairly certain that the Gradebook will be updating a selected record, use SELECT FOR UPDATE to reduce the chances of deadlock. We can make that decision much smarter if we change the application logic for Assignment Details and Course Grade to:
- Remember the old score/grade values in the bean.
- Compare the form's old values to the form's new values and use that to decide what has to be updated (rather than comparing the form's new values to what's currently in the DB).
* Consider breaking the multiple-update transactions into multiple single-update transactions. (Ask our local DBAs for advice about this.)
* This is more of an annoyance when analyzing logs rather than a real performance drain, but I see truly ridiculous numbers of read-only transactions coming from getGradebookUid and isSiteMemberInRole, probably due to JSF calling stuff like rendered="#{overviewBean.userAbleToGradeAll}" a ridiculous number of times. Look into some simple request-scoped caching to take care of this.
* Hardest one, and possibly the most effective one: Redesign our logic to postpone the DB updating of CourseGradeRecord's PointsEarned and SortGrade as long as possible.