Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-38985

serious performance problem with counting null file_size

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: CLOSED
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.x, 2.7.x
    • Fix Version/s: 2.7.x, 2.8.x
    • Component/s: Kernel
    • Labels:
      None
    • Previous Issue Keys:
      SAK-18078, KNL-427

      Description

      In DbContentService, there is code to see whether you can use the FILE_SIZE field in CONTENT_RESOURCE. The code has really bad effects.

      The problem is that it checks FILE_SIZE IS NULL. FILE_SIZE is not indexed, so this does a full table scan. On our system this takes several minutes if the data isn't in memory. The scan is done at startup and every 20 minutes until the check succeeds. Here are the problems:

      1) It slows down startup significantly.
      2) We had a few bogus entries in CONTENT_RESOURCE, going back for years. The conversion utility was unable to compute FILE_SIZE for them. So this check was done every 20 minutes in production. [I've fixed that by finding the entries and setting FILE_SIZE to 0.]
      3) The code is

      long now = TimeService.newTime().getTime();
      if(now > filesizeColumnCheckExpires)
      {
      // cached value has expired – time to renew
      int filesizeColumnCheckNullCount = countNullFilesizeValues();
      if(filesizeColumnCheckNullCount > 0)
      {
      filesizeColumnCheckExpires = now + TWENTY_MINUTES;

      This looks fine, until you realize that countNullFilesizeValues can take several minutes. During that time, any other threads that call readyToUseFilesizeColumn will also find now > filesizeColumnCheckExpires. Thus you can have a large number of threads in this code. This is a performance disaster.

      One approach is a sakai.properties entry to set filesizeColumnReady to true, for sites that know this isn't an issue. Indeed the conversion is enough releases ago that you might want to default it to true.

        Gliffy Diagrams

          Zeplin

            Attachments

              Issue Links

                Activity

                  People

                  Assignee:
                  k1team KERNEL TEAM (Inactive)
                  Reporter:
                  hedrick Charles Hedrick
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  0 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved:

                      Git Integration