Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-2015

Foreign Character Encoding Issue

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: CLOSED
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.4.0, 2.4.1
    • Fix Version/s: 2.5.0
    • Labels:
      None
    • Environment:
      MySQL 4.1 and RHEL 3 or Mac OS X 10.4

      Description

      Hello,

      Below are our notes on research an ongoing Unicode issue.

      Discovered in Assignments and Resources
      Manifestation - Assignments:

      • If a user creates an assignment with a Unicode character in the Title field, the assignment does not show up in the list of assignments
      • The catalina.out log reveals a XML parsing error.
      • The MySQL database field contains an incomplete XML file that ends at the point where the Unicode character was found.

      Manifestation - Resources:

      • If a file upload contains a foreign character in the file name, the Resources tool may become unstable throughout the system and be unavailable to all sites.

      Comments

      • When the form used by the "New Assignment" page is processed, it generates an entire XML file and stores it in the database.
      • Examples are from the from assignment_content table. Other tables tables may be affected as well.
      • Example good record:
        <?xml version="1.0" encoding="UTF-8"?>
        <content allowattach="true"
        context="b32cfe66-9ecc-4cba-0043-1e0df90222c1"
        datecreated="20050826171605824" groupproject="true" honorpledge="1"
        id="54293ba9-f85d-4d17-80ca-bbdd837754aa" indivgraded="false"
        instructions="SMOpcsOpIGlzIGEgbsOpdyBzdHJpbmcu "
        instructions-html="SMOpcsOpIGlzIGEgbsOpdyBzdHJpbmcu "
        lastmod="20050826171605824" numberofattachments="0"
        numberofauthors="0" releasegrades="false" scaled_maxgradepoint="0"
        submissiontype="3" title="Foreign Title" typeofgrade="1">
        <properties>
        <property enc="BASE64" name="CHEF:creator" value="YWRtaW4= "/>
        <property enc="BASE64" name="CHEF:modifiedby" value="YWRtaW4= "/>
        <property enc="BASE64" name="DAV:getlastmodified" value="MjAwNTA4MjYxNzE2MDgxMzU= "/>
        <property enc="BASE64" name="DAV:creationdate" value="MjAwNTA4MjYxNzE2MDU4MjQ= "/>
        </properties>
        </content>

      Example bad record:
      <?xml version="1.0" encoding="UTF-8"?>
      <content allowattach="true"
      context="b32cfe66-9ecc-4cba-0043-1e0df90222c1"
      datecreated="20050826171447016" groupproject="true" honorpledge="1"
      id="ac84f7f7-e5a2-47aa-0090-eb72fe7b073b" indivgraded="false"
      instructions="VGhpcyBpcyBhIGZvcmVpZ24tbMO4w7hraW5nIHDDqmPDvGNlIG9mIHTDqXh0Lg== "
      instructions-html="VGhpcyBpcyBhIGZvcmVpZ24tbMO4w7hraW5nIHDDqmPDvGNlIG9mIHTDqXh0Lg== "
      lastmod="20050826171447016" numberofattachments="0"
      numberofauthors="0" releasegrades="false" scaled_maxgradepoint="0"
      submissiontype="3" title="Foreign Titl

      • Note that in the "bad record" that the data is truncated at the point where the original field data was "Foreign TitlĂ©"
      • The "instructions" and "instructions-html" data values are being BASE64 encoded and thus are not vulnerable to this error.
      • The "title" data is stored as-entered by the user.
      • The data from the entire "New Assignment" form is being serialized into XML string and then stored in the database.
      • Upon retrieval of the data for display (as when refreshing the screen immediately after submission), the catalina.out log file shows the error and the new Assignment does not display in the list of Assignments. There is no warning to the user.
      • The error is: "WARN: readDocumentFromString: org.xml.sax.SAXParseException: XML document structures must start and end within the same entity." and "[Fatal Error] :8:67: XML document structures must start and end within the same entity."
      • The character set for the MySQL table is UTF-8

        Gliffy Diagrams

          Zeplin

            Attachments

              Issue Links

                Activity

                  People

                  Assignee:
                  Unassigned Unassigned
                  Reporter:
                  jwilliamson@oid.ucla.edu James Williamson (Inactive)
                  Votes:
                  2 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved:

                      Git Integration