Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-41585

Video playback connection error with Firefox and load balancer pass-through settings

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: OPEN
    • Priority: Major
    • Resolution: Non-Issue
    • Affects Version/s: 12.5
    • Fix Version/s: None
    • Component/s: Content, Lessons, Resources
    • Labels:
      None
    • Test Plan:
      Hide

      Procedure 1 (with LB)

      1. configure Load Balancer with "pass-through" configuration
      2. open Firefox 65+
      3. open the Resources tool
      4. upload the file Paperman Disney Short _ Disney Short Films.mp4
      5. after the upload completes click the "Paperman Disney Short _ Disney Short Films.mp4" link
      6. Firefox should begin playing the video in a new tab
      7. after about 45s, Firefox throws a secure_connection_fail.png error

      procedure 2

      1. do not use a Load Balancer
      2. upload videos of 5m and 10+ minutes
      3. try to view the video
      4. network_error_no_LB.png may occur, but the audio keeps playing
      Show
      Procedure 1 (with LB) configure Load Balancer with "pass-through" configuration open Firefox 65+ open the Resources tool upload the file Paperman Disney Short _ Disney Short Films.mp4 after the upload completes click the "Paperman Disney Short _ Disney Short Films.mp4" link Firefox should begin playing the video in a new tab after about 45s, Firefox throws a secure_connection_fail.png error procedure 2 do not use a Load Balancer upload videos of 5m and 10+ minutes try to view the video network_error_no_LB.png may occur, but the audio keeps playing

      Description

      https://groups.google.com/a/apereo.org/forum/#!topic/sakai-dev/A3XtReNvWT4

      Just wondering if anyone has seen any issues with viewing/streaming video via the content resources tool URL in Firefox 60+ behind a load balancer? e.g.

      /access/content/group/abc1234d-a123-456c-7e89-123b552b6733/MyVideo.mp4

      What's happening is that "some" .mp4 videos will fail with a "Connection Failed" error in Firefox after playing for around 45s.

      This problem started happening on our test system after we configured our F5 load balancer with a pass-through configuration to route incoming and outgoing traffic through the load balancer. Previously, outgoing traffic was sent directly back to the client (nPath config) and the problem did not happen.

      I suspect that there's some combination of Firefox, Sakai, and the new Load Balancer config. that's causing the connection to fail mid stream.

      However, what's complicated is that in the past when Firefox 60 first came out, this problem also happened when the .mp4 file was served by apache only and Sakai was not involved at all. But with the most recent Firefox 65.0.2, when the file is served by apache, the problem went away, so Firefox must have improved something in newer versions. But... the problem still happens with the lastest FF when the file is served up by Sakai through the load balancer.

      Also, we're using Apache / modjk / tomcat but I've confirmed that those are not involved by running a vanilla tomcat project (no Sakai code at all) with the same apache / modjk config and the videos don't fail. Also note that with this basic setup, the older FF 60.0 also still fails, so that indicates as I mentioned before that FF must have improved something with 65.0. But unfortunately that leaves the Sakai code as being involved.

      What's even stranger is that the problem happens with some .mp4s and not others. In particular .mp4s downloaded from youtube fail. But mp4s I created with DaVinci Resolve did not fail. But what I noticed is that the failing .mp4s seem to fail while streaming (I think), but the videos I created, appear to be completely downloaded before it plays, which I suspect is why they don't fail.

      What looks like is happening, is that both files are sent to

      BaseContentService.java

      protected IOException copyRange(InputStream istream, OutputStream ostream, start, long end)
      

      and at some point the mp4 from youtube fails in

      if (bytesToRead >= len) \{
      
      ostream.write(buffer, 0, len);
      
      bytesToRead -= len;
      
      }
      

       

      where the ostream.write() fails and throws an error

      org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

       
      Also, further up the chain (in BaseContentService.java) there's some code that sets the response headers to Accept-Ranges: none

      e.g. there's a comment

      				// If there is a direct link to the asset, no sense streaming it.
      				// Send the asset directly to the load-balancer or to the client
      				URI directLinkUri = m_storage.getDirectLink(resource);
      
      				ArrayList<Range> ranges = parseRange(req, res, len);
      				if (directLinkUri != null || req.getHeader("Range") == null || (ranges == null) || (ranges.isEmpty())) {
      					res.addHeader("Accept-Ranges", "none");
      

       

      which I think is also involved. But what's stranger, is that despite this header, the failing mp4 still appears to stream... at least it plays right away when it's loaded. vs. the mp4 I created will show a spinner icon while the video is loading before it plays.

      (Another note, in Chrome, it will make an initial request where Sakai will set the Accept-Ranges: none, but then Chrome makes another request right away that contains a "Range" header, so Sakai will instead set the response header to Accept-Ranges: bytes and the video streams just fine.)

       

      Thanks and sorry that this is so complicated, but it's been quite an interesting problem! But I'm still confused about where the main point of failure is... the Sakai streaming code, Firefox, or the Load Balancer?

        Gliffy Diagrams

          Zeplin

            Attachments

              Issue Links

                Activity

                  People

                  Assignee:
                  Unassigned Unassigned
                  Reporter:
                  austinUH Austin
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved:

                      Git Integration