Uploaded image for project: 'Sakai'
  1. Sakai
  2. SAK-41585

Video playback connection error with Firefox and load balancer pass-through settings

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: CLOSED
    • Priority: Major
    • Resolution: Non-Issue
    • Affects Version/s: 12.5
    • Fix Version/s: None
    • Component/s: Content, Lessons, Resources
    • Labels:
      None
    • Test Plan:
      Hide
      1. configure Load Balancer with "pass-through" configuration
      2. open Firefox 65+
      3. open the Resources tool
      4. upload the file Paperman Disney Short _ Disney Short Films.mp4
      5. after the upload completes click the "Paperman Disney Short _ Disney Short Films.mp4" link
      6. Firefox should begin playing the video in a new tab
      7. after about 45s, Firefox throws a "connection failed" error
      Show
      configure Load Balancer with "pass-through" configuration open Firefox 65+ open the Resources tool upload the file Paperman Disney Short _ Disney Short Films.mp4 after the upload completes click the "Paperman Disney Short _ Disney Short Films.mp4" link Firefox should begin playing the video in a new tab after about 45s, Firefox throws a "connection failed" error

      Description

      https://groups.google.com/a/apereo.org/forum/#!topic/sakai-dev/A3XtReNvWT4

      Just wondering if anyone has seen any issues with viewing/streaming video via the content resources tool URL in Firefox 60+ behind a load balancer? e.g.

      /access/content/group/abc1234d-a123-456c-7e89-123b552b6733/MyVideo.mp4

      What's happening is that "some" .mp4 videos will fail with a "Connection Failed" error in Firefox after playing for around 45s.

      This problem started happening on our test system after we configured our F5 load balancer with a pass-through configuration to route incoming and outgoing traffic through the load balancer. Previously, outgoing traffic was sent directly back to the client (nPath config) and the problem did not happen.

      I suspect that there's some combination of Firefox, Sakai, and the new Load Balancer config. that's causing the connection to fail mid stream.

      However, what's complicated is that in the past when Firefox 60 first came out, this problem also happened when the .mp4 file was served by apache only and Sakai was not involved at all. But with the most recent Firefox 65.0.2, when the file is served by apache, the problem went away, so Firefox must have improved something in newer versions. But... the problem still happens with the lastest FF when the file is served up by Sakai through the load balancer.

      Also, we're using Apache / modjk / tomcat but I've confirmed that those are not involved by running a vanilla tomcat project (no Sakai code at all) with the same apache / modjk config and the videos don't fail. Also note that with this basic setup, the older FF 60.0 also still fails, so that indicates as I mentioned before that FF must have improved something with 65.0. But unfortunately that leaves the Sakai code as being involved.

      What's even stranger is that the problem happens with some .mp4s and not others. In particular .mp4s downloaded from youtube fail. But mp4s I created with DaVinci Resolve did not fail. But what I noticed is that the failing .mp4s seem to fail while streaming (I think), but the videos I created, appear to be completely downloaded before it plays, which I suspect is why they don't fail.

      What looks like is happening, is that both files are sent to

      BaseContentService.java

      protected IOException copyRange(InputStream istream, OutputStream ostream, start, long end)
      

      and at some point the mp4 from youtube fails in

      if (bytesToRead >= len) \{
      
      ostream.write(buffer, 0, len);
      
      bytesToRead -= len;
      
      }
      

       

      where the ostream.write() fails and throws an error

      org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

       
      Also, further up the chain (in BaseContentService.java) there's some code that sets the response headers to Accept-Ranges: none

      e.g. there's a comment

      				// If there is a direct link to the asset, no sense streaming it.
      				// Send the asset directly to the load-balancer or to the client
      				URI directLinkUri = m_storage.getDirectLink(resource);
      
      				ArrayList<Range> ranges = parseRange(req, res, len);
      				if (directLinkUri != null || req.getHeader("Range") == null || (ranges == null) || (ranges.isEmpty())) {
      					res.addHeader("Accept-Ranges", "none");
      

       

      which I think is also involved. But what's stranger, is that despite this header, the failing mp4 still appears to stream... at least it plays right away when it's loaded. vs. the mp4 I created will show a spinner icon while the video is loading before it plays.

      (Another note, in Chrome, it will make an initial request where Sakai will set the Accept-Ranges: none, but then Chrome makes another request right away that contains a "Range" header, so Sakai will instead set the response header to Accept-Ranges: bytes and the video streams just fine.)

       

      Thanks and sorry that this is so complicated, but it's been quite an interesting problem! But I'm still confused about where the main point of failure is... the Sakai streaming code, Firefox, or the Load Balancer?

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  austinUH Austin
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Git Source Code