Affects Version/s: 12.5
Fix Version/s: None
Procedure 1 (with LB)
- configure Load Balancer with "pass-through" configuration
- open Firefox 65+
- open the Resources tool
- upload the file Paperman Disney Short _ Disney Short Films.mp4
- after the upload completes click the "Paperman Disney Short _ Disney Short Films.mp4" link
- Firefox should begin playing the video in a new tab
- after about 45s, Firefox throws a secure_connection_fail.png error
Procedure 1 (with LB) configure Load Balancer with "pass-through" configuration open Firefox 65+ open the Resources tool upload the file Paperman Disney Short _ Disney Short Films.mp4 after the upload completes click the "Paperman Disney Short _ Disney Short Films.mp4" link Firefox should begin playing the video in a new tab after about 45s, Firefox throws a secure_connection_fail.png error procedure 2 do not use a Load Balancer upload videos of 5m and 10+ minutes try to view the video network_error_no_LB.png may occur, but the audio keeps playing
- do not use a Load Balancer
- upload videos of 5m and 10+ minutes
- try to view the video
- network_error_no_LB.png may occur, but the audio keeps playing
Just wondering if anyone has seen any issues with viewing/streaming video via the content resources tool URL in Firefox 60+ behind a load balancer? e.g.
What's happening is that "some" .mp4 videos will fail with a "Connection Failed" error in Firefox after playing for around 45s.
This problem started happening on our test system after we configured our F5 load balancer with a pass-through configuration to route incoming and outgoing traffic through the load balancer. Previously, outgoing traffic was sent directly back to the client (nPath config) and the problem did not happen.
I suspect that there's some combination of Firefox, Sakai, and the new Load Balancer config. that's causing the connection to fail mid stream.
However, what's complicated is that in the past when Firefox 60 first came out, this problem also happened when the .mp4 file was served by apache only and Sakai was not involved at all. But with the most recent Firefox 65.0.2, when the file is served by apache, the problem went away, so Firefox must have improved something in newer versions. But... the problem still happens with the lastest FF when the file is served up by Sakai through the load balancer.
Also, we're using Apache / modjk / tomcat but I've confirmed that those are not involved by running a vanilla tomcat project (no Sakai code at all) with the same apache / modjk config and the videos don't fail. Also note that with this basic setup, the older FF 60.0 also still fails, so that indicates as I mentioned before that FF must have improved something with 65.0. But unfortunately that leaves the Sakai code as being involved.
What's even stranger is that the problem happens with some .mp4s and not others. In particular .mp4s downloaded from youtube fail. But mp4s I created with DaVinci Resolve did not fail. But what I noticed is that the failing .mp4s seem to fail while streaming (I think), but the videos I created, appear to be completely downloaded before it plays, which I suspect is why they don't fail.
What looks like is happening, is that both files are sent to
and at some point the mp4 from youtube fails in
where the ostream.write() fails and throws an error
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
Also, further up the chain (in BaseContentService.java) there's some code that sets the response headers to Accept-Ranges: none
e.g. there's a comment
which I think is also involved. But what's stranger, is that despite this header, the failing mp4 still appears to stream... at least it plays right away when it's loaded. vs. the mp4 I created will show a spinner icon while the video is loading before it plays.
(Another note, in Chrome, it will make an initial request where Sakai will set the Accept-Ranges: none, but then Chrome makes another request right away that contains a "Range" header, so Sakai will instead set the response header to Accept-Ranges: bytes and the video streams just fine.)
Thanks and sorry that this is so complicated, but it's been quite an interesting problem! But I'm still confused about where the main point of failure is... the Sakai streaming code, Firefox, or the Load Balancer?