Skip to content

feat(server): hls with real-time transcoding#28230

Open
mertalev wants to merge 27 commits into
mainfrom
feat/server-hls
Open

feat(server): hls with real-time transcoding#28230
mertalev wants to merge 27 commits into
mainfrom
feat/server-hls

Conversation

@mertalev
Copy link
Copy Markdown
Member

@mertalev mertalev commented May 4, 2026 •
edited
Loading

Description

This PR is the core HLS + transcoding implementation. It allows clients to select a variant and have the server transcode it on the fly. It also allows the client to switch to another variant, in which case it kills the transcode and restarts. Likewise, it restarts transcoding if the user seeks behind the transcode's start point or enough ahead that it's quicker to restart. The implementation is split into an HlsService used by the API worker to handle playlist generation. This service emits websocket events which a TranscodingService (possibly on a different machine) listens and responds to.

It is an alpha feature with several items deferred for later PRs:

  • Client implementations for web and mobile
  • Exposing configuration for some things like supported codecs
  • Advertising the original as a remuxed variant
  • Pre-transcoding the first few seconds to mask start latency
  • Tuning FFmpeg commands for live playback
  • HDR transcoding and remuxing

@mertalev mertalev requested a review from danieldietzler as a code owner May 4, 2026 15:28
@mertalev mertalev requested a review from shenlong-tanwen as a code owner May 4, 2026 15:30
meesfrensel
meesfrensel reviewed May 5, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@meesfrensel meesfrensel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool, I'm really excited for this!

One thing I saw from a quick scan is mentioned in the comments. Would you like an actual review from me or prefer to keep it in the team? I could get to it in the next 2-3 days.

Comment thread server/src/dtos/streaming.dto.ts
Comment thread server/src/constants.ts Outdated
@mertalev
Copy link
Copy Markdown
Member Author

mertalev commented May 5, 2026 •
edited
Loading

@meesfrensel An actual review would be very welcome!

danieldietzler
danieldietzler reviewed May 5, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@danieldietzler danieldietzler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minus the sorcery this is actually really well readable!

Comment thread server/src/repositories/media.repository.ts Outdated
Comment thread server/src/services/hls.service.ts Outdated
Comment thread server/src/services/hls.service.ts Outdated
Comment thread server/src/services/hls.service.ts
Comment thread server/src/services/hls.service.ts Outdated
Comment thread server/src/services/media.service.ts Outdated
Comment thread server/src/utils/media.ts
Comment thread server/src/utils/media.ts
Comment thread server/tsconfig.json
@mertalev mertalev mentioned this pull request May 5, 2026
Merged
meesfrensel
meesfrensel reviewed May 7, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@meesfrensel meesfrensel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the first look, I thought this was (overly) complex, but I agree with Daniel that it's actually quite readable and the implementation makes sense.

Starting a session and then trying to stream a variant playlist with ffplay, the experience is not great with a lot of stuttering. Is that expected? When playing an HLS playlist from the filesystem or some other external URL, I don't see this.

  • How can we make sure playlist generation and the FFmpeg command/settings don't get out of sync in the future? Should there be medium/e2e tests?
  • The playlist generation and HLS command will need to change quite a bit for remuxing the original, right? Not a bad thing, but I wonder where/when the whole keyframe-based segments come in :)
  • Should the client be able to communicate pause/play event on which the transcoding service can pause/resume? Or do you prefer to let this be handled 100% by the backpressure system?
  • The client will implement an onDestroy type handler that DELETEs the HLS session, right? And the lease time and cleanup jobs are there as a second line of defense so to say.

Comment thread server/src/controllers/video-stream.controller.ts Outdated
Comment thread server/src/utils/event.ts Outdated
Comment thread server/src/utils/event.ts Outdated
Comment thread server/src/services/hls.service.ts
Comment thread server/src/services/hls.service.ts
Comment thread server/src/utils/media.ts
Comment thread server/src/services/transcoding.service.ts
Comment thread server/src/services/transcoding.service.ts
Comment thread server/src/services/transcoding.service.ts
Comment thread server/src/constants.ts Outdated
@mertalev
Copy link
Copy Markdown
Member Author

mertalev commented May 7, 2026 •
edited
Loading

Starting a session and then trying to stream a variant playlist with ffplay, the experience is not great with a lot of stuttering. Is that expected?

Do you have default transcoding settings? It also depends on the variant and the input video. It might just be too much for the server to keep up on CPU.

How can we make sure playlist generation and the FFmpeg command/settings don't get out of sync in the future? Should there be medium/e2e tests?

The thing is that getting the "gold" playlist from FFmpeg requires actually transcoding the video, so it would kind of suck for that to be running all the time for local development and CI. The current approach is to run it offline and use its values for the test expectations. Maybe a solution would be to have a script that updates the fixtures to make it easier to keep up-to-date.

The playlist generation and HLS command will need to change quite a bit for remuxing the original, right? Not a bad thing, but I wonder where/when the whole keyframe-based segments come in :)

The playlist generation will definitely need some tweaks, but I think the HLS command itself will be more or less the same.

Should the client be able to communicate pause/play event on which the transcoding service can pause/resume? Or do you prefer to let this be handled 100% by the backpressure system?

Hmm, I think letting backpressure handle it is fine. It's a good thing to have a healthy buffer ready.

The client will implement an onDestroy type handler that DELETEs the HLS session, right? And the lease time and cleanup jobs are there as a second line of defense so to say.

Yup, the DELETE is just a cooperative fast path. If you swipe to the next video, it's good to let the server know so it can kill the old transcode right away. We can't rely on that for cleanup, though, since the client might never send that event for one reason or another.

@mertalev mertalev force-pushed the feat/server-hls branch from 0facae4 to 6ebc5e3 Compare May 7, 2026 22:59
@mertalev mertalev force-pushed the feat/server-hls branch from 6ebc5e3 to c8bdf36 Compare May 7, 2026 23:29
@meesfrensel
Copy link
Copy Markdown
Collaborator

meesfrensel commented May 8, 2026

With VLC it plays smoothly so I'll have to assume that the problem is with ffplay (I can see that the video buffer empties out, even though cpu usage isn't high at all and segments are readily available).

The thing is that getting the "gold" playlist from FFmpeg requires actually transcoding the video, so it would kind of suck for that to be running all the time for local development and CI. (...) Maybe a solution would be to have a script that updates the fixtures to make it easier to keep up-to-date.

I see the problem. Only run on CI when the relevant files have changes? Will be more prone to breakages from refactors and big changes though. A manual script will probably be used by no-one :p

One more comment on the implementation: when stopping playback through vlc, the server kept transcoding (or at least, firing hlssegmentresult websocket events). I noticed the heartbeat did not get sent anymore, which might be expected, but that the transcode continued well past the 30 segment limit seems like unintended behavior.

@mertalev
Copy link
Copy Markdown
Member Author

mertalev commented May 8, 2026

One more comment on the implementation: when stopping playback through vlc, the server kept transcoding (or at least, firing hlssegmentresult websocket events). I noticed the heartbeat did not get sent anymore, which might be expected, but that the transcode continued well past the 30 segment limit seems like unintended behavior.

Hmm, that seems like a bug. I'll take a closer look at this.

@mertalev
Copy link
Copy Markdown
Member Author

mertalev commented May 8, 2026

Should be fixed now :)

@mertalev mertalev mentioned this pull request May 9, 2026
Open
meesfrensel
meesfrensel reviewed May 11, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@meesfrensel meesfrensel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two more issues, these are with the HLS web player from your other PR, but they're server-side issues.

It seems like the (back?)seek detection is not working correctly when the client requests segments as fast as the server produces them (for example, when filling the buffer at the start of playback). But I'm not sure if that's really the cause. The lead number goes into the negative, sometimes even -2, and the transcode restarts sometimes. Logs just say 'stopped transcoding' and starts a new one.

Whether the cpu can keep up is irrelevant to this, because the client will always request some buffer really quickly at the start of playback. The transcode shouldn't restart, that's just overhead.

Log 
[Nest] 24802  - 05/11/2026, 2:26:34 PM   DEBUG [Api:WebsocketRepository~v1ko3a8q] Server event: HlsSessionRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:34 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSessionRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:34 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSessionResult (send)
[Nest] 24802  - 05/11/2026, 2:26:34 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSessionResult (receive)
[Nest] 24802  - 05/11/2026, 2:26:34 PM   DEBUG [Api:LoggingInterceptor~v1ko3a8q] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/main.m3u8 200 24.31ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:34 PM   DEBUG [Api:LoggingInterceptor~honx9s0z] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/ocr 200 6.35ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:34 PM   DEBUG [Api:LoggingInterceptor~1sd7nk7p] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/playlist.m3u8 200 4.05ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:34 PM   DEBUG [Api:WebsocketRepository~sijtz2y9] Server event: HlsHeartbeat (send)
[Nest] 24802  - 05/11/2026, 2:26:34 PM   DEBUG [Api:WebsocketRepository~sijtz2y9] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:34 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:34 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:34 PM     LOG [Microservices:TranscodingService] Starting HLS transcode for asset 315640ae-e935-460a-abb8-1f73991ccc5c variant 1 with command: ffmpeg -nostdin -nostats -i /data/library/admin/2013/2013-12/bbb_sunflower_2160p_60fps_normal.mp4 -c:v hevc -c:a aac -map 0:0 -map_metadata -1 -map 0:2 -g 120 -keyint_min 120 -tag:v hvc1 -preset ultrafast -crf 23 -maxrate 1200k -bufsize 2400k -x265-params no-scenecut=1:no-open-gop=1 -copyts -r 2284320000/38072000 -avoid_negative_ts disabled -f hls -hls_time 2 -hls_list_size 0 -hls_segment_type fmp4 -hls_fmp4_init_filename init.mp4 -hls_segment_options movflags=+frag_discont -hls_flags temp_file -hls_segment_filename /data/encoded-video/33e67826-48fc-4851-bb13-db18af2d221b/0f/93/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_%d.m4s -start_number 0 -vf scale=-2:480 /data/encoded-video/33e67826-48fc-4851-bb13-db18af2d221b/0f/93/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/playlist.m3u8
[Nest] 24710  - 05/11/2026, 2:26:35 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24802  - 05/11/2026, 2:26:35 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24710  - 05/11/2026, 2:26:35 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:35 PM   DEBUG [Api:LoggingInterceptor~sijtz2y9] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/init.mp4 200 1165.08ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:35 PM   DEBUG [Api:WebsocketRepository~v6wx6k0d] Server event: HlsHeartbeat (send)
[Nest] 24710  - 05/11/2026, 2:26:35 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:35 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:35 PM   DEBUG [Api:LoggingInterceptor~v6wx6k0d] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_0.m4s 200 10.83ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:36 PM   DEBUG [Api:LoggingInterceptor~7l7jhk1g] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/10/playlist.m3u8 200 10.48ms ::ffff:127.0.0.1
[Nest] 24710  - 05/11/2026, 2:26:36 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24802  - 05/11/2026, 2:26:36 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24710  - 05/11/2026, 2:26:36 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 1 segments
[Nest] 24802  - 05/11/2026, 2:26:36 PM   DEBUG [Api:WebsocketRepository~ovu9iatv] Server event: HlsHeartbeat (send)
[Nest] 24802  - 05/11/2026, 2:26:36 PM   DEBUG [Api:WebsocketRepository~ovu9iatv] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:36 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:36 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24710  - 05/11/2026, 2:26:36 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:36 PM   DEBUG [Microservices:TranscodingService] Stopped transcoding for session 0f93c55c-5686-4a5a-b605-44e5c1a71476
[Nest] 24710  - 05/11/2026, 2:26:36 PM     LOG [Microservices:TranscodingService] Starting HLS transcode for asset 315640ae-e935-460a-abb8-1f73991ccc5c variant 10 with command: ffmpeg -ss 1.9916666666666667 -nostdin -nostats -i /data/library/admin/2013/2013-12/bbb_sunflower_2160p_60fps_normal.mp4 -c:v hevc -c:a aac -map 0:0 -map_metadata -1 -map 0:2 -g 120 -keyint_min 120 -tag:v hvc1 -preset ultrafast -crf 23 -maxrate 8000k -bufsize 16000k -x265-params no-scenecut=1:no-open-gop=1 -copyts -r 2284320000/38072000 -avoid_negative_ts disabled -f hls -hls_time 2 -hls_list_size 0 -hls_segment_type fmp4 -hls_fmp4_init_filename init.mp4 -hls_segment_options movflags=+frag_discont -hls_flags temp_file -hls_segment_filename /data/encoded-video/33e67826-48fc-4851-bb13-db18af2d221b/0f/93/0f93c55c-5686-4a5a-b605-44e5c1a71476/10/seg_%d.m4s -start_number 1 -vf scale=-2:1440 /data/encoded-video/33e67826-48fc-4851-bb13-db18af2d221b/0f/93/0f93c55c-5686-4a5a-b605-44e5c1a71476/10/playlist.m3u8
[Nest] 24710  - 05/11/2026, 2:26:39 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24802  - 05/11/2026, 2:26:39 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24710  - 05/11/2026, 2:26:39 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:39 PM   DEBUG [Api:LoggingInterceptor~ovu9iatv] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/10/init.mp4 200 3170.33ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:40 PM   DEBUG [Api:WebsocketRepository~b6akf5mf] Server event: HlsHeartbeat (send)
[Nest] 24710  - 05/11/2026, 2:26:40 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:40 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:40 PM   DEBUG [Api:LoggingInterceptor~b6akf5mf] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/10/seg_1.m4s 200 13.47ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:40 PM   DEBUG [Api:WebsocketRepository~rzro5wnh] Server event: HlsHeartbeat (send)
[Nest] 24802  - 05/11/2026, 2:26:40 PM   DEBUG [Api:WebsocketRepository~rzro5wnh] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:40 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:40 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24710  - 05/11/2026, 2:26:40 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24802  - 05/11/2026, 2:26:42 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24710  - 05/11/2026, 2:26:42 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24802  - 05/11/2026, 2:26:42 PM   DEBUG [Api:LoggingInterceptor~rzro5wnh] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/10/seg_2.m4s 200 1966.01ms ::ffff:127.0.0.1
[Nest] 24710  - 05/11/2026, 2:26:42 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:43 PM   DEBUG [Api:WebsocketRepository~7fg03dsb] Server event: HlsHeartbeat (send)
[Nest] 24710  - 05/11/2026, 2:26:43 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24802  - 05/11/2026, 2:26:43 PM   DEBUG [Api:WebsocketRepository~7fg03dsb] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:43 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24710  - 05/11/2026, 2:26:43 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:43 PM   DEBUG [Microservices:TranscodingService] Stopped transcoding for session 0f93c55c-5686-4a5a-b605-44e5c1a71476
[Nest] 24710  - 05/11/2026, 2:26:43 PM     LOG [Microservices:TranscodingService] Starting HLS transcode for asset 315640ae-e935-460a-abb8-1f73991ccc5c variant 1 with command: ffmpeg -ss 5.991666666666666 -nostdin -nostats -i /data/library/admin/2013/2013-12/bbb_sunflower_2160p_60fps_normal.mp4 -c:v hevc -c:a aac -map 0:0 -map_metadata -1 -map 0:2 -g 120 -keyint_min 120 -tag:v hvc1 -preset ultrafast -crf 23 -maxrate 1200k -bufsize 2400k -x265-params no-scenecut=1:no-open-gop=1 -copyts -r 2284320000/38072000 -avoid_negative_ts disabled -f hls -hls_time 2 -hls_list_size 0 -hls_segment_type fmp4 -hls_fmp4_init_filename init.mp4 -hls_segment_options movflags=+frag_discont -hls_flags temp_file -hls_segment_filename /data/encoded-video/33e67826-48fc-4851-bb13-db18af2d221b/0f/93/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_%d.m4s -start_number 3 -vf scale=-2:480 /data/encoded-video/33e67826-48fc-4851-bb13-db18af2d221b/0f/93/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/playlist.m3u8
[Nest] 24710  - 05/11/2026, 2:26:45 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24710  - 05/11/2026, 2:26:45 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:45 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24802  - 05/11/2026, 2:26:45 PM   DEBUG [Api:LoggingInterceptor~7fg03dsb] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_3.m4s 200 2093.35ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:45 PM   DEBUG [Api:WebsocketRepository~uptk7eu5] Server event: HlsHeartbeat (send)
[Nest] 24802  - 05/11/2026, 2:26:45 PM   DEBUG [Api:WebsocketRepository~uptk7eu5] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:45 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:45 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24710  - 05/11/2026, 2:26:45 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:46 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24710  - 05/11/2026, 2:26:46 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:46 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24802  - 05/11/2026, 2:26:46 PM   DEBUG [Api:LoggingInterceptor~uptk7eu5] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_4.m4s 200 646.73ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:46 PM   DEBUG [Api:WebsocketRepository~v6cewxd7] Server event: HlsHeartbeat (send)
[Nest] 24802  - 05/11/2026, 2:26:46 PM   DEBUG [Api:WebsocketRepository~v6cewxd7] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:46 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:46 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24710  - 05/11/2026, 2:26:46 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:47 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24710  - 05/11/2026, 2:26:47 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:47 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24802  - 05/11/2026, 2:26:47 PM   DEBUG [Api:LoggingInterceptor~v6cewxd7] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_5.m4s 200 678.86ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:47 PM   DEBUG [Api:WebsocketRepository~rztqk34a] Server event: HlsHeartbeat (send)
[Nest] 24802  - 05/11/2026, 2:26:47 PM   DEBUG [Api:WebsocketRepository~rztqk34a] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:47 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:47 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24710  - 05/11/2026, 2:26:47 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:48 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24802  - 05/11/2026, 2:26:48 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24710  - 05/11/2026, 2:26:48 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:48 PM   DEBUG [Api:LoggingInterceptor~rztqk34a] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_6.m4s 200 849.54ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:48 PM   DEBUG [Api:WebsocketRepository~yctgu6uw] Server event: HlsHeartbeat (send)
[Nest] 24710  - 05/11/2026, 2:26:48 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:48 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24802  - 05/11/2026, 2:26:48 PM   DEBUG [Api:WebsocketRepository~yctgu6uw] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:48 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:49 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24802  - 05/11/2026, 2:26:49 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24710  - 05/11/2026, 2:26:49 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:49 PM   DEBUG [Api:LoggingInterceptor~yctgu6uw] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_7.m4s 200 691.21ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:49 PM   DEBUG [Api:WebsocketRepository~t8gcogii] Server event: HlsHeartbeat (send)
[Nest] 24710  - 05/11/2026, 2:26:49 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24802  - 05/11/2026, 2:26:49 PM   DEBUG [Api:WebsocketRepository~t8gcogii] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:49 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24710  - 05/11/2026, 2:26:49 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)
[Nest] 24710  - 05/11/2026, 2:26:50 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentResult (send)
[Nest] 24802  - 05/11/2026, 2:26:50 PM   DEBUG [Api:WebsocketRepository] Server event: HlsSegmentResult (receive)
[Nest] 24710  - 05/11/2026, 2:26:50 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is 0 segments
[Nest] 24802  - 05/11/2026, 2:26:50 PM   DEBUG [Api:LoggingInterceptor~t8gcogii] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/0f93c55c-5686-4a5a-b605-44e5c1a71476/1/seg_8.m4s 200 742.00ms ::ffff:127.0.0.1
[Nest] 24802  - 05/11/2026, 2:26:50 PM   DEBUG [Api:WebsocketRepository~nf7k6yac] Server event: HlsHeartbeat (send)
[Nest] 24802  - 05/11/2026, 2:26:50 PM   DEBUG [Api:WebsocketRepository~nf7k6yac] Server event: HlsSegmentRequest (send)
[Nest] 24710  - 05/11/2026, 2:26:50 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsHeartbeat (receive)
[Nest] 24710  - 05/11/2026, 2:26:50 PM   DEBUG [Microservices:TranscodingService] Session 0f93c55c-5686-4a5a-b605-44e5c1a71476 lead is -1 segments
[Nest] 24710  - 05/11/2026, 2:26:50 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSegmentRequest (receive)

It seems like pending segments are not (always) rejected correctly. However, I only saw this twice (after clicking back on the video asset navbar), and cannot reproduce this now.

Log 
[Nest] 1800  - 05/11/2026, 1:23:37 PM   DEBUG [Api:WebsocketRepository~4kd0ns4j] Server event: HlsSessionEnd (send)
[Nest] 1800  - 05/11/2026, 1:23:37 PM   DEBUG [Api:LoggingInterceptor~4kd0ns4j] DELETE /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/5d956008-122d-43e0-821c-142f7330cc24 204 6.91ms ::ffff:127.0.0.1
[Nest] 1730  - 05/11/2026, 1:23:37 PM   DEBUG [Microservices:WebsocketRepository] Server event: HlsSessionEnd (receive)
[Nest] 1730  - 05/11/2026, 1:23:37 PM   DEBUG [Microservices:TranscodingService] Stopped transcoding for session 5d956008-122d-43e0-821c-142f7330cc24
[Nest] 1800  - 05/11/2026, 1:23:52 PM   ERROR [Api:LoggingRepository~nqz34zrk] Unable to send file: Error: Request timed out
Error: Request timed out
    at PendingEvents.complete (/usr/src/app/server/src/utils/event.ts:32:21)
    at Timeout. (/usr/src/app/server/src/utils/event.ts:19:43)
    at listOnTimeout (node:internal/timers:605:17)
    at process.processTimers (node:internal/timers:541:7)
[Nest] 1800  - 05/11/2026, 1:23:52 PM   ERROR [Api:GlobalExceptionFilter~nqz34zrk] Unknown error: Error: Request timed out
Error: Request timed out
    at PendingEvents.complete (/usr/src/app/server/src/utils/event.ts:32:21)
    at Timeout. (/usr/src/app/server/src/utils/event.ts:19:43)
    at listOnTimeout (node:internal/timers:605:17)
    at process.processTimers (node:internal/timers:541:7)
[Nest] 1800  - 05/11/2026, 1:23:52 PM   DEBUG [Api:LoggingInterceptor~nqz34zrk] GET /api/assets/315640ae-e935-460a-abb8-1f73991ccc5c/video/stream/5d956008-122d-43e0-821c-142f7330cc24/7/seg_40.m4s 500 15009.85ms undefined

Something else: interestingly, ffmpeg docs and other websites say that fMP4 requires at least hls version 7. I can't find this in the spec, nor another good source, so it's not clear to me what this is based on.

Comment thread server/src/services/transcoding.service.ts
Comment on lines +322 to +323
// SIGTERM makes it rename .tmp segments to .m4s even if they're still incomplete
session.process.kill('SIGKILL');
Copy link
Copy Markdown
Collaborator

@meesfrensel meesfrensel May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically a guess, but I wonder what the effect is on ffmpeg's resource cleanup. I assume all memory will be reclaimed by the kernel, but what about open file handles, tmp files, ...? And will sigkill make ffmpeg generate some kind of core dump that can fill the disk if not cleaned?

Copy link
Copy Markdown
Member Author

@mertalev mertalev May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open file handles get closed by the kernel. I'm not sure if it produces a core dump, though.

@mertalev
Copy link
Copy Markdown
Member Author

mertalev commented May 11, 2026

@meesfrensel I think the first logs you showed look more or less normal. seg_0 uses a low resolution variant. The client sees that it downloads quickly (it seems not to include the init.mp4 for bandwidth estimation), so it requests 1440p HEVC for seg_1. The server can't keep up with 1440p, so the client runs out of buffer and jumps down to 480p in an attempt to keep things moving.

The overall expectation is that the server can transcode any variant it offers in real-time, as it won't be a good experience if it can't keep up with the client. This is doubly true because going below real-time makes ABR more erratic as you've seen. On my laptop, it's stable at 1440p AV1 without any jumping because it takes 1.4s to produce a 2s segment. I omitted 2160p from the default list since it was too much for my laptop, but it's probably best to remove 1440p as well to be safer. This is where admin configuration comes in: a server might only have access to HEVC encode and not AV1 encode, some might handle 2160p while others can only do 1080p, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@meesfrensel meesfrensel meesfrensel left review comments

@danieldietzler danieldietzler danieldietzler left review comments

@shenlong-tanwen shenlong-tanwen Awaiting requested review from shenlong-tanwen shenlong-tanwen is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

changelog:feature 🗄️server 🖥️web

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mertalev @meesfrensel @danieldietzler