Answers.Body: <p>You have to render the audio in one command.</p>
  <p>MDCT-based codecs like AAC or MP3 carry out their transforms over multiple adjoining frames. When doing a segmented encode of a single stream, the encoder 'supplies' silent frames at the top and tail of each segment.</p>
  <p>From <a href="https://developer.apple.com/library/content/documentation/QuickTime/QTFF/QTFFAppenG/QTFFAppenG.html" rel="nofollow noreferrer">Apple</a>:</p>
  <blockquote>
  <p>AAC requires data beyond the source PCM audio samples in order to
  correctly encode and decode audio samples due to the nature of the
  encoding algorithm. AAC encoding uses a transform over consecutive
  sets of 2048 audio samples, applied every 1024 audio samples
  (overlapped). For correct audio to be decoded, both transforms for any
  period of 1024 audio samples are needed. For this reason, encoders add
  at least 1024 samples of silence before the first ‘true’ audio sample,
  and often add more. This is called variously “priming”, “priming
  samples”, or “encoder delay”.</p>
  </blockquote>
  <p>During muxing, these priming samples can either be discarded or kept. If kept, there will be a gap at the seams. If discarded, the decoder will use the tail frame of the previous segment for overlap purposes and the result will be a glitch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T08:43:22.360Z
Answers.Text: So, in essence, there could be a dedicated process rendering the .ts chunks for audio completely separately — but how would you then combine them back into a single `.m3u8` master playlist?
Answers.UserDisplayName: slhck
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T10:52:04.407Z
Answers.Text: See option var_stream_map in the hls muxer docs. You will have two inputs in ffmpeg: `-f concat -i video_segs_list -i audio_stream -map 0 -map 1 ...`. You can also concat the video segs into one TS file and use that.
Answers.UserDisplayName: Gyan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T13:35:17.757Z
Answers.Text: That indeed produces a playable stream. Thanks! But due to the live requirement and wanting to be able to append video segments to the playlist as they are being generated, I can't do a single remux command, because that will rewrite all the segments and playlist files. I can generate the audio in a separate thread before, and only then start outputting the video segments, but that approach yields a stream that never starts playing in HLS.js. It seems to wait for an initial video PTS but doesn't get any: https://pastebin.com/raw/MAYJyHs5 — any idea?
Answers.UserDisplayName: slhck
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T18:41:19.740Z
Answers.Text: I'll check out the details of your process tomorrow.
Answers.UserDisplayName: Gyan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-18T08:05:25.620Z
Answers.Text: Much appreciated. In case this becomes more complex, if you are interested, we can discuss consulting offline.
Answers.UserDisplayName: slhck
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-18T08:46:41.777Z
Answers.Text: I *think* I was able to make it work with the individual audio and video generation by ensuring that audio is generated with ffmpeg's `discont_start`, and video is *not*. For some reason, video still begins with `#EXT-X-DISCONTINUITY`, so I had to put an extra one into the audio stream to have the continuity counters align.
Answers.UserDisplayName: slhck
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-26T03:54:16.273Z
Answers.Text: @slhck I am curious to know if your results would be helpful to me in solving https://sound.stackexchange.com/questions/51209/possible-to-ffmpeg-encode-an-mp3-media-segment-for-hls-to-follow-existing-mp3 also let's get a beer sometime :)
Answers.UserDisplayName: Nick Charney Kaye
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T04:29:54.823Z
Answers.LastActivityDate: 2021-11-17T04:29:54.823Z
Answers.UserDisplayName: Gyan
Answers.Body: <p>I was able to make it work with the individual audio and video generation by ensuring that audio is generated with ffmpeg's <code>discont_start</code> flag set.</p>
  <pre><code>-hls_flags independent_segments+append_list+discont_start
  </code></pre>
  <p>I noticed that when looking at the CC counters in the hls.js debug logs, these were not aligned. Video started with 1, audio witht 0.</p>
  <p>Looking at the playlists, for some reason, video began with <code>#EXT-X-DISCONTINUITY</code>, so I had to put an extra one into the audio stream to have the continuity counters align (i.e., start with 1 for both streams).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T17:04:42.800Z
Answers.LastActivityDate: 2021-12-27T17:04:42.800Z
Answers.UserDisplayName: slhck
Body: <p>I am creating MPEG-TS segments for HLS playback from multiple ffmpeg processes (it will be used for parallel encoding at a later stage). The commands are as follows — they can be run in sequence for testing purposes:</p>
  <pre class="lang-bash prettyprint-override"><code>#!/usr/bin/env bash
  rm seg*.ts master.m3u8
  
  ffmpeg \
    -f lavfi -i testsrc=s=320x240:r=30 -f lavfi -i sine=440 \
    -vf 'drawtext=text=%{n}:fontsize=72:r=60:x=(w-tw)/2: y=h-(2*lh): fontcolor=white: box=1: boxcolor=0x00000099' \
    -pix_fmt yuv420p \
    -preset ultrafast -c:v:0 libx264 -x264-params &quot;nal-hrd=cbr:force-cfr=1&quot; -b:v:0 4M -maxrate:v:0 4M -minrate:v:0 4M -bufsize:v:0 8M -g 60 -sc_threshold 0 -keyint_min 60 \
    -f hls -hls_time 2 -hls_playlist_type event \
    -hls_flags independent_segments+append_list+omit_endlist \
    -hls_segment_type mpegts -hls_list_size 0 \
    -hls_segment_filename seg_01_%02d.ts -master_pl_name master.m3u8 -start_number 0 \
    -muxdelay 0 \
    -muxpreload 0 \
    -output_ts_offset 0 \
    -t 10 \
    master.m3u8
  
  ffmpeg \
    -f lavfi -i testsrc=s=320x240:r=30 -f lavfi -i sine=440 \
    -vf 'drawtext=text=%{n}:fontsize=72:r=60:x=(w-tw)/2: y=h-(2*lh): fontcolor=white: box=1: boxcolor=0x00000099' \
    -pix_fmt yuv420p \
    -preset ultrafast -c:v:0 libx264 -x264-params &quot;nal-hrd=cbr:force-cfr=1&quot; -b:v:0 4M -maxrate:v:0 4M -minrate:v:0 4M -bufsize:v:0 8M -g 60 -sc_threshold 0 -keyint_min 60 \
    -f hls -hls_time 2 -hls_playlist_type event \
    -hls_flags independent_segments+append_list \
    -hls_segment_type mpegts -hls_list_size 0 \
    -hls_segment_filename seg_02_%02d.ts -master_pl_name master.m3u8 -start_number 0 \
    -muxdelay 0 \
    -muxpreload 0 \
    -output_ts_offset 10 \
    -ss 10 \
    -to 20 \
    master.m3u8
  </code></pre>
  <p>You can load this in your browser if you simply run a web server like <code>python3 -m http.server</code> with:</p>
  <pre class="lang-html prettyprint-override"><code>&lt;html&gt;
    &lt;head&gt;&lt;/head&gt;
    &lt;body&gt;
      &lt;script src=&quot;https://cdn.jsdelivr.net/npm/hls.js@latest&quot;&gt;&lt;/script&gt;
      &lt;video id=&quot;video&quot; controls autoplay&gt;&lt;/video&gt;
      &lt;script&gt;
        var video = document.getElementById(&quot;video&quot;);
        var videoSrc = &quot;master.m3u8&quot;;
        var hls = new Hls({
          debug: true,
        });
        hls.loadSource(videoSrc);
        hls.attachMedia(video);
      &lt;/script&gt;
    &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  <p>Now, the video seems to play fine. Looking at the frame counter, I see that no frames are lost (although it seems like the very last frame of segment set 1 is displayed a little bit shorter).</p>
  <p><strong>The audio, however, has a small dropout at the switch from segment set 1 to 2.</strong></p>
  <p>The generated M3U8 has an <code>#EXT-X-DISCONTINUITY</code> tag before switching to the second segment set.</p>
  <p>I noted that the start times of the segments don't quite correspond when comparing the audio and video streams. To test this, I run ffprobe and look at the <code>start_time</code> and <code>start_pts</code> values:</p>
  <pre class="lang-bash prettyprint-override"><code>for s in *.ts; do ffprobe -loglevel error -hide_banner -select_streams v -show_streams -of json &quot;$s&quot; | jq --arg input &quot;$s&quot; '.streams[0] | {$input, duration, start_time, start_pts}'; done | jq -s . 
  </code></pre>
  <p>This is the output:</p>
  <pre class="lang-json prettyprint-override"><code>[
    {
      &quot;input&quot;: &quot;seg_01_00.ts&quot;,
      &quot;duration&quot;: &quot;2.000000&quot;,
      &quot;start_time&quot;: &quot;0.023222&quot;,
      &quot;start_pts&quot;: 2090
    },
    // ...
    {
      &quot;input&quot;: &quot;seg_01_04.ts&quot;,
      &quot;duration&quot;: &quot;2.000000&quot;,
      &quot;start_time&quot;: &quot;8.023222&quot;,
      &quot;start_pts&quot;: 722090
    },
    // ...
    {
      &quot;input&quot;: &quot;seg_02_05.ts&quot;,
      &quot;duration&quot;: &quot;2.000000&quot;,
      &quot;start_time&quot;: &quot;10.000000&quot;,
      &quot;start_pts&quot;: 900000
    },
    // ...
  ]
  </code></pre>
  <p>The second set is forced to start at 10 seconds, while the last segment of the first set actually extends to 8.023222 + 2 = 10.023222 seconds. This is due to the non-negative offset of the first segment of the first set.</p>
  <p>For audio, the timestamps and durations are completely different:</p>
  <pre class="lang-json prettyprint-override"><code>[
    {
      &quot;input&quot;: &quot;seg_01_00.ts&quot;,
      &quot;duration&quot;: &quot;2.043344&quot;,
      &quot;start_time&quot;: &quot;0.000000&quot;,
      &quot;start_pts&quot;: 0
    },
    // ...
    {
      &quot;input&quot;: &quot;seg_01_04.ts&quot;,
      &quot;duration&quot;: &quot;1.996900&quot;,
      &quot;start_time&quot;: &quot;8.034111&quot;,
      &quot;start_pts&quot;: 723070
    },
    {
      &quot;input&quot;: &quot;seg_02_05.ts&quot;,
      &quot;duration&quot;: &quot;2.043344&quot;,
      &quot;start_time&quot;: &quot;9.976778&quot;,
      &quot;start_pts&quot;: 897910
    },
    // ...
  ]
  </code></pre>
  <p>Is there any way to fix this, so that the segments generated by the second ffmpeg command can be stitched during playback, without choppy audio?</p>
  <hr />
  <p>I have tried another approach, which consists of generating the audio stream separately before starting to encode the video. The audio and video playlists are written into dedicated <code>.m3u8</code> files which I am simply referencing from a <code>master.m3u8</code> playlist. This one I have to  manually generate, because otherwise ffmpeg would overwrite it. The whole approach is <a href="https://pastebin.com/raw/MAYJyHs5" rel="nofollow noreferrer">shown here</a>.</p>
  <p>The problem is that the video won't start in hls.js:</p>
  <blockquote>
  <p>[log] &gt; Unknown video PTS for cc 0, waiting for video PTS before demuxing audio frag 1 of [0 ,10],track 0</p>
  </blockquote>
  <p>This can be demoed <a href="https://storage.googleapis.com/aveq-storage/examples/video.stackexchange.com-34627-ffmpeg-hls-stitching/index.html" rel="nofollow noreferrer">here</a>.</p>
  <p>Once I remux everything into new sets of segments, as suggested by @Gyan, it works again. Demo is <a href="https://storage.googleapis.com/aveq-storage/examples/video.stackexchange.com-34627-ffmpeg-hls-stitching/index-remux.html" rel="nofollow noreferrer">here</a>.</p>
  <p>However, this is not a workable solution here, since I need to be able to generate the HLS segments on the fly, and append to the playlists later.</p>
  
Tags: <ffmpeg><hls>
Title: Audio discontinuities when generating HLS segments in different processes
UserDisplayName: slhck
ThreadText: # Audio discontinuities when generating HLS segments in different processes
  
  (Asked by: username_0 on 2021-11-16)
  
  I am creating MPEG-TS segments for HLS playback from multiple ffmpeg processes (it will be used for parallel encoding at a later stage). The commands are as follows — they can be run in sequence for testing purposes:
  
  ```bash
  #!/usr/bin/env bash
  rm seg*.ts master.m3u8
  
  ffmpeg \
    -f lavfi -i testsrc=s=320x240:r=30 -f lavfi -i sine=440 \
    -vf 'drawtext=text=%{n}:fontsize=72:r=60:x=(w-tw)/2: y=h-(2*lh): fontcolor=white: box=1: boxcolor=0x00000099' \
    -pix_fmt yuv420p \
    -preset ultrafast -c:v:0 libx264 -x264-params "nal-hrd=cbr:force-cfr=1" -b:v:0 4M -maxrate:v:0 4M -minrate:v:0 4M -bufsize:v:0 8M -g 60 -sc_threshold 0 -keyint_min 60 \
    -f hls -hls_time 2 -hls_playlist_type event \
    -hls_flags independent_segments+append_list+omit_endlist \
    -hls_segment_type mpegts -hls_list_size 0 \
    -hls_segment_filename seg_01_%02d.ts -master_pl_name master.m3u8 -start_number 0 \
    -muxdelay 0 \
    -muxpreload 0 \
    -output_ts_offset 0 \
    -t 10 \
    master.m3u8
  
  ffmpeg \
    -f lavfi -i testsrc=s=320x240:r=30 -f lavfi -i sine=440 \
    -vf 'drawtext=text=%{n}:fontsize=72:r=60:x=(w-tw)/2: y=h-(2*lh): fontcolor=white: box=1: boxcolor=0x00000099' \
    -pix_fmt yuv420p \
    -preset ultrafast -c:v:0 libx264 -x264-params "nal-hrd=cbr:force-cfr=1" -b:v:0 4M -maxrate:v:0 4M -minrate:v:0 4M -bufsize:v:0 8M -g 60 -sc_threshold 0 -keyint_min 60 \
    -f hls -hls_time 2 -hls_playlist_type event \
    -hls_flags independent_segments+append_list \
    -hls_segment_type mpegts -hls_list_size 0 \
    -hls_segment_filename seg_02_%02d.ts -master_pl_name master.m3u8 -start_number 0 \
    -muxdelay 0 \
    -muxpreload 0 \
    -output_ts_offset 10 \
    -ss 10 \
    -to 20 \
    master.m3u8
  
  ```
  
  You can load this in your browser if you simply run a web server like `python3 -m http.server` with:
  
  ```html
  <html>
    <head></head>
    <body>
      <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
      <video id="video" controls autoplay></video>
      <script>
        var video = document.getElementById("video");
        var videoSrc = "master.m3u8";
        var hls = new Hls({
          debug: true,
        });
        hls.loadSource(videoSrc);
        hls.attachMedia(video);
      </script>
    </body>
  </html>
  
  ```
  
  Now, the video seems to play fine. Looking at the frame counter, I see that no frames are lost (although it seems like the very last frame of segment set 1 is displayed a little bit shorter).
  
  **The audio, however, has a small dropout at the switch from segment set 1 to 2.**
  
  The generated M3U8 has an `#EXT-X-DISCONTINUITY` tag before switching to the second segment set.
  
  I noted that the start times of the segments don't quite correspond when comparing the audio and video streams. To test this, I run ffprobe and look at the `start_time` and `start_pts` values:
  
  ```bash
  for s in *.ts; do ffprobe -loglevel error -hide_banner -select_streams v -show_streams -of json "$s" | jq --arg input "$s" '.streams[0] | {$input, duration, start_time, start_pts}'; done | jq -s . 
  
  ```
  
  This is the output:
  
  ```json
  [
    {
      "input": "seg_01_00.ts",
      "duration": "2.000000",
      "start_time": "0.023222",
      "start_pts": 2090
    },
    // ...
    {
      "input": "seg_01_04.ts",
      "duration": "2.000000",
      "start_time": "8.023222",
      "start_pts": 722090
    },
    // ...
    {
      "input": "seg_02_05.ts",
      "duration": "2.000000",
      "start_time": "10.000000",
      "start_pts": 900000
    },
    // ...
  ]
  
  ```
  
  The second set is forced to start at 10 seconds, while the last segment of the first set actually extends to 8.023222 + 2 = 10.023222 seconds. This is due to the non-negative offset of the first segment of the first set.
  
  For audio, the timestamps and durations are completely different:
  
  ```json
  [
    {
      "input": "seg_01_00.ts",
      "duration": "2.043344",
      "start_time": "0.000000",
      "start_pts": 0
    },
    // ...
    {
      "input": "seg_01_04.ts",
      "duration": "1.996900",
      "start_time": "8.034111",
      "start_pts": 723070
    },
    {
      "input": "seg_02_05.ts",
      "duration": "2.043344",
      "start_time": "9.976778",
      "start_pts": 897910
    },
    // ...
  ]
  
  ```
  
  Is there any way to fix this, so that the segments generated by the second ffmpeg command can be stitched during playback, without choppy audio?
  
  ---
  
  I have tried another approach, which consists of generating the audio stream separately before starting to encode the video. The audio and video playlists are written into dedicated `.m3u8` files which I am simply referencing from a `master.m3u8` playlist. This one I have to manually generate, because otherwise ffmpeg would overwrite it. The whole approach is [shown here](https://pastebin.com/raw/MAYJyHs5).
  
  The problem is that the video won't start in hls.js:
  
  > [log] > Unknown video PTS for cc 0, waiting for video PTS before demuxing audio frag 1 of [0 ,10],track 0
  
  This can be demoed [here](https://storage.googleapis.com/aveq-storage/examples/video.stackexchange.com-34627-ffmpeg-hls-stitching/index.html).
  
  Once I remux everything into new sets of segments, as suggested by @username_1, it works again. Demo is [here](https://storage.googleapis.com/aveq-storage/examples/video.stackexchange.com-34627-ffmpeg-hls-stitching/index-remux.html).
  
  However, this is not a workable solution here, since I need to be able to generate the HLS segments on the fly, and append to the playlists later.
  
  **Tags:** ffmpeg, hls
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2021-12-27. Score: 3 | Accepted answer
  
  I was able to make it work with the individual audio and video generation by ensuring that audio is generated with ffmpeg's `discont_start` flag set.
  
  ```
  -hls_flags independent_segments+append_list+discont_start
  
  ```
  
  I noticed that when looking at the CC counters in the hls.js debug logs, these were not aligned. Video started with 1, audio witht 0.
  
  Looking at the playlists, for some reason, video began with `#EXT-X-DISCONTINUITY`, so I had to put an extra one into the audio stream to have the continuity counters align (i.e., start with 1 for both streams).
  
  
  ---
  
  ### Answer by username_1 on 2021-11-17. Score: 3
  
  You have to render the audio in one command.
  
  MDCT-based codecs like AAC or MP3 carry out their transforms over multiple adjoining frames. When doing a segmented encode of a single stream, the encoder 'supplies' silent frames at the top and tail of each segment.
  
  From [Apple](https://developer.apple.com/library/content/documentation/QuickTime/QTFF/QTFFAppenG/QTFFAppenG.html):
  
  > AAC requires data beyond the source PCM audio samples in order to
  > correctly encode and decode audio samples due to the nature of the
  > encoding algorithm. AAC encoding uses a transform over consecutive
  > sets of 2048 audio samples, applied every 1024 audio samples
  > (overlapped). For correct audio to be decoded, both transforms for any
  > period of 1024 audio samples are needed. For this reason, encoders add
  > at least 1024 samples of silence before the first ‘true’ audio sample,
  > and often add more. This is called variously “priming”, “priming
  > samples”, or “encoder delay”.
  
  During muxing, these priming samples can either be discarded or kept. If kept, there will be a gap at the seams. If discarded, the decoder will use the tail frame of the previous segment for overlap purposes and the result will be a glitch.
  
  > **Comments:**
  > **username_0**: So, in essence, there could be a dedicated process rendering the .ts chunks for audio completely separately — but how would you then combine them back into a single `.m3u8` master playlist?  
  > **username_1**: See option var_stream_map in the hls muxer docs. You will have two inputs in ffmpeg: `-f concat -i video_segs_list -i audio_stream -map 0 -map 1 ...`. You can also concat the video segs into one TS file and use that.  
  > **username_0**: That indeed produces a playable stream. Thanks! But due to the live requirement and wanting to be able to append video segments to the playlist as they are being generated, I can't do a single remux command, because that will rewrite all the segments and playlist files. I can generate the audio in a separate thread before, and only then start outputting the video segments, but that approach yields a stream that never starts playing in HLS.js. It seems to wait for an initial video PTS but doesn't get any: https://pastebin.com/raw/MAYJyHs5 — any idea?  
  > **username_1**: I'll check out the details of your process tomorrow.  
  > **username_0**: Much appreciated. In case this becomes more complex, if you are interested, we can discuss consulting offline.  
  > **username_0**: I *think* I was able to make it work with the individual audio and video generation by ensuring that audio is generated with ffmpeg's `discont_start`, and video is *not*. For some reason, video still begins with `#EXT-X-DISCONTINUITY`, so I had to put an extra one into the audio stream to have the continuity counters align.  
  > **username_2**: @username_0 I am curious to know if your results would be helpful to me in solving https://sound.stackexchange.com/questions/51209/possible-to-ffmpeg-encode-an-mp3-media-segment-for-hls-to-follow-existing-mp3 also let's get a beer sometime :)  
  
  ---
  
  