Answers.Body: <p>FFmpeg does not support writing multiple sample descriptions for a MP4 track, which is what would be required for codec parameter changes to be signaled.</p>
  <p>FFmpeg also does not support an encoding session with resolution change except for raw uncompressed video and VP9, IIRC.</p>
  <p>However, what you can do, is segment the input video(s) at keyframes using the <a href="http://www.ffmpeg.org/ffmpeg-formats.html#segment_002c-stream_005fsegment_002c-ssegment" rel="nofollow noreferrer">segment muxer</a> into MPEG-TS segments. Encode these segments without inserting a scaler to MPEG-TS output segments.</p>
  <p>Similarly, generate one H264 in MPEG-TS segment for the images.</p>
  <p>Then concat the outputs together using streamcopy and avc3 tag (<code>-c copy -tag:v avc3</code>). The final MP4 will have only one track sample description but a compliant decoder should come across the parameter sets at segment boundaries and reset the decoder.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T15:24:51.510Z
Answers.Text: if I understand you correctly, one problem was having intermediates as MP4 as it "lost some capabilities support", and instead you're suggesting that all various transitions should be h.264 segments, one for each resolution change, and only then assembled into mp4 with `avc3` tag (what does this tag do?).  So my mistake in my "Use h.264 as intermediates" section is that I bunched a bunch of variable sized jpegs into one transcoding job?  Or is there something additional that MPEG-TS does as intermediate that h264 as intermediate fails at?
Answers.UserDisplayName: LB2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T16:51:04.103Z
Answers.Text: `.h264` can also work, but you'll lose per-frame timestamp info. `avc3` tells reader to expect mid-stream parameter sets.
Answers.UserDisplayName: Gyan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-28T19:28:31.057Z
Answers.Text: I finally got back to this project.  I found that concatenated .h264 don't squish and retain everything properly.  The issues is introduced when final .h264 is being converted to .mp4.  I tried the `-tag:v avc3` (after `-c copy`), but it doesn't seem to have any influence on output, and continues to have images squished.  Any thoughts on why `avc3` directive might not be working?
Answers.UserDisplayName: LB2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T14:44:37.793Z
Answers.LastActivityDate: 2023-07-03T14:44:37.793Z
Answers.UserDisplayName: Gyan
Body: <p><em>This will be a long post, so I thank you in advance for your patience in digesting it.</em></p>
  <h1>Context</h1>
  <p>I have different sources that generate visual content that eventually need to be all composed into a single .mp4 file.  The sources are:</p>
  <ul>
  <li>H.264 video (encoded using CUDA NVENC).
  <ul>
  <li>This video can have in-sequence resolution change that is natively supported by H.264 codec.</li>
  <li>I.e. stream may start as HxW resolution and mid-stream change to WxH. This behavior happens because it comes from a camera device that can be rotated and flipped between portrait and landscape (e.g. think of a phone camera recording video and phone being flipped from one orientation to another, and video recording adjusting its encoding for proper video scaling and orientation).</li>
  <li>When rotation occurs, most of the time H &amp; W are just swaps, but may actually be entirely new values -- e.g. in some cases 1024x768 will switch to 768x1024, but in other cases 1024x768 may become 460x640 (depends on source camera capabilities that I have no control over).</li>
  </ul>
  </li>
  <li>JPEGs.  A series (a.k.a. batch) of still JPEGs.
  <ul>
  <li>The native resolution of JPEGs may or may not match the video resolution in the earlier bullet.</li>
  <li>JPEGs can also reflect rotation of device and so some JPEGs in a sequence may start at HxW resolution and then from some arbitrary JPEG file can flip and become WxH. Similar to video, resolution dimensions are likely to be just a swap, but may become altogether different values.</li>
  </ul>
  </li>
  <li>There can be any number of batches and intermixes between video and still sources.  E.g. V<sub>1</sub> + S<sub>2</sub> + S<sub>3</sub> + V<sub>4</sub> + V<sub>5</sub> + V<sub>6</sub> + S<sub>7</sub> + ...</li>
  <li>There can be any number of resolution changes between or within batches. e.g. V<sub>1;r1</sub> + V<sub>1;r2</sub> + S<sub>2;r1</sub> + S<sub>2;r3</sub> + V<sub>3;r2</sub> + ... (where first subscript is batch sequence; rX is resolution)</li>
  </ul>
  <h1>Problem</h1>
  <p>I'm attempting to do this conversion with <code>ffmpeg</code> and can't quite get it right.  The problem is that I can't get output to respect source resolutions, and it just squishes all into a single output resolution.</p>
  <p><a href="https://i.sstatic.net/QWwMQ.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QWwMQ.jpg" alt="Example of squishing problem" /></a></p>
  <p>As already mentioned above, H.264 supports resolution changes in-sequence (mid-stream), and it should be possible to convert and concatenate all the content and have final output contain in-sequence resolution changes.</p>
  <p>Since MP4 is just a container, I'm assuming that MP4 files can do so as well?</p>
  <h2>Attempts so far</h2>
  <p>The approach thus far has been to take each batch of content (i.e. .h264 video or a set of JPEGs), and individually convert to .mp4. Video is converted using <code>-c copy</code> to ensure it doesn't try to transcode, e.g.:</p>
  <pre><code>ffmpeg -hide_banner -i videoX.h264 -c copy -vsync vfr -video_track_timescale 90000 intermediateX.mp4
  </code></pre>
  <p>... and JPEGs are converted using <code>-f concat</code></p>
  <pre><code>ffmpeg -hide_banner -f concat -safe 0 -i jpegsX.txt -vf 'scale=trunc(iw/2)*2:trunc(ih/2)*2' -r 30 -vsync vfr -video_track_timescale 90000 intermediateX.mp4
  </code></pre>
  <p>... and then all the intermediates concatenated together</p>
  <pre><code>ffmpeg -hide_banner -f concat -safe 0 -i final.txt -pix_fmt yuv420p -c copy -vsync vfr -video_track_timescale 90000 -metadata title='yabadabadoo' -fflags +bitexact -flags:v +bitexact -flags:a +bitexact final.mp4
  </code></pre>
  <p>This concatenates, but if resolution changes at some mid point, then that part of content comes up squished/stretched in final output.</p>
  <h3>Use h.264 as intermediates</h3>
  <p>All the intermediates are produced the same, except as .h264.  All intermediate .h264 are <code>cat</code>'ed together like `cat intermediate1.h264 intermediate2.264 &gt; final.h264.</p>
  <p>If final output is <code>final.mp4</code>, the output is incorrect and images are squished/stretched.</p>
  <p>If <code>final.h264</code>, then at least it seems to be respecting aspect ratios of input and managing to produce correctly looking output.  However, examining with <code>ffprobe</code> it seems that it uses SAR weird ratios, where first frames are <code>width=1440 height=3040 sample_aspect_ratio=1:1</code>, but later SAR takes on values like <code>width=176 height=340 sample_aspect_ratio=1545:176</code>, which I suspect isn't right, since all original input was with &quot;square pixels&quot;.  I think the reason for it is that it was composed out of different sized JPEGs, and concat filter somehow caused ffmpeg to manipulate SAR &quot;to get things fit&quot;.</p>
  <p>But at least it renders respectably, though hard to say with <code>ffplay</code> if player would actually see resolution change and resize accordingly .</p>
  <p>And, that's .h264; and I need final output to be .mp4.</p>
  <h3>Use <code>-vf</code> filter</h3>
  <p>I tried enforcing SAR using <code>-vf 'scale=trunc(iw/2)*2:trunc(ih/2)*2,setsar=1:1'</code> (scaling is to deal with odd dimension JPEGs), but it still produces frames with SAR like stated earlier.</p>
  <h3>Other thoughts</h3>
  <p>For now, while I haven't given up, I'm trying to avoid in my code examining each individual JEPG in a batch to see if there are differing sizes, and splitting batch so that each sub-batch is homogenous resolution-wise, and generating individual intermediate .h264 so that SAR remains sane, and keep fingers crossed that the final would work correctly. It'll be very slow, unfortunately.</p>
  <h1>Question</h1>
  <p>What's the right way to deal with all that using <code>ffmpeg</code>, and how to concatenate multiple varying resolution sources into a final mp4 so that it respects resolution changes mid-stream?</p>
  
Tags: <ffmpeg><mp4><h.264>
Title: ffmpeg h264 to mp4 conversion from multiple files fails to preserve in-sequence resolution changes
UserDisplayName: LB2
ThreadText: # ffmpeg h264 to mp4 conversion from multiple files fails to preserve in-sequence resolution changes
  
  (Asked by: username_0 on 2023-07-03)
  
  *This will be a long post, so I thank you in advance for your patience in digesting it.*
  
  # Context
  
  I have different sources that generate visual content that eventually need to be all composed into a single .mp4 file. The sources are:
  
  * H.264 video (encoded using CUDA NVENC).
    + This video can have in-sequence resolution change that is natively supported by H.264 codec.
    + I.e. stream may start as HxW resolution and mid-stream change to WxH. This behavior happens because it comes from a camera device that can be rotated and flipped between portrait and landscape (e.g. think of a phone camera recording video and phone being flipped from one orientation to another, and video recording adjusting its encoding for proper video scaling and orientation).
    + When rotation occurs, most of the time H & W are just swaps, but may actually be entirely new values -- e.g. in some cases 1024x768 will switch to 768x1024, but in other cases 1024x768 may become 460x640 (depends on source camera capabilities that I have no control over).
  * JPEGs. A series (a.k.a. batch) of still JPEGs.
    + The native resolution of JPEGs may or may not match the video resolution in the earlier bullet.
    + JPEGs can also reflect rotation of device and so some JPEGs in a sequence may start at HxW resolution and then from some arbitrary JPEG file can flip and become WxH. Similar to video, resolution dimensions are likely to be just a swap, but may become altogether different values.
  * There can be any number of batches and intermixes between video and still sources. E.g. V1 + S2 + S3 + V4 + V5 + V6 + S7 + ...
  * There can be any number of resolution changes between or within batches. e.g. V1;r1 + V1;r2 + S2;r1 + S2;r3 + V3;r2 + ... (where first subscript is batch sequence; rX is resolution)
  
  # Problem
  
  I'm attempting to do this conversion with `ffmpeg` and can't quite get it right. The problem is that I can't get output to respect source resolutions, and it just squishes all into a single output resolution.
  
  [![Example of squishing problem](https://i.sstatic.net/QWwMQ.jpg)](https://i.sstatic.net/QWwMQ.jpg)
  
  As already mentioned above, H.264 supports resolution changes in-sequence (mid-stream), and it should be possible to convert and concatenate all the content and have final output contain in-sequence resolution changes.
  
  Since MP4 is just a container, I'm assuming that MP4 files can do so as well?
  
  ## Attempts so far
  
  The approach thus far has been to take each batch of content (i.e. .h264 video or a set of JPEGs), and individually convert to .mp4. Video is converted using `-c copy` to ensure it doesn't try to transcode, e.g.:
  
  ```
  ffmpeg -hide_banner -i videoX.h264 -c copy -vsync vfr -video_track_timescale 90000 intermediateX.mp4
  
  ```
  
  ... and JPEGs are converted using `-f concat`
  
  ```
  ffmpeg -hide_banner -f concat -safe 0 -i jpegsX.txt -vf 'scale=trunc(iw/2)*2:trunc(ih/2)*2' -r 30 -vsync vfr -video_track_timescale 90000 intermediateX.mp4
  
  ```
  
  ... and then all the intermediates concatenated together
  
  ```
  ffmpeg -hide_banner -f concat -safe 0 -i final.txt -pix_fmt yuv420p -c copy -vsync vfr -video_track_timescale 90000 -metadata title='yabadabadoo' -fflags +bitexact -flags:v +bitexact -flags:a +bitexact final.mp4
  
  ```
  
  This concatenates, but if resolution changes at some mid point, then that part of content comes up squished/stretched in final output.
  
  ### Use h.264 as intermediates
  
  All the intermediates are produced the same, except as .h264. All intermediate .h264 are `cat`'ed together like `cat intermediate1.h264 intermediate2.264 > final.h264.
  
  If final output is `final.mp4`, the output is incorrect and images are squished/stretched.
  
  If `final.h264`, then at least it seems to be respecting aspect ratios of input and managing to produce correctly looking output. However, examining with `ffprobe` it seems that it uses SAR weird ratios, where first frames are `width=1440 height=3040 sample_aspect_ratio=1:1`, but later SAR takes on values like `width=176 height=340 sample_aspect_ratio=1545:176`, which I suspect isn't right, since all original input was with "square pixels". I think the reason for it is that it was composed out of different sized JPEGs, and concat filter somehow caused ffmpeg to manipulate SAR "to get things fit".
  
  But at least it renders respectably, though hard to say with `ffplay` if player would actually see resolution change and resize accordingly .
  
  And, that's .h264; and I need final output to be .mp4.
  
  ### Use `-vf` filter
  
  I tried enforcing SAR using `-vf 'scale=trunc(iw/2)*2:trunc(ih/2)*2,setsar=1:1'` (scaling is to deal with odd dimension JPEGs), but it still produces frames with SAR like stated earlier.
  
  ### Other thoughts
  
  For now, while I haven't given up, I'm trying to avoid in my code examining each individual JEPG in a batch to see if there are differing sizes, and splitting batch so that each sub-batch is homogenous resolution-wise, and generating individual intermediate .h264 so that SAR remains sane, and keep fingers crossed that the final would work correctly. It'll be very slow, unfortunately.
  
  # Question
  
  What's the right way to deal with all that using `ffmpeg`, and how to concatenate multiple varying resolution sources into a final mp4 so that it respects resolution changes mid-stream?
  
  **Tags:** ffmpeg, mp4, h.264
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-03. Score: 0
  
  FFmpeg does not support writing multiple sample descriptions for a MP4 track, which is what would be required for codec parameter changes to be signaled.
  
  FFmpeg also does not support an encoding session with resolution change except for raw uncompressed video and VP9, IIRC.
  
  However, what you can do, is segment the input video(s) at keyframes using the [segment muxer](http://www.ffmpeg.org/ffmpeg-formats.html#segment_002c-stream_005fsegment_002c-ssegment) into MPEG-TS segments. Encode these segments without inserting a scaler to MPEG-TS output segments.
  
  Similarly, generate one H264 in MPEG-TS segment for the images.
  
  Then concat the outputs together using streamcopy and avc3 tag (`-c copy -tag:v avc3`). The final MP4 will have only one track sample description but a compliant decoder should come across the parameter sets at segment boundaries and reset the decoder.
  
  > **Comments:**
  > **username_0**: if I understand you correctly, one problem was having intermediates as MP4 as it "lost some capabilities support", and instead you're suggesting that all various transitions should be h.264 segments, one for each resolution change, and only then assembled into mp4 with `avc3` tag (what does this tag do?).  So my mistake in my "Use h.264 as intermediates" section is that I bunched a bunch of variable sized jpegs into one transcoding job?  Or is there something additional that MPEG-TS does as intermediate that h264 as intermediate fails at?  
  > **username_1**: `.h264` can also work, but you'll lose per-frame timestamp info. `avc3` tells reader to expect mid-stream parameter sets.  
  > **username_0**: I finally got back to this project.  I found that concatenated .h264 don't squish and retain everything properly.  The issues is introduced when final .h264 is being converted to .mp4.  I tried the `-tag:v avc3` (after `-c copy`), but it doesn't seem to have any influence on output, and continues to have images squished.  Any thoughts on why `avc3` directive might not be working?  
  
  ---
  
  