Answers.Body: <p>I wrote the following script to solve my problem. It uses <code>framemd5</code> checksums to find the inpoints in time and then uses the <code>concat</code> demuxer (thanks @Gyan for the tip!).</p>
  <pre class="lang-py prettyprint-override"><code># Usage: ffmpeg $(python3 script.py input1.mp4 input2.mp4) -y -an output.mp4
  
  import os
  import sys
  import subprocess
  
  ffmpeg_concat_echo, md5_last_frame = '', None
  
  for video_path in map(os.path.abspath, sys.argv[1:]):
          framemd5 = subprocess.check_output(['ffmpeg', '-nostats', '-hide_banner', '-i', video_path, '-an', '-f', 'framemd5', '-c', 'copy', '-'], stderr = open(os.devnull, 'w'))
          tb_num, tb_den = 1, 1
          inpoint = 0.0
          for line in framemd5.split(b'\n'):
                  if line.startswith(b'#tb'):
                          tb_num, tb_den = list(map(int, line.split()[-1].split(b'/')))
                  elif line and not line.startswith(b'#'):
                          splitted = line.split(b',')
                          md5 = splitted[-1].strip()
                          pts_time = float(splitted[2]) * tb_num / tb_den
                          if md5_last_frame is not None and md5 == md5_last_frame:
                                  inpoint = pts_time
          md5_last_frame = md5
          ffmpeg_concat_echo += 'echo &quot;file {}&quot;; echo &quot;inpoint {}&quot;;'.format(video_path, inpoint);
  
  print('-nostats -hide_banner -avoid_negative_ts make_zero -fflags +genpts -f concat -safe 0 -i &lt;({}) -c copy'.format(ffmpeg_concat_echo))
  </code></pre>
  <p>Below is another version (<a href="https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18" rel="nofollow noreferrer">source</a>, <a href="https://web.archive.org/web/20230311234256/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18" rel="nofollow noreferrer">archived</a>) which supports exact PTS inpoints using <code>concat</code> video filter; computing inpoints using explicit frame differences.</p>
  <pre class="lang-py prettyprint-override"><code># Usage: ffmpeg $(( echo input1.mp4; echo input2.mp4 ) | python3 concat_overlapping_video_framemd5_demuxer.py concat_file.txt) -y -an output.mp4
  
  import os
  import sys
  import subprocess
  
  ffmpeg_concat_file, md5_last_frame = open(sys.argv[1], 'w'), None
  for video_path in map(os.path.abspath, list(map(str.strip, sys.stdin))):
          framemd5 = subprocess.check_output(['ffmpeg', '-nostats', '-hide_banner', '-i', video_path, '-an', '-f', 'framemd5', '-c', 'copy', '-'], stderr = open(os.devnull, 'w'))
          inpoint, tb_num, tb_den = 0, 1, 1
          for line in framemd5.split(b'\n'):
                  if line.startswith(b'#tb'):
                          tb_num, tb_den = list(map(int, line.split()[-1].split(b'/')))
                  elif not line.startswith(b'#') and len(line) &gt; 0:
                          splitted = line.split(b',')
                          md5, pts_time = splitted[-1].strip(), float(splitted[2]) * tb_num / tb_den
                          if md5 == md5_last_frame:
                                  inpoint = pts_time
          md5_last_frame = md5
          ffmpeg_concat_file.write('file {}\ninpoint {}\n'.format(video_path, inpoint))
  
  print('-nostats -hide_banner -avoid_negative_ts make_zero -fflags +genpts -f concat -safe 0 -i {} -c copy'.format(sys.argv[1]))
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-22T09:19:42.240Z
Answers.Text: This may result in "Non-monotonous DTS" errors. Some ways to solve:
  1) Use `concat` video filter. Downsides: will re-encode
  2) [Extract streams and re-assemble them back](https://stackoverflow.com/questions/31691943/ffmpeg-concat-poduces-dts-out-of-order-errors). Downsides: will make a file copy, unless [named pipes are used](https://trac.ffmpeg.org/wiki/Concatenate#Usingnamedpipestoavoidintermediatefiles)
  3) Wait until [this patch](http://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/226706.html) gets merged. Hopefully it would fix the problem
Answers.UserDisplayName: Vadim Kantorov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T05:10:29.550Z
Answers.Text: is there better version of it, gist isn't working
Answers.UserDisplayName: Muhammad Umer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-21T15:40:22.103Z
Answers.Text: This worked great for stitching some security camera footage together, thank you.
Answers.UserDisplayName: hannasm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-06T23:02:55.547Z
Answers.Text: The gist link is broken, could you please share the code again in a more permanent way?
Answers.UserDisplayName: Isti115
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T23:48:35.217Z
Answers.Text: [Revised gist](https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18) ([archived](https://web.archive.org/web/20230311234256/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18)), [original gist](https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/321290839cde9a33efa22ad458e5e653de1e5a97) ([archived](https://web.archive.org/web/20230311234353/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/321290839cde9a33efa22ad458e5e653de1e5a97)), [revisions](https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/revisions).
Answers.UserDisplayName: mrienstra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T23:49:59.143Z
Answers.Text: Continued from [above](https://video.stackexchange.com/questions/24681/concat-overlapping-videos-with-ffmpeg-using-python3#comment46618_24689): [archived revisions](https://web.archive.org/web/20230311234407/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/revisions).
Answers.UserDisplayName: mrienstra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-21T14:03:53.893Z
Answers.LastActivityDate: 2023-03-18T19:30:56.337Z
Answers.LastEditDate: 2023-03-18T19:30:56.337Z
Answers.UserDisplayName: Vadim Kantorov
Body: <p>I have a number of videos that overlap in time that were downloaded from an online video stream:
  <code>
  0m00s  - 15m10s
  14m50s - 30m20s
  ...
  </code>
  The first few seconds (usually around 20 seconds) of a video segment duplicate last few seconds of the previous segment.</p>
  
  <p>I'd like to concat the video segments and form a single video file without duplicate parts. I can do it programmatically using <code>ffmpeg</code> libraries to find the cut points by minimizing error between a frame of the current segment and the last frame of the previous segment.</p>
  
  <p>Is there a way to achieve this with an ffmpeg command?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T10:37:50.073Z
Comments.Text: If the files have absolute timestamps, it may be possible. But if no, it's not possible in a single command. You can run one command to find join point betweeen two consecutive segments using blend filter/blackframe and then write a concat demuxer text file with the required inpoint/outpoint
Comments.UserDisplayName: Gyan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T10:43:26.143Z
Comments.Text: I just checked with `-vf showinfo`, and it seems they do not have absolute timestamps, but they do have some checksums (are these useful for finding exact duplicates?)
  
  Thanks for the pointer on blend/blackframe - googling it.
Comments.UserDisplayName: Vadim Kantorov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T10:46:35.337Z
Comments.Text: You need to add `-copyts` else ffmpeg will remove any starting offset before it gets to the filter.
Comments.UserDisplayName: Gyan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T12:56:31.457Z
Comments.Text: It seems `framemd5` matching is enough for my videos. I can make `framemd5` files for every video segment and fetch out dts / pts of the starting frame in every segment. Is there a way to concat with dts / pts inpoints instead of timestamps?
Comments.UserDisplayName: Vadim Kantorov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T14:23:49.690Z
Comments.Text: No, time has to be in seconds.
Comments.UserDisplayName: Gyan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T22:10:24.617Z
Comments.Text: I found that `trim` filter has `start_pts` option, but I can't make it work with `pts` values extracted from `framemd5` output (`start` with seconds works, but `start_pts` does not)
Comments.UserDisplayName: Vadim Kantorov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-17T23:33:49.213Z
Comments.Text: If someone lands here needing a "fuzzier" match than `framemd5`, see https://superuser.com/questions/663731/find-video-file-position-using-an-image-match
Comments.UserDisplayName: mrienstra
Tags: <ffmpeg>
Title: Concat overlapping videos with ffmpeg using python3
UserDisplayName: Vadim Kantorov
ThreadText: # Concat overlapping videos with ffmpeg using python3
  
  (Asked by: username_0 on 2018-08-20)
  
  I have a number of videos that overlap in time that were downloaded from an online video stream:
  `0m00s - 15m10s
  14m50s - 30m20s
  ...`
  The first few seconds (usually around 20 seconds) of a video segment duplicate last few seconds of the previous segment.
  
  I'd like to concat the video segments and form a single video file without duplicate parts. I can do it programmatically using `ffmpeg` libraries to find the cut points by minimizing error between a frame of the current segment and the last frame of the previous segment.
  
  Is there a way to achieve this with an ffmpeg command?
  
  **Tags:** ffmpeg
  
  > **Comments:**
  > **username_1**: If the files have absolute timestamps, it may be possible. But if no, it's not possible in a single command. You can run one command to find join point betweeen two consecutive segments using blend filter/blackframe and then write a concat demuxer text file with the required inpoint/outpoint  
  > **username_0**: I just checked with `-vf showinfo`, and it seems they do not have absolute timestamps, but they do have some checksums (are these useful for finding exact duplicates?)  Thanks for the pointer on blend/blackframe - googling it.  
  > **username_1**: You need to add `-copyts` else ffmpeg will remove any starting offset before it gets to the filter.  
  > **username_0**: It seems `framemd5` matching is enough for my videos. I can make `framemd5` files for every video segment and fetch out dts / pts of the starting frame in every segment. Is there a way to concat with dts / pts inpoints instead of timestamps?  
  > **username_1**: No, time has to be in seconds.  
  > **username_0**: I found that `trim` filter has `start_pts` option, but I can't make it work with `pts` values extracted from `framemd5` output (`start` with seconds works, but `start_pts` does not)  
  > **username_2**: If someone lands here needing a "fuzzier" match than `framemd5`, see https://superuser.com/questions/663731/find-video-file-position-using-an-image-match  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-08-21. Score: 1 | Accepted answer
  
  I wrote the following script to solve my problem. It uses `framemd5` checksums to find the inpoints in time and then uses the `concat` demuxer (thanks @username_1 for the tip!).
  
  ```py
  # Usage: ffmpeg $(python3 script.py input1.mp4 input2.mp4) -y -an output.mp4
  
  import os
  import sys
  import subprocess
  
  ffmpeg_concat_echo, md5_last_frame = '', None
  
  for video_path in map(os.path.abspath, sys.argv[1:]):
          framemd5 = subprocess.check_output(['ffmpeg', '-nostats', '-hide_banner', '-i', video_path, '-an', '-f', 'framemd5', '-c', 'copy', '-'], stderr = open(os.devnull, 'w'))
          tb_num, tb_den = 1, 1
          inpoint = 0.0
          for line in framemd5.split(b'\n'):
                  if line.startswith(b'#tb'):
                          tb_num, tb_den = list(map(int, line.split()[-1].split(b'/')))
                  elif line and not line.startswith(b'#'):
                          splitted = line.split(b',')
                          md5 = splitted[-1].strip()
                          pts_time = float(splitted[2]) * tb_num / tb_den
                          if md5_last_frame is not None and md5 == md5_last_frame:
                                  inpoint = pts_time
          md5_last_frame = md5
          ffmpeg_concat_echo += 'echo "file {}"; echo "inpoint {}";'.format(video_path, inpoint);
  
  print('-nostats -hide_banner -avoid_negative_ts make_zero -fflags +genpts -f concat -safe 0 -i <({}) -c copy'.format(ffmpeg_concat_echo))
  
  ```
  
  Below is another version ([source](https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18), [archived](https://web.archive.org/web/20230311234256/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18)) which supports exact PTS inpoints using `concat` video filter; computing inpoints using explicit frame differences.
  
  ```py
  # Usage: ffmpeg $(( echo input1.mp4; echo input2.mp4 ) | python3 concat_overlapping_video_framemd5_demuxer.py concat_file.txt) -y -an output.mp4
  
  import os
  import sys
  import subprocess
  
  ffmpeg_concat_file, md5_last_frame = open(sys.argv[1], 'w'), None
  for video_path in map(os.path.abspath, list(map(str.strip, sys.stdin))):
          framemd5 = subprocess.check_output(['ffmpeg', '-nostats', '-hide_banner', '-i', video_path, '-an', '-f', 'framemd5', '-c', 'copy', '-'], stderr = open(os.devnull, 'w'))
          inpoint, tb_num, tb_den = 0, 1, 1
          for line in framemd5.split(b'\n'):
                  if line.startswith(b'#tb'):
                          tb_num, tb_den = list(map(int, line.split()[-1].split(b'/')))
                  elif not line.startswith(b'#') and len(line) > 0:
                          splitted = line.split(b',')
                          md5, pts_time = splitted[-1].strip(), float(splitted[2]) * tb_num / tb_den
                          if md5 == md5_last_frame:
                                  inpoint = pts_time
          md5_last_frame = md5
          ffmpeg_concat_file.write('file {}\ninpoint {}\n'.format(video_path, inpoint))
  
  print('-nostats -hide_banner -avoid_negative_ts make_zero -fflags +genpts -f concat -safe 0 -i {} -c copy'.format(sys.argv[1]))
  
  ```
  
  > **Comments:**
  > **username_0**: This may result in "Non-monotonous DTS" errors. Some ways to solve: 1) Use `concat` video filter. Downsides: will re-encode 2) [Extract streams and re-assemble them back](https://stackoverflow.com/questions/31691943/ffmpeg-concat-poduces-dts-out-of-order-errors). Downsides: will make a file copy, unless [named pipes are used](https://trac.ffmpeg.org/wiki/Concatenate#Usingnamedpipestoavoidintermediatefiles) 3) Wait until [this patch](http://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/226706.html) gets merged. Hopefully it would fix the problem  
  > **username_3**: is there better version of it, gist isn't working  
  > **username_4**: This worked great for stitching some security camera footage together, thank you.  
  > **username_5**: The gist link is broken, could you please share the code again in a more permanent way?  
  > **username_2**: [Revised gist](https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18) ([archived](https://web.archive.org/web/20230311234256/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18)), [original gist](https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/321290839cde9a33efa22ad458e5e653de1e5a97) ([archived](https://web.archive.org/web/20230311234353/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/321290839cde9a33efa22ad458e5e653de1e5a97)), [revisions](https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/revisions).  
  > **username_2**: Continued from [above](https://video.stackexchange.com/questions/24681/concat-overlapping-videos-with-ffmpeg-using-python3#comment46618_24689): [archived revisions](https://web.archive.org/web/20230311234407/https://gist.github.com/yar/5056e5bd303a23de98139365d2ff5d18/revisions).  
  
  ---
  
  