Answers.Body: <p>This seems related to : <a href="https://video.stackexchange.com/a/33521/32436">https://video.stackexchange.com/a/33521/32436</a></p>
  <blockquote>
  <p>You could check out
  <a href="https://theophanemayaud.github.io/video-simili-duplicate-cleaner/" rel="nofollow noreferrer">https://theophanemayaud.github.io/video-simili-duplicate-cleaner/</a>
  (macOS and Windows binaries available) It's open source, I created it with QT and it relies on ffmpeg.</p>
  <p>The idea is to extract a few (eg 2, the first and the last) frames,
  and compare them. If the duration is close, and the frames look
  similar : you've got a duplicate.</p>
  </blockquote>
  <p>You can &quot;manually compare the extracted frames&quot;, or use thresholds to configure some automatic behaviors, and trash the duplicates.</p>
  <blockquote>
  <p>It uses opencv for frame comparison, and ffmpeg for frame extraction
  (ffmpeg is also for video metadata which is very useful when trying to
  tell which video to keep !). Since it's open source, you can improve
  it to fit your needs !</p>
  </blockquote>
  <p>I've found it to be really helpful in a situation like yours, where I had about 15 000 videos from google photos which compresses them, Picasa, handbrake compressed, original copies, and other changed versions of the same videos !</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-29T09:58:54.750Z
Answers.LastActivityDate: 2022-03-16T13:25:00.893Z
Answers.LastEditDate: 2022-03-16T13:25:00.893Z
Answers.UserDisplayName: Théophane
Answers.Body: <p>You can try Video Comparer (shareware).</p>
  
  <p><a href="http://www.video-comparer.com" rel="nofollow">http://www.video-comparer.com</a></p>
  
  <p>It quickly detects similar videos with image transformations and supports split videos into multiple CDs.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-25T23:02:43.883Z
Answers.Text: This software did not detect *similar* videos (same scene/person taken a few minutes apart).
Answers.UserDisplayName: Clay Nichols
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2013-03-13T12:30:59.123Z
Answers.LastActivityDate: 2013-03-13T12:30:59.123Z
Answers.UserDisplayName: Rigop
Answers.Body: <p>I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is <a href="http://duplicatevideosearch.com" rel="nofollow">http://duplicatevideosearch.com</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2013-11-12T08:27:30.407Z
Answers.LastActivityDate: 2013-11-12T08:27:30.407Z
Answers.UserDisplayName: Max Smirnoff
Answers.Body: <p>Similar images - <a href="https://tn123.org/simimages/" rel="nofollow">https://tn123.org/simimages/</a> is a bit old, but still functional.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-03-19T00:23:12.770Z
Answers.LastActivityDate: 2015-03-19T00:23:12.770Z
Answers.UserDisplayName: Lion
Answers.Body: <p>I'd recommend a hybrid approach using both computers and people.</p>
  
  <ol>
  <li>Bucket the videos by their length (round to the nearest second)</li>
  <li>For each bucket, use ffmpeg to <a href="http://blog.prashanthellina.com/2008/03/29/creating-video-thumbnails-using-ffmpeg/">generate thumbnails</a> at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)</li>
  <li>Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.</li>
  </ol>
  
  <p>You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse.  Good luck!</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2012-11-13T15:30:50.820Z
Answers.Text: Thumbnails are a great idea! The new videos came in as FLV, which I can't seem to thumbnail no matter what I try. This could work quite well
Answers.UserDisplayName: Zelda
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2012-11-13T19:05:26.637Z
Answers.Text: ffmpeg should be able to convert the flv files into something else (avi, mp4, etc).
Answers.UserDisplayName: jimbo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2012-11-13T19:11:58.903Z
Answers.Text: Yeah, I'll eventually convert them to MP4 but I want to make sure I lose as little quality as possible when I do that, so I've put it off until I can take the time
Answers.UserDisplayName: Zelda
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2013-06-18T20:16:45.503Z
Answers.Text: If you wanted to, you could find a program that would analyze the thumbnails to find duplicates. However, the time spent finding such a program then configuring it would not be worth it for a small project.
Answers.UserDisplayName: Cole Tobin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-14T04:23:52.867Z
Answers.Text: @BenBrocka Another solution is you make an unique hash string (exampley 8 digits character) from your file in uploading time  and store it on your db, and add a condition on stored unique string for checking duplicate videos in upload time.
Answers.UserDisplayName: Parsa Saei
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2012-11-13T15:16:55.360Z
Answers.LastActivityDate: 2012-11-13T15:16:55.360Z
Answers.UserDisplayName: jimbo
Body: <p>After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting. </p>
  
  <p>To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.</p>
  
  <p>How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually. </p>
  
  <p>Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2012-11-12T10:40:50.110Z
Comments.Text: Very interesting question!
Comments.UserDisplayName: Dr Mayhem
Tags: <video>
Title: How can I find duplicate videos from a large set of videos of varying formats?
UserDisplayName: Zelda
ThreadText: # How can I find duplicate videos from a large set of videos of varying formats?
  
  (Asked by: username_0 on 2012-11-11)
  
  After losing a backup device with a bunch of my original videos I've had to download a large number of them (~300) back from Youtube. The new downloads are a mix of MP4 and FLV, and the originals are a mix of almost any video format due to lots of experimenting.
  
  To make matters worse there's no clear cutoff for when my local copies disappeared; I've found a point in my youtube videos chonologically where I have some of the videos and I don't have others. I've been trying to manually weed out videos where I already have a copy, but with this many videos I'm sure I've missed some.
  
  How can I effectively go through several hundred videos of different formats and weed out duplicates? The names will be similar but not identical. The formats may be different, and the file size may be different due to that (and or encoding differences). The file lengths should be the same, and visually the videos should be similar enough, but I'm not sure there's any tool to sort that out visually.
  
  Am I doomed to go through the list based on running time and manually look for duplicates or is there a better way?
  
  **Tags:** video
  
  > **Comments:**
  > **username_1**: Very interesting question!  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_7 on 2012-11-13. Score: 13 | Accepted answer
  
  I'd recommend a hybrid approach using both computers and people.
  
  1. Bucket the videos by their length (round to the nearest second)
  2. For each bucket, use ffmpeg to [generate thumbnails](http://blog.prashanthellina.com/2008/03/29/creating-video-thumbnails-using-ffmpeg/) at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)
  3. Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.
  
  You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse. Good luck!
  
  > **Comments:**
  > **username_0**: Thumbnails are a great idea! The new videos came in as FLV, which I can't seem to thumbnail no matter what I try. This could work quite well  
  > **username_7**: ffmpeg should be able to convert the flv files into something else (avi, mp4, etc).  
  > **username_0**: Yeah, I'll eventually convert them to MP4 but I want to make sure I lose as little quality as possible when I do that, so I've put it off until I can take the time  
  > **username_8**: If you wanted to, you could find a program that would analyze the thumbnails to find duplicates. However, the time spent finding such a program then configuring it would not be worth it for a small project.  
  > **username_9**: @BenBrocka Another solution is you make an unique hash string (exampley 8 digits character) from your file in uploading time  and store it on your db, and add a condition on stored unique string for checking duplicate videos in upload time.  
  
  ---
  
  ### Answer by username_2 on 2021-03-29. Score: 2
  
  This seems related to : <https://video.stackexchange.com/a/33521/32436>
  
  > You could check out
  > <https://theophanemayaud.github.io/video-simili-duplicate-cleaner/>
  > (macOS and Windows binaries available) It's open source, I created it with QT and it relies on ffmpeg.
  >
  > The idea is to extract a few (eg 2, the first and the last) frames,
  > and compare them. If the duration is close, and the frames look
  > similar : you've got a duplicate.
  
  You can "manually compare the extracted frames", or use thresholds to configure some automatic behaviors, and trash the duplicates.
  
  > It uses opencv for frame comparison, and ffmpeg for frame extraction
  > (ffmpeg is also for video metadata which is very useful when trying to
  > tell which video to keep !). Since it's open source, you can improve
  > it to fit your needs !
  
  I've found it to be really helpful in a situation like yours, where I had about 15 000 videos from google photos which compresses them, Picasa, handbrake compressed, original copies, and other changed versions of the same videos !
  
  
  ---
  
  ### Answer by username_3 on 2013-03-13. Score: 1
  
  You can try Video Comparer (shareware).
  
  <http://www.video-comparer.com>
  
  It quickly detects similar videos with image transformations and supports split videos into multiple CDs.
  
  > **Comments:**
  > **username_4**: This software did not detect *similar* videos (same scene/person taken a few minutes apart).  
  
  ---
  
  ### Answer by username_5 on 2013-11-12. Score: 1
  
  I had the same problem recently and I found the program that helped me. It doesn't compare file names or calculate hashes. It compare real video content. It is not free, but if you haven't many duplicates, the trial version is enough. The link to the program is <http://duplicatevideosearch.com>
  
  
  ---
  
  ### Answer by username_6 on 2015-03-19. Score: -1
  
  Similar images - <https://tn123.org/simimages/> is a bit old, but still functional.
  
  
  ---
  
  