Answers.Body: <p>Yes, you have the right idea.  If stib's suggestion of making multiple outputs in parallel with the same ffmpeg commandline doesn't quite do the trick, then use a temp file to hold a lossless copy of the output of any slow filtering:</p>
  
  <pre><code>ffmpeg -i src.mkv -vf yadif=3:1,mcdeint=2:1 -c:a copy -c:v libx264 -preset ultrafast -qp 0 deinterlaced.mkv
  </code></pre>
  
  <p>(never use <code>ultrafast</code> except with lossless, it's WAY worse than <code>superfast -x264-params cabac=0</code>, and not much faster.)  CABAC takes most of the CPU time at lossless bitrates, but it does save 15% bitrate vs. CAVLC.  If you have a ton of scratch space to work with, you could use an uncompressed output file.  Lossless x264 is about as fast to decode as <code>ffvhuff</code>, if not faster, but half the file size.  I think it's your best bet for scratch files (pretty good CPU usage, ASM-optimized decoders, and one of the best as far as compression ratio).  Simpler codecs like ffvhuff might have a place if you need to frame-step backwards without taking seconds per frame, or you could just lower <code>keyint</code> in x264.</p>
  
  <p>Then do whatever you normally do, but with <code>deinterlaced.mkv</code> as your source.</p>
  
  <p>I think I remember reading about someone having added support to x264 for making multiple encodes at different bitrates at the same time.  So motion-search, frame-type decision, and a bunch of other analysis only has to be done once.  I'm not sure if that was just for google's internal use, and it never got merged, or what.  Neither the vanilla x264 cli frontend, nor the ffmpeg libx264 wrapper, have options to activate a feature like this.  I'm pretty sure I would have noticed that in <code>x264 --fullhelp</code> output at some point.</p>
  
  <p>If you have to do a lot of encoding, it'd be worth searching around to see if there's any way to get your hands on a publicly available implementation of this.</p>
  
  <p>This is different from having ffmpeg feed the decoded/filtered video to multiple instances of libx264 in parallel, since each instance would still need to do all the analysis independently.</p>
  
  <p>If you do use stib's suggestion of multiple outputs from the same ffmpeg, reduce x264's number of threads.  By default, each separate output file will auto-detect the number of CPUs in the whole machine, so you'll have 6 times the number of worker threads needed to saturate your CPU cores.  Maybe use 2 or 3 threads for the HD outputs, and single-threaded for the lower-rez outputs, since you want to balance things so all your available CPU time gets used, without having more threads than needed causing extra context-switching overhead.</p>
  
  <p>For HD video with a lot of reference frames, each libx264 instance would take a good chunk of RAM, too.  Reducing thread count doesn't help with this much, since all the threads are reading the same data.</p>
  
  <p>6 instances of libx264 at the same time, instead of multiple threads from the same libx264, will be competing for memory bandwidth / cache space.  They'll each have their own copy of the frames, instead of getting cache hits from each other.  This might or might not be an issue on your hardware.  If this or the RAM-size is an issue, use a temp file and do 3 outputs from one run, and 3 outputs from the next.</p>
  
  <p>yadif (and mcdeint) are not multithreaded.</p>
  
  <p>Or you could deinterlace and output to lossless + some of your desired output rezzes, then generate the rest of your rezzes in another pass of the lossless file.  If you don't have multiple videos queued up to process, this would let you pipeline things to get max use out of all your cores while still not doing all the encodes at once.</p>
  
  <p>If you do have multiple videos to process, you could have a video deinterlacing while you encode another, if you think setting that up is simpler than multiple outputs + lossless from the ffmpeg that runs the deinterlace.  Run the encodes with <code>nice -19</code> or something, so they don't take CPU away from the <code>ffmpeg</code> process doing the deinterlace.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-02-23T09:43:33.030Z
Answers.Text: Thanks for the long answer. I will try out your solution and let you know of the results.
Answers.UserDisplayName: Gorgi Rankovski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-03T16:24:06.213Z
Answers.Text: Not sure if you could do that when the question was asked, but now (not sure since when) ffmpeg supports a yadif_cuda filter that can use a GPU for part of the heavy lifting.
Answers.UserDisplayName: rbanffy
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-02-23T05:55:12.407Z
Answers.LastActivityDate: 2015-02-23T05:55:12.407Z
Answers.UserDisplayName: Peter Cordes
Body: <p>We are using two-step encoding with ffmpeg to encode raw .mov files to multiple .mp4 encodings ready for streaming over the Internet. Some of the videos are interlaced and we are using yadif to deinterlace them with the following options: yadif=1:-1:0. </p>
  
  <p>Without yadif, regular 2 hour videos take about 3 hours to encode into 6 different .mp4 files (different resolution and bitrates mostly). When adding yadif, encoding takes 6-10 hours. </p>
  
  <p>Our knowledge with videos is limited, so we are wondering can we make the deinterlacing faster? What if first deinterlace the source .mov file into new .mov file and use that as a source for mp4s, will that be faster?</p>
  
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-16T19:00:05.673Z
Comments.Text: Please show all of your `ffmpeg` commands and the complete console output from each. You can add `-t 10` from your commands to just output a 10 second segment.
Comments.UserDisplayName: llogan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-17T11:13:08.023Z
Comments.Text: The ffmpeg pipeline basically goes: decode the image and create a raw frame > apply filters, scaling, cropping rotation etc. to the raw frame > encode to output(s). I doubt that repeating the first and third step will make the process faster. You know you can encode to multiple outputs in the one command don't you? Thus: `ffmpeg -i in.mov -c:v foo out1.mov -c:v bar out2.mov -c:v fuz out3.mkv`. This saves on the decoding step for each frame.
Comments.UserDisplayName: stib
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-02-23T09:42:57.700Z
Comments.Text: Thanks @stib, I am actually using multiple outputs with one command. I am wandering if we can get any faster then that.
Comments.UserDisplayName: Gorgi Rankovski
Tags: <ffmpeg><encoding>
Title: Faster deinterlacing with ffmpeg and yadif
UserDisplayName: Gorgi Rankovski
ThreadText: # Faster deinterlacing with ffmpeg and yadif
  
  (Asked by: username_0 on 2015-02-16)
  
  We are using two-step encoding with ffmpeg to encode raw .mov files to multiple .mp4 encodings ready for streaming over the Internet. Some of the videos are interlaced and we are using yadif to deinterlace them with the following options: yadif=1:-1:0.
  
  Without yadif, regular 2 hour videos take about 3 hours to encode into 6 different .mp4 files (different resolution and bitrates mostly). When adding yadif, encoding takes 6-10 hours.
  
  Our knowledge with videos is limited, so we are wondering can we make the deinterlacing faster? What if first deinterlace the source .mov file into new .mov file and use that as a source for mp4s, will that be faster?
  
  Thanks.
  
  **Tags:** ffmpeg, encoding
  
  > **Comments:**
  > **username_1**: Please show all of your `ffmpeg` commands and the complete console output from each. You can add `-t 10` from your commands to just output a 10 second segment.  
  > **username_2**: The ffmpeg pipeline basically goes: decode the image and create a raw frame > apply filters, scaling, cropping rotation etc. to the raw frame > encode to output(s). I doubt that repeating the first and third step will make the process faster. You know you can encode to multiple outputs in the one command don't you? Thus: `ffmpeg -i in.mov -c:v foo out1.mov -c:v bar out2.mov -c:v fuz out3.mkv`. This saves on the decoding step for each frame.  
  > **username_0**: Thanks @username_2, I am actually using multiple outputs with one command. I am wandering if we can get any faster then that.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2015-02-23. Score: 6
  
  Yes, you have the right idea. If username_2's suggestion of making multiple outputs in parallel with the same ffmpeg commandline doesn't quite do the trick, then use a temp file to hold a lossless copy of the output of any slow filtering:
  
  ```
  ffmpeg -i src.mkv -vf yadif=3:1,mcdeint=2:1 -c:a copy -c:v libx264 -preset ultrafast -qp 0 deinterlaced.mkv
  
  ```
  
  (never use `ultrafast` except with lossless, it's WAY worse than `superfast -x264-params cabac=0`, and not much faster.) CABAC takes most of the CPU time at lossless bitrates, but it does save 15% bitrate vs. CAVLC. If you have a ton of scratch space to work with, you could use an uncompressed output file. Lossless x264 is about as fast to decode as `ffvhuff`, if not faster, but half the file size. I think it's your best bet for scratch files (pretty good CPU usage, ASM-optimized decoders, and one of the best as far as compression ratio). Simpler codecs like ffvhuff might have a place if you need to frame-step backwards without taking seconds per frame, or you could just lower `keyint` in x264.
  
  Then do whatever you normally do, but with `deinterlaced.mkv` as your source.
  
  I think I remember reading about someone having added support to x264 for making multiple encodes at different bitrates at the same time. So motion-search, frame-type decision, and a bunch of other analysis only has to be done once. I'm not sure if that was just for google's internal use, and it never got merged, or what. Neither the vanilla x264 cli frontend, nor the ffmpeg libx264 wrapper, have options to activate a feature like this. I'm pretty sure I would have noticed that in `x264 --fullhelp` output at some point.
  
  If you have to do a lot of encoding, it'd be worth searching around to see if there's any way to get your hands on a publicly available implementation of this.
  
  This is different from having ffmpeg feed the decoded/filtered video to multiple instances of libx264 in parallel, since each instance would still need to do all the analysis independently.
  
  If you do use username_2's suggestion of multiple outputs from the same ffmpeg, reduce x264's number of threads. By default, each separate output file will auto-detect the number of CPUs in the whole machine, so you'll have 6 times the number of worker threads needed to saturate your CPU cores. Maybe use 2 or 3 threads for the HD outputs, and single-threaded for the lower-rez outputs, since you want to balance things so all your available CPU time gets used, without having more threads than needed causing extra context-switching overhead.
  
  For HD video with a lot of reference frames, each libx264 instance would take a good chunk of RAM, too. Reducing thread count doesn't help with this much, since all the threads are reading the same data.
  
  6 instances of libx264 at the same time, instead of multiple threads from the same libx264, will be competing for memory bandwidth / cache space. They'll each have their own copy of the frames, instead of getting cache hits from each other. This might or might not be an issue on your hardware. If this or the RAM-size is an issue, use a temp file and do 3 outputs from one run, and 3 outputs from the next.
  
  yadif (and mcdeint) are not multithreaded.
  
  Or you could deinterlace and output to lossless + some of your desired output rezzes, then generate the rest of your rezzes in another pass of the lossless file. If you don't have multiple videos queued up to process, this would let you pipeline things to get max use out of all your cores while still not doing all the encodes at once.
  
  If you do have multiple videos to process, you could have a video deinterlacing while you encode another, if you think setting that up is simpler than multiple outputs + lossless from the ffmpeg that runs the deinterlace. Run the encodes with `nice -19` or something, so they don't take CPU away from the `ffmpeg` process doing the deinterlace.
  
  > **Comments:**
  > **username_0**: Thanks for the long answer. I will try out your solution and let you know of the results.  
  > **username_4**: Not sure if you could do that when the question was asked, but now (not sure since when) ffmpeg supports a yadif_cuda filter that can use a GPU for part of the heavy lifting.  
  
  ---
  
  