Answers.Body: <p>Since <a href="https://summerofcode.withgoogle.com/archive/2018/projects/5661133578960896/" rel="nofollow noreferrer">Google Summer of Code 2018</a>, FFMpeg has supported <a href="https://ffmpeg.org/ffmpeg-filters.html#sr-1" rel="nofollow noreferrer">the <code>sr</code> filter</a> for applying super-resolution methods based on convolutional neural networks. However, as you have discovered, few super-resolution tutorials exist, and compiling FFMpeg with proper libraries and preparing models for super-resolution requires expert knowledge.</p>
  <p>To make super-resolution in FFMpeg easier, <a href="https://video.stackexchange.com/users/32410/mikul%c3%a1%c5%a1-bankovi%c4%8d">Mikuláš</a>, <a href="https://github.com/dlueth" rel="nofollow noreferrer">Dirk</a> and I have taken the excellent answer of <a href="https://video.stackexchange.com/users/32099/mark">Mark</a>, and we used it to prepare <a href="https://hub.docker.com/r/miratmu/ffmpeg-tensorflow" rel="nofollow noreferrer">a Docker image with FFMpeg and Libtensorflow</a>. We also wrote a tutorial that explains step-by-step how to use the Docker image for super-resolution in FFMpeg: <a href="https://github.com/MIR-MU/ffmpeg-tensorflow#ffmpeg-with-libtensorflow" rel="nofollow noreferrer">https://github.com/MIR-MU/ffmpeg-tensorflow#ffmpeg-with-libtensorflow</a>.</p>
  <p>After installation, super-resolution in FFMpeg becomes as easy as this:</p>
  <pre><code>$ wget https://media.xiph.org/video/derf/y4m/flower_cif.y4m
  $ alias ffmpeg-tensorflow='docker run --rm --gpus all -u $(id -u):$(id -g) -v &quot;$PWD&quot;:/data -w /data -it miratmu/ffmpeg-tensorflow'
  $ ffmpeg-tensorflow -i flower_cif.y4m -filter_complex '
  &gt;   [0:v] format=pix_fmts=yuv420p, extractplanes=y+u+v [y][u][v];
  &gt;   [y] sr=dnn_backend=tensorflow:scale_factor=2:model=espcn.pb [y_scaled];
  &gt;   [u] scale=iw*2:ih*2 [u_scaled];
  &gt;   [v] scale=iw*2:ih*2 [v_scaled];
  &gt;   [y_scaled][u_scaled][v_scaled] mergeplanes=0x001020:yuv420p [merged]
  &gt; ' -map [merged] -sws_flags lanczos \
  &gt; -c:v libx264 -crf 17 -preset ultrafast -tune film \
  &gt; -c:a copy \
  &gt; -y flower_cif_2x.mp4
  </code></pre>
  <p>Compare upscaling using Lanczos (left) with the ESPCN super-resolution model (right):</p>
  <p><a href="https://i.sstatic.net/DzYz3.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DzYz3.jpg" alt="Comparison of Lanczos and ESPCN" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-10T18:48:18.047Z
Answers.LastActivityDate: 2021-02-09T11:25:27.047Z
Answers.LastEditDate: 2021-02-09T11:25:27.047Z
Answers.UserDisplayName: Witiko
Answers.Body: <p>Although Mark has provided great answer, you don't need to train models from scratch, all Super-Resolution models that ffmpeg supports have pre-trained checkpoints:
  <a href="https://github.com/XueweiMeng/sr/tree/sr_dnn_native" rel="nofollow noreferrer">https://github.com/XueweiMeng/sr/tree/sr_dnn_native</a>
  all you need to do is to convert it to frozen graph (.pb) file and then use it in ffmpeg.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:33:48.030Z
Answers.Text: Upvoted. Could you elaborate the _"all you need to do"_ part so that from reading the answer alone, the reader will know what to do?
Answers.UserDisplayName: Redsandro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T17:55:33.717Z
Answers.Text: Sure, no problem.
  1. Download repo with models https://github.com/XueweiMeng/sr/tree/sr_dnn_native
  2. Install all requirements using `pip install`, for me it was only tensorflow and numpy packages.
  3. Run script to generate model you need: `python3 generate_header_and_model.py --model=espcn --ckpt_path=checkpoints/espcn/`
  4. 2 model files will be generated - `espcn.mode`l and `espcn.pb` (you can check that model works by evaluating .pb model in tensorflow)
  5. Use one of these models in ffmpeg with sr filter (or dnn_processing filter), as stated in documentation.
Answers.UserDisplayName: Alex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T18:00:08.430Z
Answers.Text: Note: you might need to install TensorFlow C library and configure ffmpeg if you want to use tensorflow dnn_backend.
Answers.UserDisplayName: Alex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T20:03:43.787Z
Answers.Text: @Alex Very interesting! We provide a Docker image with precompiled tensorflow-capable FFMpeg and `.pb` files in https://github.com/MIR-MU/ffmpeg-tensorflow. However, I haven't found much documentation about these `.model` and `.pb` files. Are there sources that you would recommend for reading up on the common formats for storing computational graphs?
Answers.UserDisplayName: Witiko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T11:00:03.277Z
Answers.Text: @Witiko Well, it's just a specific format for saved models:
  https://stackoverflow.com/questions/51278213/what-is-the-use-of-a-pb-file-in-tensorflow-and-how-does-it-work
Answers.UserDisplayName: Alex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-27T05:55:05.280Z
Answers.Text: These guys have ~70 Mb prtrained tflite model , they claim it is optimized to work better with different resolutions. They tiled GAN I guess. 
  Once I tiled ESR-GAN it doesnt show any difference in speed or quality however it introduces some blurred lines at tiling border. So be careful
  https://www.collabora.com/news-and-blog/blog/2020/09/21/open-source-meets-super-resolution-part-1/
Answers.UserDisplayName: Gediz GÜRSU
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-05T19:13:18.717Z
Answers.LastActivityDate: 2021-04-05T19:13:18.717Z
Answers.UserDisplayName: Alex
Answers.Body: <p>If all you need to do is superscale an image, going through the trouble of training neural networks is re-inventing the wheel.  By all means, if you're studying computer science and are interested in AI/ML, I'd encourage you to look into it, but to just superscale an image, you don't need to train a neural network.  There are tools available.</p>
  
  <p>In DaVinci Resolve, for instance, all you need to do is right-click on a clip in the media pool, select "clip attributes...," then at the bottom of the "video" tab, change "Super Scale: None" to 2x, 3x, or 4x.  I'm not sure if this feature is available in the free version, but I wouldn't be surprised if it is, because all they've done is integrated the open source ML bits of FFMPEG for you, trained the models for you, and bundled it as a feature.  I'm sure Adobe has an equivalent feature, though I haven't used it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-21T12:57:55.737Z
Answers.Text: I don't mind using a pre-trained model, but I would like to know how to do this using `ffmpeg` exclusively so I can incorporate it in a pipeline (e.g. `kdenlive`). No commercial software.
Answers.UserDisplayName: Redsandro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-17T20:50:02.610Z
Answers.Text: According to [this](https://www.toolfarm.com/tutorial/in-depth-davinci-resolve-studio-vs-the-free-version/) it's not in the free version (which *is* DaVinci Resolve, as opposed to DaVinci Resolve Studio). Given that, and given DaVince Resolve Studio costs $300 dollars, and *especially* given what you claim about them basing their SR on FFmpeg (I would definitely be interested in a source for that claim), FFmpeg is currently the best free tool for the purpose out there.
Answers.UserDisplayName: Hashim Aziz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-17T21:31:08.633Z
Answers.Text: @HashimAziz That’s about R16.  R17 has been out for a while.  I don’t know if superscale is one of the features that got “unlocked” this time around, but BMD always promotes some previously studio-only features to the free version with every update cycle, so there’s no harm in trying it out.  Plus, R18 shouldn’t be too far away.
Answers.UserDisplayName: Jason Conrad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-09T22:55:42.487Z
Answers.Text: @JasonConrad R19 is now in beta, and Super Scale remains exclusive to (non-free) Resolve Studio.
Answers.UserDisplayName: mysteryegg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-19T21:13:14.923Z
Answers.LastActivityDate: 2020-01-19T21:13:14.923Z
Answers.UserDisplayName: Jason Conrad
Answers.Body: <p>I realise this question is pretty old now but it still comes up quite high in search results so I would like to document how I got the &quot;sr&quot; to work (August 2020) in case it can help someone else. Before proceeding it is worth saying that I was not overly impressed by the result. I felt on my test videos that lanczos did an ever so slightly better job. Bear in mind that my test was upscaling from 640x360 flv file to 1920x1080 mp4. I have not tested many other videos and I also have quite a lot to learn about SR like if I feed it different training material will I get a better result.</p>
  <p><strong>What you will need</strong></p>
  <ul>
  <li>A CUDA enabled GPU with a capability of at least 6.0. I worked all
  the way through this once with a GeForce GT710 (capability of 3.5)
  only to get to the final ffmpeg upscaling for it to tell me I needed
  a capability of 6.0. It did proceed to use the CPU but was very slow.
  I then went through it again with a GeForce GTX 1050TI 4GB card and
  saw a noticable performance gain.</li>
  <li>At least 70GB free disk space. The training scripts download about 42GB worth of videos and 9GB worth of images. The rest is used up generating the model.</li>
  <li>Patience. Generating the datasets took a good few hours and training just one model took nearly the whole day.</li>
  <li>I had a couple of spare computers (one server and one pc) so I was able to build this from scratch and let it run. First I tried it on Ubuntu Server 20.04 and 18.04. Final run through with the new GPU was on Xubuntu 20.04. The only reason I went with Xubuntu in the end was because I was also investigating a new USB capture device for VHS but the instructions below will work with Server.</li>
  </ul>
  <p><strong>Setup</strong></p>
  <p>THIS GUIDE IS OFFERED AS IS WITH NO GUARANTEES IT WILL WORK FOR YOU. PLEASE TAKE CARE AND ONLY FOLLOW IF YOU KNOW WHAT YOU ARE DOING.</p>
  <p>My starting point is a fresh install of Xubuntu 20.04. I have a GeForce GTX 1050TI GPU that is not my main GPU and there is no monitor connected to it. I do not think it should matter though if it is your main GPU or if there is a monitor connected.</p>
  <p><em>FROM <a href="https://www.tensorflow.org/install/gpu" rel="noreferrer">https://www.tensorflow.org/install/gpu</a></em></p>
  <p>Add NVIDIA package repositories</p>
  <pre><code>wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
  sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
  sudo dpkg -i cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
  sudo apt-get update
  wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
  sudo apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
  sudo apt-get update
  </code></pre>
  <p>Install NVIDIA driver</p>
  <pre><code>sudo apt-get install --no-install-recommends nvidia-driver-450
  </code></pre>
  <p>REBOOT, then check that GPUs are visible using the command: <code>nvidia-smi</code>. Do not proceed if you do not see a table output. If you have a message saying there is no device then you need to troubleshoot this stage. For example in my first build the BIOS was set to use the onboard graphics card first and changing it to use the offboard card got it working. I didn't have this problem in the last build on a different PC.</p>
  <p>Install development and runtime libraries (~4GB). Here we are installing both CUDA 10.0, 10.1. The training scripts and filter are a couple of years old now and we therefore need to accommodate them.</p>
  <pre><code>sudo apt-get install --no-install-recommends \
      cuda-10-0 \
      libcudnn7=7.6.5.32-1+cuda10.0  \
      libcudnn7-dev=7.6.5.32-1+cuda10.0
  </code></pre>
  <p>Install TensorRT. Requires that libcudnn7 is installed above.</p>
  <pre><code>sudo apt-get install --no-install-recommends libnvinfer6=6.0.1-1+cuda10.0 \
      libnvinfer-dev=6.0.1-1+cuda10.0 \
      libnvinfer-plugin6=6.0.1-1+cuda10.0
  </code></pre>
  <p>And now the 10.1 versions</p>
  <pre><code>sudo apt-get install --no-install-recommends \
      cuda-10-1 \
      libcudnn7=7.6.5.32-1+cuda10.1  \
      libcudnn7-dev=7.6.5.32-1+cuda10.1
  
  sudo apt-get install --no-install-recommends libnvinfer6=6.0.1-1+cuda10.1 \
      libnvinfer-dev=6.0.1-1+cuda10.1 \
      libnvinfer-plugin6=6.0.1-1+cuda10.1
  </code></pre>
  <p><em>FROM <a href="https://www.tensorflow.org/install/pip#system-install" rel="noreferrer">https://www.tensorflow.org/install/pip#system-install</a></em></p>
  <p>The guide above suggests to install venv here but this was not a concern for me so I left it out. If you are familiar with Python then go ahead and use venv.</p>
  <pre><code>sudo apt install python3-dev python3-pip
  sudo pip3 install --upgrade tensorflow
  </code></pre>
  <p>Add some libraries temporarily to LD_LIBRARY_PATH. We will make this permanent a little later.</p>
  <pre><code>export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/local/cuda-10.0/lib64:/usr/local/cuda-10.1/lib64:/usr/local/cuda-10.2/lib64:/usr/local/cuda/extras/CUPTI/lib64
  </code></pre>
  <p>You will notice I am adding cuda 10.2 here as well but we did not install it. For some reason one of the library files (libcublas) when installing 10.1 is taken from 10.2.</p>
  <p>We can verify with the below (check for errors). There is a lot of output, mainly information and you want to check that it has seen your GPU and has successfully opened all the libraries.</p>
  <pre><code>python3 -c &quot;import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))&quot;
  </code></pre>
  <p>Add LD_LIBRARY_PATH to ld.so.conf.d</p>
  <pre><code>sudo vi /etc/ld.so.conf.d/cuda-additional.conf
  </code></pre>
  <p>Add the lines</p>
  <pre><code>/usr/local/cuda/extras/CUPTI/lib64
  /usr/local/cuda-10.2/lib64
  </code></pre>
  <p>The other 2 paths are automatically added for us during install.</p>
  <p><em>FROM <a href="https://ffmpeg.org/ffmpeg-filters.html#sr-1" rel="noreferrer">https://ffmpeg.org/ffmpeg-filters.html#sr-1</a></em>
  Install the TensorFlow for C library. If you follow the link given at the above it shows you Tensorflow version 2.3.0 (or later), however, as mentioned earlier the filter code is old and requires version 1.15.0. So:</p>
  <pre><code>wget https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-1.15.0.tar.gz
  sudo tar -C /usr/local -xzf libtensorflow-gpu-linux-x86_64-1.15.0.tar.gz
  </code></pre>
  <p><em>FROM <a href="https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu" rel="noreferrer">https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu</a></em>
  Now we fetch and compile ffmpeg. So install some dependencies first:</p>
  <pre><code>sudo apt-get update -qq &amp;&amp; sudo apt-get -y install \
    autoconf \
    automake \
    build-essential \
    cmake \
    git-core \
    libass-dev \
    libfreetype6-dev \
    libgnutls28-dev \
    libsdl2-dev \
    libtool \
    libva-dev \
    libvdpau-dev \
    libvorbis-dev \
    libxcb1-dev \
    libxcb-shm0-dev \
    libxcb-xfixes0-dev \
    pkg-config \
    texinfo \
    wget \
    yasm \
    zlib1g-dev
  
  mkdir -p ~/ffmpeg_sources ~/bin
  sudo apt-get install nasm libx264-dev libx265-dev libnuma-dev libvpx-dev libfdk-aac-dev libmp3lame-dev libopus-dev libunistring-dev
  
  cd ~/ffmpeg_sources &amp;&amp; \
  wget -O ffmpeg-snapshot.tar.bz2 https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2 &amp;&amp; \
  tar xjvf ffmpeg-snapshot.tar.bz2 &amp;&amp; \
  cd ffmpeg &amp;&amp; \
  PATH=&quot;$HOME/bin:$PATH&quot; PKG_CONFIG_PATH=&quot;$HOME/ffmpeg_build/lib/pkgconfig&quot; ./configure \
    --prefix=&quot;$HOME/ffmpeg_build&quot; \
    --pkg-config-flags=&quot;--static&quot; \
    --extra-cflags=&quot;-I$HOME/ffmpeg_build/include&quot; \
    --extra-ldflags=&quot;-L$HOME/ffmpeg_build/lib&quot; \
    --extra-libs=&quot;-lpthread -lm&quot; \
    --bindir=&quot;$HOME/bin&quot; \
    --enable-gpl \
    --enable-gnutls \
    --enable-libass \
    --enable-libfdk-aac \
    --enable-libfreetype \
    --enable-libmp3lame \
    --enable-libopus \
    --enable-libvorbis \
    --enable-libvpx \
    --enable-libx264 \
    --enable-libx265 \
    --enable-libtensorflow \
    --enable-nonfree &amp;&amp; \
  PATH=&quot;$HOME/bin:$PATH&quot; make &amp;&amp; \
  make install &amp;&amp; \
  hash -r
  </code></pre>
  <p>Notice that I have left out libaom and added libtensorflow. This has created new binaries under ~/bin and added it to your profile and path so it can be called from anywhere. If you do not want to replace your existing ffmpeg then be sure to change the bindir and paths as required.</p>
  <p>Reload our profile</p>
  <pre><code>source ~/.profile
  </code></pre>
  <hr />
  <p><em>FROM <a href="https://github.com/XueweiMeng/sr/tree/sr_dnn_native" rel="noreferrer">https://github.com/XueweiMeng/sr/tree/sr_dnn_native</a></em></p>
  <p>So now we are almost at the stage where we can start to generate and train the models, however, the scripts are old so we need to modify them somewhat. Get them first (you can specify any path to download it to or get it manually with GUI:</p>
  <pre><code>cd ~
  wget https://github.com/XueweiMeng/sr/archive/sr_dnn_native.zip
  unzip sr_dnn_native.zip
  rm sr_dnn_native.zip
  cd sr-sr_dnn_native
  </code></pre>
  <p>Now edit all python files that have &quot;import tensorflow as tf&quot;, i.e.</p>
  <pre><code>datasets/prepare_dataset.py
  datasets/prepare_div2k_dataset.py
  evaluate.py
  generate_header_and_model.py
  models/model_vespcn.py
  models/model_espcn.py
  models/image_warp.py
  models/model_srcnn.py
  models/model.py
  models/dataset.py
  models/model_vsrnet.py
  train.py
  </code></pre>
  <p>After the imports in each script add the line:</p>
  <pre><code>tf = tf.compat.v1
  </code></pre>
  <p>For example using datasets/prepare_dataset.py:</p>
  <pre><code>import os
  import argparse
  from tqdm import tqdm
  import cv2
  import numpy as np
  import json
  import tensorflow as tf
  from PIL import Image
  
  tf = tf.compat.v1
  
  
  class SceneChangeDetector:
  </code></pre>
  <p>There is one other depreciated function we need to take care of and that is imresize.</p>
  <p>Edit the file <code>datasets/prepare_dataset.py</code></p>
  <p>Replace line 8: <code>from scipy.misc import imresize</code></p>
  <p>With: <code>from PIL import Image</code></p>
  <p>Replace line 127: <code>frame_lr = imresize(frames[k], (lr_h, lr_w), interp='bicubic')</code></p>
  <p>With: <code>frame_lr = np.array(Image.fromarray(frames[k]).resize(size=(lr_h, lr_w)))</code></p>
  <p>We can finally start generating. Install some dependencies.</p>
  <pre><code>sudo pip3 install Pillow tqdm opencv-python
  </code></pre>
  <p>Generate (expect it to take a good few hours and you will likely see a lot of information output and warnings of almost depreciated functions, these are ok):</p>
  <pre><code>sh generate_datasets.sh
  </code></pre>
  <p>This will download 11 videos totalling around 43GB and images totalling around 9GB and can take a few hours to run depending on hardware.</p>
  <p>And now for the training (expect these to take about a day each). The models after this will be found in the extracted <code>~/sr-sr_dnn_native</code> folder if that is where you saved the Git zip file to:</p>
  <p>SRCNN</p>
  <pre><code>sh train_srcnn.sh
  python3 generate_header_and_model.py --model=srcnn --ckpt_path=logdir/srcnn_batch_32_lr_1e-3_decay_adam/train
  </code></pre>
  <p>ESPCN</p>
  <pre><code>sh train_espcn.sh
  python3 generate_header_and_model.py --model=espcn --ckpt_path=logdir/espcn_batch_32_lr_1e-3_decay_adam/train
  </code></pre>
  <hr />
  <p>Once the training is complete you can use the filter. It seems as though ESPCN only upscales by a factor of 2 whereas with SRCNN you can specify 2, 3, or 4. I moved the models generated into the same folder as my videos for ease of use but you can simply point to the models wherever they are.</p>
  <p>e.g.</p>
  <pre><code>ffmpeg -i &lt;input_video&gt; -vf sr=dnn_backend=tensorflow:scale_factor=3:model=srcnn.pb -q 15 -preset slow &lt;output_video&gt;
  ffmpeg -i &lt;input_video&gt; -vf sr=dnn_backend=tensorflow:model=espcn.pb -q 15 -preset slow &lt;output_video&gt;
  </code></pre>
  <hr />
  <p>Here is an interesting video of this in use and was probably one of the only places I have seen anyone use this. In it he suggests it can only work on a single plane but I did not find that to be the case. His results actually appear to be quite good:</p>
  <p><div class="youtube-embed"><div>
                  <iframe width="640px" height="395px" src="https://www.youtube.com/embed/R6Ly0GYlXa4?start=0"></iframe>
              </div></div></p>
  <p>This is the ffmpeg command he refers to in the video at 14:44:</p>
  <pre><code>ffmpeg -i my_video_540p.mp4 -filter_complex &quot;format=pix_fmts=yuv420p,extractplanes=y+u+v[y][u][v];[y] sr=dnn_backend=tensorflow:scale_factor=2:model=espcn.model [y_scale];[u] scale=960:-2 [u_scale];[v] scale=960:-2 [v_scale];[y_scale][u_scale][v_scale] mergeplanes=0x001020:yuv420p [merged]&quot; -map &quot;[merged]&quot; -c:v libx264 -crf 18 -pix_fmt yuv420p my_video_1080p.mp4
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T11:28:38.493Z
Answers.Text: I am happy with your extensive write-up, and I have accepted your answer after reading this. Also I'm a bit sad that you made a lot of effort yet still observe results similar lanczos at best.
  
  Great first contribution! Thank you.
Answers.UserDisplayName: Redsandro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-03T12:40:47.210Z
Answers.Text: Thank you. I am not giving up on it just yet because looking at "datasets/prepare_dataset.py" from line 166 and also in "train.py" there are many options that we can play with (passed in as parameters) and I believe with the right training I could achieve better results. I'll post back if I manage to get something amazing.
Answers.UserDisplayName: Mark
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-14T01:45:21.070Z
Answers.Text: Using a linux box with an old nvidia driver (418), I decided to install to a docker image `nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04`. This meant skipping everything until libtensorflow installation and installing libtensorflow 1.12.3 rather than 1.15.0 (see https://www.tensorflow.org/install/source#gpu). Additionally, HighVoltageRocknRoll's repo is more up to date than XueweiMeng's, so I used the former, and since libtensorflow was precompiled for higher compute capability (6) than my GPU (5), I compiled it via `tensorflow/tools/ci_build/linux/libtensorflow_gpu.sh` from the Tensorflow repo.
Answers.UserDisplayName: Witiko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-16T01:58:24.277Z
Answers.Text: I really wish this were a container image
Answers.UserDisplayName: mikebabcock
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-10T18:49:25.290Z
Answers.Text: @mikebabcock [Mikuláš](https://video.stackexchange.com/users/32410/mikul%c3%a1%c5%a1-bankovi%c4%8d) and I have prepared a Docker image, see [my answer](https://video.stackexchange.com/a/32418/23079).
Answers.UserDisplayName: Witiko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-08T18:09:04.163Z
Answers.Text: Is there any google colab available?
Answers.UserDisplayName: Radical Edward
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-17T20:52:44.903Z
Answers.Text: Thank you for this. Hopefully as the technology matures the results will get better.
Answers.UserDisplayName: Hashim Aziz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-02T10:25:21.427Z
Answers.LastActivityDate: 2020-09-02T10:25:21.427Z
Answers.UserDisplayName: Mark
Body: <p>I need to incorporate a small amount of 1080p HD shots in a 2160p 4K composition. I'm not impressed by default upscaling techniques like lanczos/bicubic/trilinear; it seems like my television does a better job of upscaling 1080p in real time. It may be time to learn some new tricks.</p>
  
  <p>The FFmpeg manual mention a technique called <a href="https://ffmpeg.org/ffmpeg-filters.html#sr" rel="noreferrer">Super Resolution</a>, and it comes in two forms:</p>
  
  <ul>
  <li><code>SRCNN</code>: Super-Resolution Convolutional Neural Network model</li>
  <li><code>ESPCN</code>: Efficient Sub-Pixel Convolutional Neural Network model</li>
  </ul>
  
  <p>Usually I can find many online sources for figuring out how to do something with FFmpeg. But I can't seem to find any tutorials on how to use this. I don't really understand it. I need to train (how?) a model (from what?), or get a pre-trained model (where?). I'd like to know how to get from 1080p to 2160p using this technique, assuming I know nothing about it.</p>
  
Tags: <ffmpeg><scale><quality>
Title: How do the super resolution filters in FFmpeg work?
UserDisplayName: Redsandro
ThreadText: # How do the super resolution filters in FFmpeg work?
  
  (Asked by: username_0 on 2020-01-19)
  
  I need to incorporate a small amount of 1080p HD shots in a 2160p 4K composition. I'm not impressed by default upscaling techniques like lanczos/bicubic/trilinear; it seems like my television does a better job of upscaling 1080p in real time. It may be time to learn some new tricks.
  
  The FFmpeg manual mention a technique called [Super Resolution](https://ffmpeg.org/ffmpeg-filters.html#sr), and it comes in two forms:
  
  * `SRCNN`: Super-Resolution Convolutional Neural Network model
  * `ESPCN`: Efficient Sub-Pixel Convolutional Neural Network model
  
  Usually I can find many online sources for figuring out how to do something with FFmpeg. But I can't seem to find any tutorials on how to use this. I don't really understand it. I need to train (how?) a model (from what?), or get a pre-trained model (where?). I'd like to know how to get from 1080p to 2160p using this technique, assuming I know nothing about it.
  
  **Tags:** ffmpeg, scale, quality
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_7 on 2020-09-02. Score: 15 | Accepted answer
  
  I realise this question is pretty old now but it still comes up quite high in search results so I would like to document how I got the "sr" to work (August 2020) in case it can help someone else. Before proceeding it is worth saying that I was not overly impressed by the result. I felt on my test videos that lanczos did an ever so slightly better job. Bear in mind that my test was upscaling from 640x360 flv file to 1920x1080 mp4. I have not tested many other videos and I also have quite a lot to learn about SR like if I feed it different training material will I get a better result.
  
  **What you will need**
  
  * A CUDA enabled GPU with a capability of at least 6.0. I worked all
    the way through this once with a GeForce GT710 (capability of 3.5)
    only to get to the final ffmpeg upscaling for it to tell me I needed
    a capability of 6.0. It did proceed to use the CPU but was very slow.
    I then went through it again with a GeForce GTX 1050TI 4GB card and
    saw a noticable performance gain.
  * At least 70GB free disk space. The training scripts download about 42GB worth of videos and 9GB worth of images. The rest is used up generating the model.
  * Patience. Generating the datasets took a good few hours and training just one model took nearly the whole day.
  * I had a couple of spare computers (one server and one pc) so I was able to build this from scratch and let it run. First I tried it on Ubuntu Server 20.04 and 18.04. Final run through with the new GPU was on Xubuntu 20.04. The only reason I went with Xubuntu in the end was because I was also investigating a new USB capture device for VHS but the instructions below will work with Server.
  
  **Setup**
  
  THIS GUIDE IS OFFERED AS IS WITH NO GUARANTEES IT WILL WORK FOR YOU. PLEASE TAKE CARE AND ONLY FOLLOW IF YOU KNOW WHAT YOU ARE DOING.
  
  My starting point is a fresh install of Xubuntu 20.04. I have a GeForce GTX 1050TI GPU that is not my main GPU and there is no monitor connected to it. I do not think it should matter though if it is your main GPU or if there is a monitor connected.
  
  *FROM <https://www.tensorflow.org/install/gpu>*
  
  Add NVIDIA package repositories
  
  ```
  wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
  sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
  sudo dpkg -i cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
  sudo apt-get update
  wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
  sudo apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
  sudo apt-get update
  
  ```
  
  Install NVIDIA driver
  
  ```
  sudo apt-get install --no-install-recommends nvidia-driver-450
  
  ```
  
  REBOOT, then check that GPUs are visible using the command: `nvidia-smi`. Do not proceed if you do not see a table output. If you have a message saying there is no device then you need to troubleshoot this stage. For example in my first build the BIOS was set to use the onboard graphics card first and changing it to use the offboard card got it working. I didn't have this problem in the last build on a different PC.
  
  Install development and runtime libraries (~4GB). Here we are installing both CUDA 10.0, 10.1. The training scripts and filter are a couple of years old now and we therefore need to accommodate them.
  
  ```
  sudo apt-get install --no-install-recommends \
      cuda-10-0 \
      libcudnn7=7.6.5.32-1+cuda10.0  \
      libcudnn7-dev=7.6.5.32-1+cuda10.0
  
  ```
  
  Install TensorRT. Requires that libcudnn7 is installed above.
  
  ```
  sudo apt-get install --no-install-recommends libnvinfer6=6.0.1-1+cuda10.0 \
      libnvinfer-dev=6.0.1-1+cuda10.0 \
      libnvinfer-plugin6=6.0.1-1+cuda10.0
  
  ```
  
  And now the 10.1 versions
  
  ```
  sudo apt-get install --no-install-recommends \
      cuda-10-1 \
      libcudnn7=7.6.5.32-1+cuda10.1  \
      libcudnn7-dev=7.6.5.32-1+cuda10.1
  
  sudo apt-get install --no-install-recommends libnvinfer6=6.0.1-1+cuda10.1 \
      libnvinfer-dev=6.0.1-1+cuda10.1 \
      libnvinfer-plugin6=6.0.1-1+cuda10.1
  
  ```
  
  *FROM <https://www.tensorflow.org/install/pip#system-install>*
  
  The guide above suggests to install venv here but this was not a concern for me so I left it out. If you are familiar with Python then go ahead and use venv.
  
  ```
  sudo apt install python3-dev python3-pip
  sudo pip3 install --upgrade tensorflow
  
  ```
  
  Add some libraries temporarily to LD\_LIBRARY\_PATH. We will make this permanent a little later.
  
  ```
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/local/cuda-10.0/lib64:/usr/local/cuda-10.1/lib64:/usr/local/cuda-10.2/lib64:/usr/local/cuda/extras/CUPTI/lib64
  
  ```
  
  You will notice I am adding cuda 10.2 here as well but we did not install it. For some reason one of the library files (libcublas) when installing 10.1 is taken from 10.2.
  
  We can verify with the below (check for errors). There is a lot of output, mainly information and you want to check that it has seen your GPU and has successfully opened all the libraries.
  
  ```
  python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
  
  ```
  
  Add LD\_LIBRARY\_PATH to ld.so.conf.d
  
  ```
  sudo vi /etc/ld.so.conf.d/cuda-additional.conf
  
  ```
  
  Add the lines
  
  ```
  /usr/local/cuda/extras/CUPTI/lib64
  /usr/local/cuda-10.2/lib64
  
  ```
  
  The other 2 paths are automatically added for us during install.
  
  *FROM <https://ffmpeg.org/ffmpeg-filters.html#sr-1>*
  Install the TensorFlow for C library. If you follow the link given at the above it shows you Tensorflow version 2.3.0 (or later), however, as mentioned earlier the filter code is old and requires version 1.15.0. So:
  
  ```
  wget https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-1.15.0.tar.gz
  sudo tar -C /usr/local -xzf libtensorflow-gpu-linux-x86_64-1.15.0.tar.gz
  
  ```
  
  *FROM <https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu>*
  Now we fetch and compile ffmpeg. So install some dependencies first:
  
  ```
  sudo apt-get update -qq && sudo apt-get -y install \
    autoconf \
    automake \
    build-essential \
    cmake \
    git-core \
    libass-dev \
    libfreetype6-dev \
    libgnutls28-dev \
    libsdl2-dev \
    libtool \
    libva-dev \
    libvdpau-dev \
    libvorbis-dev \
    libxcb1-dev \
    libxcb-shm0-dev \
    libxcb-xfixes0-dev \
    pkg-config \
    texinfo \
    wget \
    yasm \
    zlib1g-dev
  
  mkdir -p ~/ffmpeg_sources ~/bin
  sudo apt-get install nasm libx264-dev libx265-dev libnuma-dev libvpx-dev libfdk-aac-dev libmp3lame-dev libopus-dev libunistring-dev
  
  cd ~/ffmpeg_sources && \
  wget -O ffmpeg-snapshot.tar.bz2 https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2 && \
  tar xjvf ffmpeg-snapshot.tar.bz2 && \
  cd ffmpeg && \
  PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure \
    --prefix="$HOME/ffmpeg_build" \
    --pkg-config-flags="--static" \
    --extra-cflags="-I$HOME/ffmpeg_build/include" \
    --extra-ldflags="-L$HOME/ffmpeg_build/lib" \
    --extra-libs="-lpthread -lm" \
    --bindir="$HOME/bin" \
    --enable-gpl \
    --enable-gnutls \
    --enable-libass \
    --enable-libfdk-aac \
    --enable-libfreetype \
    --enable-libmp3lame \
    --enable-libopus \
    --enable-libvorbis \
    --enable-libvpx \
    --enable-libx264 \
    --enable-libx265 \
    --enable-libtensorflow \
    --enable-nonfree && \
  PATH="$HOME/bin:$PATH" make && \
  make install && \
  hash -r
  
  ```
  
  Notice that I have left out libaom and added libtensorflow. This has created new binaries under ~/bin and added it to your profile and path so it can be called from anywhere. If you do not want to replace your existing ffmpeg then be sure to change the bindir and paths as required.
  
  Reload our profile
  
  ```
  source ~/.profile
  
  ```
  
  ---
  
  *FROM <https://github.com/XueweiMeng/sr/tree/sr_dnn_native>*
  
  So now we are almost at the stage where we can start to generate and train the models, however, the scripts are old so we need to modify them somewhat. Get them first (you can specify any path to download it to or get it manually with GUI:
  
  ```
  cd ~
  wget https://github.com/XueweiMeng/sr/archive/sr_dnn_native.zip
  unzip sr_dnn_native.zip
  rm sr_dnn_native.zip
  cd sr-sr_dnn_native
  
  ```
  
  Now edit all python files that have "import tensorflow as tf", i.e.
  
  ```
  datasets/prepare_dataset.py
  datasets/prepare_div2k_dataset.py
  evaluate.py
  generate_header_and_model.py
  models/model_vespcn.py
  models/model_espcn.py
  models/image_warp.py
  models/model_srcnn.py
  models/model.py
  models/dataset.py
  models/model_vsrnet.py
  train.py
  
  ```
  
  After the imports in each script add the line:
  
  ```
  tf = tf.compat.v1
  
  ```
  
  For example using datasets/prepare\_dataset.py:
  
  ```
  import os
  import argparse
  from tqdm import tqdm
  import cv2
  import numpy as np
  import json
  import tensorflow as tf
  from PIL import Image
  
  tf = tf.compat.v1
  
  class SceneChangeDetector:
  
  ```
  
  There is one other depreciated function we need to take care of and that is imresize.
  
  Edit the file `datasets/prepare_dataset.py`
  
  Replace line 8: `from scipy.misc import imresize`
  
  With: `from PIL import Image`
  
  Replace line 127: `frame_lr = imresize(frames[k], (lr_h, lr_w), interp='bicubic')`
  
  With: `frame_lr = np.array(Image.fromarray(frames[k]).resize(size=(lr_h, lr_w)))`
  
  We can finally start generating. Install some dependencies.
  
  ```
  sudo pip3 install Pillow tqdm opencv-python
  
  ```
  
  Generate (expect it to take a good few hours and you will likely see a lot of information output and warnings of almost depreciated functions, these are ok):
  
  ```
  sh generate_datasets.sh
  
  ```
  
  This will download 11 videos totalling around 43GB and images totalling around 9GB and can take a few hours to run depending on hardware.
  
  And now for the training (expect these to take about a day each). The models after this will be found in the extracted `~/sr-sr_dnn_native` folder if that is where you saved the Git zip file to:
  
  SRCNN
  
  ```
  sh train_srcnn.sh
  python3 generate_header_and_model.py --model=srcnn --ckpt_path=logdir/srcnn_batch_32_lr_1e-3_decay_adam/train
  
  ```
  
  ESPCN
  
  ```
  sh train_espcn.sh
  python3 generate_header_and_model.py --model=espcn --ckpt_path=logdir/espcn_batch_32_lr_1e-3_decay_adam/train
  
  ```
  
  ---
  
  Once the training is complete you can use the filter. It seems as though ESPCN only upscales by a factor of 2 whereas with SRCNN you can specify 2, 3, or 4. I moved the models generated into the same folder as my videos for ease of use but you can simply point to the models wherever they are.
  
  e.g.
  
  ```
  ffmpeg -i <input_video> -vf sr=dnn_backend=tensorflow:scale_factor=3:model=srcnn.pb -q 15 -preset slow <output_video>
  ffmpeg -i <input_video> -vf sr=dnn_backend=tensorflow:model=espcn.pb -q 15 -preset slow <output_video>
  
  ```
  
  ---
  
  Here is an interesting video of this in use and was probably one of the only places I have seen anyone use this. In it he suggests it can only work on a single plane but I did not find that to be the case. His results actually appear to be quite good:
  
  This is the ffmpeg command he refers to in the video at 14:44:
  
  ```
  ffmpeg -i my_video_540p.mp4 -filter_complex "format=pix_fmts=yuv420p,extractplanes=y+u+v[y][u][v];[y] sr=dnn_backend=tensorflow:scale_factor=2:model=espcn.model [y_scale];[u] scale=960:-2 [u_scale];[v] scale=960:-2 [v_scale];[y_scale][u_scale][v_scale] mergeplanes=0x001020:yuv420p [merged]" -map "[merged]" -c:v libx264 -crf 18 -pix_fmt yuv420p my_video_1080p.mp4
  
  ```
  
  > **Comments:**
  > **username_0**: I am happy with your extensive write-up, and I have accepted your answer after reading this. Also I'm a bit sad that you made a lot of effort yet still observe results similar lanczos at best.  Great first contribution! Thank you.  
  > **username_7**: Thank you. I am not giving up on it just yet because looking at "datasets/prepare_dataset.py" from line 166 and also in "train.py" there are many options that we can play with (passed in as parameters) and I believe with the right training I could achieve better results. I'll post back if I manage to get something amazing.  
  > **username_1**: Using a linux box with an old nvidia driver (418), I decided to install to a docker image `nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04`. This meant skipping everything until libtensorflow installation and installing libtensorflow 1.12.3 rather than 1.15.0 (see https://www.tensorflow.org/install/source#gpu). Additionally, HighVoltageRocknRoll's repo is more up to date than XueweiMeng's, so I used the former, and since libtensorflow was precompiled for higher compute capability (6) than my GPU (5), I compiled it via `tensorflow/tools/ci_build/linux/libtensorflow_gpu.sh` from the Tensorflow repo.  
  > **username_8**: I really wish this were a container image  
  > **username_1**: @username_8 [Mikuláš](https://video.stackexchange.com/users/32410/mikul%c3%a1%c5%a1-bankovi%c4%8d) and I have prepared a Docker image, see [my answer](https://video.stackexchange.com/a/32418/23079).  
  > **username_9**: Is there any google colab available?  
  > **username_5**: Thank you for this. Hopefully as the technology matures the results will get better.  
  
  ---
  
  ### Answer by username_1 on 2020-10-10. Score: 7
  
  Since [Google Summer of Code 2018](https://summerofcode.withgoogle.com/archive/2018/projects/5661133578960896/), FFMpeg has supported [the `sr` filter](https://ffmpeg.org/ffmpeg-filters.html#sr-1) for applying super-resolution methods based on convolutional neural networks. However, as you have discovered, few super-resolution tutorials exist, and compiling FFMpeg with proper libraries and preparing models for super-resolution requires expert knowledge.
  
  To make super-resolution in FFMpeg easier, [Mikuláš](https://video.stackexchange.com/users/32410/mikul%c3%a1%c5%a1-bankovi%c4%8d), [Dirk](https://github.com/dlueth) and I have taken the excellent answer of [Mark](https://video.stackexchange.com/users/32099/mark), and we used it to prepare [a Docker image with FFMpeg and Libtensorflow](https://hub.docker.com/r/miratmu/ffmpeg-tensorflow). We also wrote a tutorial that explains step-by-step how to use the Docker image for super-resolution in FFMpeg: <https://github.com/MIR-MU/ffmpeg-tensorflow#ffmpeg-with-libtensorflow>.
  
  After installation, super-resolution in FFMpeg becomes as easy as this:
  
  ```
  $ wget https://media.xiph.org/video/derf/y4m/flower_cif.y4m
  $ alias ffmpeg-tensorflow='docker run --rm --gpus all -u $(id -u):$(id -g) -v "$PWD":/data -w /data -it miratmu/ffmpeg-tensorflow'
  $ ffmpeg-tensorflow -i flower_cif.y4m -filter_complex '
  >   [0:v] format=pix_fmts=yuv420p, extractplanes=y+u+v [y][u][v];
  >   [y] sr=dnn_backend=tensorflow:scale_factor=2:model=espcn.pb [y_scaled];
  >   [u] scale=iw*2:ih*2 [u_scaled];
  >   [v] scale=iw*2:ih*2 [v_scaled];
  >   [y_scaled][u_scaled][v_scaled] mergeplanes=0x001020:yuv420p [merged]
  > ' -map [merged] -sws_flags lanczos \
  > -c:v libx264 -crf 17 -preset ultrafast -tune film \
  > -c:a copy \
  > -y flower_cif_2x.mp4
  
  ```
  
  Compare upscaling using Lanczos (left) with the ESPCN super-resolution model (right):
  
  [![Comparison of Lanczos and ESPCN](https://i.sstatic.net/DzYz3.jpg)](https://i.sstatic.net/DzYz3.jpg)
  
  
  ---
  
  ### Answer by username_2 on 2021-04-05. Score: 4
  
  Although Mark has provided great answer, you don't need to train models from scratch, all Super-Resolution models that ffmpeg supports have pre-trained checkpoints:
  <https://github.com/XueweiMeng/sr/tree/sr_dnn_native>
  all you need to do is to convert it to frozen graph (.pb) file and then use it in ffmpeg.
  
  > **Comments:**
  > **username_0**: Upvoted. Could you elaborate the _"all you need to do"_ part so that from reading the answer alone, the reader will know what to do?  
  > **username_2**: Sure, no problem. 1. Download repo with models https://github.com/XueweiMeng/sr/tree/sr_dnn_native 2. Install all requirements using `pip install`, for me it was only tensorflow and numpy packages. 3. Run script to generate model you need: `python3 generate_header_and_model.py --model=espcn --ckpt_path=checkpoints/espcn/` 4. 2 model files will be generated - `espcn.mode`l and `espcn.pb` (you can check that model works by evaluating .pb model in tensorflow) 5. Use one of these models in ffmpeg with sr filter (or dnn_processing filter), as stated in documentation.  
  > **username_2**: Note: you might need to install TensorFlow C library and configure ffmpeg if you want to use tensorflow dnn_backend.  
  > **username_1**: @username_2 Very interesting! We provide a Docker image with precompiled tensorflow-capable FFMpeg and `.pb` files in https://github.com/MIR-MU/ffmpeg-tensorflow. However, I haven't found much documentation about these `.model` and `.pb` files. Are there sources that you would recommend for reading up on the common formats for storing computational graphs?  
  > **username_2**: @username_1 Well, it's just a specific format for saved models: https://stackoverflow.com/questions/51278213/what-is-the-use-of-a-pb-file-in-tensorflow-and-how-does-it-work  
  > **username_3**: These guys have ~70 Mb prtrained tflite model , they claim it is optimized to work better with different resolutions. They tiled GAN I guess.  Once I tiled ESR-GAN it doesnt show any difference in speed or quality however it introduces some blurred lines at tiling border. So be careful https://www.collabora.com/news-and-blog/blog/2020/09/21/open-source-meets-super-resolution-part-1/  
  
  ---
  
  ### Answer by username_4 on 2020-01-19. Score: 3
  
  If all you need to do is superscale an image, going through the trouble of training neural networks is re-inventing the wheel. By all means, if you're studying computer science and are interested in AI/ML, I'd encourage you to look into it, but to just superscale an image, you don't need to train a neural network. There are tools available.
  
  In DaVinci Resolve, for instance, all you need to do is right-click on a clip in the media pool, select "clip attributes...," then at the bottom of the "video" tab, change "Super Scale: None" to 2x, 3x, or 4x. I'm not sure if this feature is available in the free version, but I wouldn't be surprised if it is, because all they've done is integrated the open source ML bits of FFMPEG for you, trained the models for you, and bundled it as a feature. I'm sure Adobe has an equivalent feature, though I haven't used it.
  
  > **Comments:**
  > **username_0**: I don't mind using a pre-trained model, but I would like to know how to do this using `ffmpeg` exclusively so I can incorporate it in a pipeline (e.g. `kdenlive`). No commercial software.  
  > **username_5**: According to [this](https://www.toolfarm.com/tutorial/in-depth-davinci-resolve-studio-vs-the-free-version/) it's not in the free version (which *is* DaVinci Resolve, as opposed to DaVinci Resolve Studio). Given that, and given DaVince Resolve Studio costs $300 dollars, and *especially* given what you claim about them basing their SR on FFmpeg (I would definitely be interested in a source for that claim), FFmpeg is currently the best free tool for the purpose out there.  
  > **username_4**: @username_5 That’s about R16.  R17 has been out for a while.  I don’t know if superscale is one of the features that got “unlocked” this time around, but BMD always promotes some previously studio-only features to the free version with every update cycle, so there’s no harm in trying it out.  Plus, R18 shouldn’t be too far away.  
  > **username_6**: @username_4 R19 is now in beta, and Super Scale remains exclusive to (non-free) Resolve Studio.  
  
  ---
  
  