Answers.Body: <p>Problem was resolved by replace </p>
  
  <pre><code>scale_npp=-2:360
  </code></pre>
  
  <p>to </p>
  
  <pre><code>scale_npp=-2:360,hwdownload,format=nv12
  </code></pre>
  
  <p>now decoders fully loaded:</p>
  
  <pre><code>nvidia-smi dmon -c 1
  # gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
  # Idx     W     C     C     %     %     %     %   MHz   MHz
      0   101    50     -    27     8    15   100  3304  1189
      1   105    54     -    25     8    12   100  3304  1201
      2   104    52     -    48    10    19   100  3304  1316
      3    97    50     -    23     9     8   100  3304  1189
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-31T07:42:42.927Z
Answers.LastActivityDate: 2018-12-31T07:42:42.927Z
Answers.UserDisplayName: Hett
Answers.Body: <p>NVidia boards has a limit number of concurrent encoding tasks, according number of encode engine chips on board. Look at <a href="https://developer.nvidia.com/video-encode-decode-gpu-support-matrix" rel="nofollow noreferrer">specifications</a> of boards you are using.<p>
  Even if you have a board with unrestricted number of sessions to it, you will have a speed degradation, because of some task pipelining on host side.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-27T08:23:43.210Z
Answers.Text: you are not right! I use full hardware transcode (learn about hwaccel) https://trac.ffmpeg.org/wiki/HWAccelIntro
Answers.UserDisplayName: Hett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-27T08:24:30.020Z
Answers.Text: processes outside of limits lead to an error
Answers.UserDisplayName: Hett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-27T06:43:09.210Z
Answers.LastActivityDate: 2018-12-27T06:43:09.210Z
Answers.UserDisplayName: PRu
Body: <p>Has 4 nvidia devices:</p>
  
  <pre><code>nvidia-smi topo --matrix
          GPU0    GPU1    GPU2    GPU3    CPU Affinity
  GPU0     X      PHB     SYS     SYS     0-5,12-17
  GPU1    PHB      X      SYS     SYS     0-5,12-17
  GPU2    SYS     SYS      X      PHB     6-11,18-23
  GPU3    SYS     SYS     PHB      X      6-11,18-23
  
  Legend:
  
    X    = Self
    SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
    NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
    PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
    PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
    PIX  = Connection traversing a single PCIe switch
    NV#  = Connection traversing a bonded set of # NVLinks
  </code></pre>
  
  <p>We want to re-code h264 videos to lower resolution. 
  When we run 2-3 processes per 0 and 1 devices:</p>
  
  <pre><code>ffmpeg -v error -y -hwaccel cuvid -hwaccel_device 0 -c:v:0 h264_cuvid -gpu 0 -i /mnt/worker/6797/be2c96e8f0fd7/objects/133607/fb3/8279fb61a4d0bf66c3f871bdb30e5fb3/1530758904.34570.data -map_metadata -1 -map_chapters -1 -map 0:v:0 -map 0:a:0 -c:v:0 h264_nvenc -gpu 0 -b:v:0 650000 -filter:v:0 scale_npp=-2:360 -c:a:0 aac -gpu 0 -b:a:0 96000 -ac:a:0 2 -max_muxing_queue_size 9999 -movflags +faststart -f mp4 /home/uploads/tmp/out.mp4
  </code></pre>
  
  <p>all works fine</p>
  
  <pre><code>nvidia-smi dmon -c 1
  # gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
  # Idx     W     C     C     %     %     %     %   MHz   MHz
      0   102    48     -    20     9    14   100  3304  1189
      1   105    54     -    25     9    17    99  3304  1201
      2    15    27     -     0     0     0     0   405   135
      3    15    27     -     0     0     0     0   405   135
  </code></pre>
  
  <p>but when works 2-3 processes per each device, then performance gets low. 
  Summary compared to 2 devices:</p>
  
  <pre><code>nvidia-smi dmon
  # gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
  # Idx     W     C     C     %     %     %     %   MHz   MHz
      0    89    45     -     8     3     5    37  3304  1189
      1    92    50     -     9     3     6    30  3304  1201
      2    76    42     -     9     3     6    29  3304  1201
      3    87    44     -     9     3     6    33  3304  1189
  </code></pre>
  
  <p>htop,top,iftop,atop show nothing overloaded</p>
  
  <p>Sometimes performance rises higher, or drops below (i`m suspect it depending on input files)</p>
  
  <p>How to I can utilize all decoders ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-26T21:45:12.603Z
Comments.Text: Realistically - get a faster machine. That's pretty much the fix.
Comments.UserDisplayName: Dr Mayhem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-27T05:05:22.817Z
Comments.Text: @DrMayhem what means faster? We have 2 CPU - Xeon(R) CPU E5-2620 v3. 20% loaded.
Comments.UserDisplayName: Hett
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-27T09:55:05.617Z
Comments.Text: Faster/more CPUs. Faster pipeline. Greater concurrency.
Comments.UserDisplayName: Dr Mayhem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-27T17:37:31.640Z
Comments.Text: @dr-mayhem cpu is not loaded more than 15% when hwaccel is used
Comments.UserDisplayName: Hett
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-27T20:49:14.923Z
Comments.Text: Hett - it's everything. cpu was one of many things worth looking at.
Comments.UserDisplayName: Dr Mayhem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-29T11:34:08.313Z
Comments.Text: @DrMayhem I`m wrote: "htop,top,iftop,atop show nothing overloaded"
Comments.UserDisplayName: Hett
Tags: <ffmpeg><cuvid>
Title: cuvid - slow performance when used 4 devices
UserDisplayName: Hett
ThreadText: # cuvid - slow performance when used 4 devices
  
  (Asked by: username_0 on 2018-12-26)
  
  Has 4 nvidia devices:
  
  ```
  nvidia-smi topo --matrix
          GPU0    GPU1    GPU2    GPU3    CPU Affinity
  GPU0     X      PHB     SYS     SYS     0-5,12-17
  GPU1    PHB      X      SYS     SYS     0-5,12-17
  GPU2    SYS     SYS      X      PHB     6-11,18-23
  GPU3    SYS     SYS     PHB      X      6-11,18-23
  
  Legend:
  
    X    = Self
    SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
    NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
    PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
    PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
    PIX  = Connection traversing a single PCIe switch
    NV#  = Connection traversing a bonded set of # NVLinks
  
  ```
  
  We want to re-code h264 videos to lower resolution.
  When we run 2-3 processes per 0 and 1 devices:
  
  ```
  ffmpeg -v error -y -hwaccel cuvid -hwaccel_device 0 -c:v:0 h264_cuvid -gpu 0 -i /mnt/worker/6797/be2c96e8f0fd7/objects/133607/fb3/8279fb61a4d0bf66c3f871bdb30e5fb3/1530758904.34570.data -map_metadata -1 -map_chapters -1 -map 0:v:0 -map 0:a:0 -c:v:0 h264_nvenc -gpu 0 -b:v:0 650000 -filter:v:0 scale_npp=-2:360 -c:a:0 aac -gpu 0 -b:a:0 96000 -ac:a:0 2 -max_muxing_queue_size 9999 -movflags +faststart -f mp4 /home/uploads/tmp/out.mp4
  
  ```
  
  all works fine
  
  ```
  nvidia-smi dmon -c 1
  # gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
  # Idx     W     C     C     %     %     %     %   MHz   MHz
      0   102    48     -    20     9    14   100  3304  1189
      1   105    54     -    25     9    17    99  3304  1201
      2    15    27     -     0     0     0     0   405   135
      3    15    27     -     0     0     0     0   405   135
  
  ```
  
  but when works 2-3 processes per each device, then performance gets low.
  Summary compared to 2 devices:
  
  ```
  nvidia-smi dmon
  # gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
  # Idx     W     C     C     %     %     %     %   MHz   MHz
      0    89    45     -     8     3     5    37  3304  1189
      1    92    50     -     9     3     6    30  3304  1201
      2    76    42     -     9     3     6    29  3304  1201
      3    87    44     -     9     3     6    33  3304  1189
  
  ```
  
  htop,top,iftop,atop show nothing overloaded
  
  Sometimes performance rises higher, or drops below (i`m suspect it depending on input files)
  
  How to I can utilize all decoders ?
  
  **Tags:** ffmpeg, cuvid
  
  > **Comments:**
  > **username_1**: Realistically - get a faster machine. That's pretty much the fix.  
  > **username_0**: @username_1 what means faster? We have 2 CPU - Xeon(R) CPU E5-2620 v3. 20% loaded.  
  > **username_1**: Faster/more CPUs. Faster pipeline. Greater concurrency.  
  > **username_0**: @username_1 cpu is not loaded more than 15% when hwaccel is used  
  > **username_1**: username_0 - it's everything. cpu was one of many things worth looking at.  
  > **username_0**: @username_1 I`m wrote: "htop,top,iftop,atop show nothing overloaded"  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2018-12-31. Score: 2 | Accepted answer
  
  Problem was resolved by replace
  
  ```
  scale_npp=-2:360
  
  ```
  
  to
  
  ```
  scale_npp=-2:360,hwdownload,format=nv12
  
  ```
  
  now decoders fully loaded:
  
  ```
  nvidia-smi dmon -c 1
  # gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
  # Idx     W     C     C     %     %     %     %   MHz   MHz
      0   101    50     -    27     8    15   100  3304  1189
      1   105    54     -    25     8    12   100  3304  1201
      2   104    52     -    48    10    19   100  3304  1316
      3    97    50     -    23     9     8   100  3304  1189
  
  ```
  
  
  ---
  
  ### Answer by username_2 on 2018-12-27. Score: 1
  
  NVidia boards has a limit number of concurrent encoding tasks, according number of encode engine chips on board. Look at [specifications](https://developer.nvidia.com/video-encode-decode-gpu-support-matrix) of boards you are using.
  
  Even if you have a board with unrestricted number of sessions to it, you will have a speed degradation, because of some task pipelining on host side.
  
  > **Comments:**
  > **username_0**: you are not right! I use full hardware transcode (learn about hwaccel) https://trac.ffmpeg.org/wiki/HWAccelIntro  
  > **username_0**: processes outside of limits lead to an error  
  
  ---
  
  