NVENC_HEVC Encoding making my files larger I got a new Nvidia GPU to speed up my video encoding. I am working with x264 files to x265 and wanted to go faster. Using the library libx265 I am getting about 85fps when working on a file that is 720p, but if I use the nvenc_hevc I get around 600-700fps
My input file is 269M and when I use the nvenc_hevc it grows to 394M. Doing the exact same command with libx265 gives me a file size of only 68M. Obviously I would like to keep the low file size, but take advantage of the GPU to process it 15X faster.
Anyone else have any experience with this video codec? My understanding is that the difference between libx265 and nvenc_hvec is not the algorithm that is used to do the compression, but that it utilizes the GPU to accelerate processing. With that understanding the resulting file sizes would be identical regardless of which I used, just the nvenc_hvec would be faster. Is that a faulty assumption?
My command is: ffmpeg -i infile.avi -c:v nvenc_hevc -rc vbr_2pass -rc-lookahead 20 -gpu any out7.mp4
vs
ffmpeg -i infile.avi -c:v libx265 -rc vbr_2pass -rc-lookahead 20 -gpu any out7.mp4
When encoding I seem to only be using a small percentage of the GPU despite the huge performance increase:
nvidia-smi -l
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     21954    C   ffmpeg                                          93MiB |
+-----------------------------------------------------------------------------+
Tue Nov  8 20:41:45 2016       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.48                 Driver Version: 367.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     Off  | 0000:42:00.0     Off |                  N/A |
|  1%   63C    P2    37W / 160W |     95MiB /  4037MiB |     11%      Default |
+-------------------------------+----------------------+----------------------+
Just because you can get a hamburger at different restaurants doesn't mean they'll taste and look the same.

Apples and oranges

x265 and NVENC are two different encoder implementations that can output the same format, HEVC. What settings, options, and algorithms they use is up to the authors as long as the output conforms to the specs.

Choose the right tool for the job

These encoders are generally for different use cases. x265 may be slower, but it provides more quality per bit. NVENC is more suitable when speed is more important than quality or encoding efficiency, or when the CPU is not available or not up to the task, such as on a mobile platform where using the CPU would possibly consume more battery power and take a long time.

Option mixup

Also, you're mixing up options: -rc vbr_2pass -rc-lookahead 20 -gpu any are private options for nvenc_hevc, but you attempted to apply them to libx265, so they were ignored. There are two types of options: general and private. The private options are specific to each individual encoder. You can list private options such as with ffmpeg -h encoder=libx265.

You'll have to experiment with the encoding presets (see -preset option for both encoders) and rate control methods to get an output that is acceptable for you.

Waste of time?

Lastly, unless your input H.264 files were encoded very inefficiently (or are lossless which is doubtful), or if storage space is of utmost importance, then re-encoding from H.264 to H.265/HEVC is a waste of time and effort.