haowxu:

The memory reduces especially for the first batch after the cudnn benchmarking is commented. But the running time takes 30% longer.


That’s expected, since cudnn benchmarking will try to find the fastest algorithm for your workload.



 haowxu:

For example, if 15 GB out of 16GB is occupied by the model training itself, will this get an OOM error?


This shouldn’t be the case and cudnn should discard all algorithms which could cause an OOM error. Have you seen an OOM using cudnn.benchmark?



 haowxu:

I just wonder is there a way to so something like making cudnn benchmarking try these algorithms before the training to reduce the peak memory allocated?


Benchmarking will run for the first iteration and cache the fastest algorithms for the current setup.
If you don’t want to trigger it in the first training iteration, you could pass a dummy tensor with the same shape as your training samples before entering the training loop.
Note that each different input shape will trigger the benchmarking again, so if you are dealing with a lot of different shapes, your code might run faster, if you disable it.