Hi! I’ve been testing resource usage of my pytorch model with and without cudnn. It seems when I have torch.backends.cudnn.enabled=False, training the network is 2 to 3 times faster and uses less memory (bigger batch size). I tested the same network on Nvidia 3090, 2080 ti and 1080 ti. The result is pretty consistent.
I thought cudnn was supposed to improve the performance of training. Is there anything I can do to investigate this further?
I use conda to manage my environment. Here are the specs:
Ubuntu 18.04
CUDA 11.1
Nvidia driver 455.32
Pytorch 1.8.1
Cudnn 8.0.5_0