Hello all,
I have a retinanet model I am trying to train on a custom dataset which consists of around 2500 images in total. The problem is that it takes approximately 30 minutes per epoch(resnet 101 as a backbone). Is it supposed to be this slow?
I have PyTorch 1.7, Cuda 11.1 installed on the system with cuDNN8.0.5.
System specs - i7 9750 with RTX 2070. All hardware benchmarks run fine. Is the slow training due to cuda and cudnn mismatch and will it go away if I build from source?