I compared several possible combinations of cuda, pytorch and gpus on my 3D ResNet model.




Cuda
pytorch
GPU
Etc
Top1
Time(s)




9.1
1.1
V100

68.7
215732


9.2
1.4
P40

68.1
305547


9.2
1.4
V100

61.7
155272


9.2
1.4
V100
Custom Conv
68.2
162994


9.2
1.4
V100
No cudnn
62.9
160497


10.2
1.5
V100

63.0
162676


10.2
1.5
V100
Custom Conv
67.6
186566



When I use the custom Conv3d above, the top1 accuracies are increased compared to nn.Conv3d.
However, it is still 0.5~1 % lower than pytoch1.1. I’m not sure it is caused by random initialization or some other issues. The wall-time decreased compared to cuda 9.1 (V100 machines are not exactly same but each has same number of CPU cores and GPUs).
When cudnn is turned off, the top1 accuracy is still lower. In order to turn off the cudnn, I make cudnn.benchmark=False. I’m not sure it is right way.