Hi all,
We recently got new server nodes with QUADRO RTX6000 GPUs and I have some problems of getting pytorch to it’s top speed on them. Weirdly the from source installation is partially slower then the conda installations. I benchmarked one forward + backward pass of a few stacks of 2d and 3d convolutions. For the 2d convolutions I could see that for both with and without the use of amp the performance was twice as slow using from source installation (CUDA 11.1, cudnn 8.0.4, pytorch 1.8) compared to the conda installation (cudatoolkit=11.0, pytorch=1.7).
I could compare the running times against my workstation that has a GEFORCE RTX 2080 ti. The 2d convolutions of the conda installation of the quadro card are a bit faster then the from source installation using the geforce card. This is to be expected as they that the same architecture and the quardo has slightly more cuda kernels.
However I mostly use 3d convolutions and amp in my research. Here the from source installation was the fastest on both cards, but the quadro is 2.5 times slower then the geforce card.
When I compare conda installations on both cards I can see that the quadro is constantly a bit faster then the geforce card, just the from source installation on the quadro is why to slow.
Does anyone have an idea what the reason behind this is? Is it possible that something is wrong with the cuda compiler? We had problems with compiling cuda code on an other end. As this is a departmental machine we’re having a range of different cuda modules.
Thanks for the help!