Hello, I changed my device from 2080Ti to 3090 recently, and I created a new conda enviroment to install pytorch 1.7.0. But I found that pytorch1.7.0 took more time training especially when I enabled nvidia-apex or torch.cuda.amp, it was even up to 6x time slower!
(2080Ti pytorch1.3.1 with nvidia-apex: 1.6it/s
3090 pytorch1.7.0 with torch.cuda.amp: 5.35s/it)
It was so strange that I had to try a number of experiments to confirm where was the key point.
(I used my old 2080Ti by the way because 3090 can’t run pytorch1.3.1)
first I created two new conda enviroments pyt1 and pyt2.
“conda create -n pyt1 python=3.6.9”
“conda create -n pyt2 python=3.6.9”
then in pyt1 I installed pytorch1.7.0 by “conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch” and “conda install tqdm”
it installed cudatoolkit 11.0.221 and pytorch1.7.0
after that I chose the official pytorch example mnist code in githubexamples 1
the only code I added is tqdm to show the progress bar.
and ran it:"(pyt1)XXXXX:~$python main.py"
the result showed that:

then in pyt2 I installed pytorch1.3.1 by “conda install pytorch torchvision” and “conda install tqdm”.
It installed default pytorch1.3.1 and cudatoolkit10.0.130
this time the result showed that:

It was clear that pytorch1.7.0 was about 1/3 slower than pytorch1.3.1
my PC enviroment:
nvidia driver 450. 66
CUDA 11.0
but conda enviroment will use its own virtual cudatoolkit right? I don’t know if the external enviroment will affect the conda internal enviroment.