Hi,
When I do an experiment, I always save the code and the config used for that experiment, so that I can easily reproduce an old experiment. Of course the computer is always only working on this experiment.
Recently, I’ve updated my NVIDIA drivers, as well as my cuda and pytorch versions.
Here’s the thing:
OLDER VERSION

Driver: Nvidia 435
Cuda: 10.1
Pytorch: 1.5.1
Maximum batch size (before OOM): 235
Time_per_iteration: .45s

AFTER UPDATE

Driver: Nvidia 460
Cuda: 11.2
Pytorch: 1.7.1
Maximum batch size (before OOM): 216
Time_per_iteration: .50s

Results after update: 8% smaller batch size, 11% slower training
I’ve tried changing cuda toolkit version, I’ve tried going back to Pytorch 1.5.1. Same results.
I’ve tried going back to Nvidia driver 435, it doesn’t work and ultimately I have the error: “nvidia-driver-435: Depends: nvidia-driver-455 …”. So it always installs newer versions of the driver and I can’t benchmark again with my older driver.
Do any of you have seen a similar behaviour? 11% Slower training really is a lot. Is Nvidia purposefully slowing down their newer drivers?
Should I try using a docker container? (I guess it won’t work either if nobody can download the older drivers without downloading the newer ones, or if they’ve changed the older drivers…)
Thanks if you have any recommendation for this situation!