As much as it sounds crazy, I have just rebooted my computer and now my program using pytorch does not really utilize the GPU. Before the reboot, it was multiple times faster and it did utilize the GPU so much so that I had to set my fans to something 80% and now I see that it only uses the GPU for a brief moment while doing validation but not during training (literally at zero to 1% (browser) during training). I did check the gpu and it is still seen and used by the code. All I did was a simple reboot really because some GUI was stuck. I have not run updates or anything. This is a curious case really. I can trace the usage using the nvidia GUI and the nvidia-smi dmon / nvidia-smi. I have NO idea why this happens.
and I did reboot again but it is still acting the same. And this is in my code (added second line just to make double sure)
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
cuda.set_device(0)

Why me