I am having this weird issue where in between an epoch training , the model is not making any progress and hangs in the middle, even the time running seems to halt. My GPU temperature subsides but the nvidia-smi output still shows the model is still there in the GPU (as the memory which would be 3GB ) reamins the same (a resnet32 model)
And surprisingly it starts to resume training when I press any KEY.
have you guys faced any issue like this?
I have tried a lot with changing the num_workers from the torch.utils.data.DataLoader and it doesn’t seem to work anything