Hi, I am currently working on a CT scan classification dataset with 5-fold cross-validation. The images go through an online preprocessing stage before entering the backbone to reduce their size, so the network fits in the GPU.  When I train the network from scratch, there are no issues. However, when I use pre-trained weights to make finetuning in the backbone, I have an error in the second fold. I am currently using PyTorch 1.6 with amp. The first fold trains without any problem, but the second one cannot. I have tried changing the fold that trains first, but the second one, no matter what fold it is, shows this error:
Traceback (most recent call last):
File “main.py”, line 399, in 
main()
File “main.py”, line 190, in main
bce, scaler)
File “main.py”, line 301, in train
scaler.scale(loss).backward()
File “/home/mfroa/anaconda3/envs/lung/lib/python3.7/site-packages/torch/tensor.py”, line 184, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/home/mfroa/anaconda3/envs/lung/lib/python3.7/site-packages/torch/autograd/init.py”, line 115, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function ‘CudnnBatchNormBackward’ returned nan values in its 0th output.