I guess the issue might be related to fp16. I’m using torch.cuda.amp fp16 to accelerate the training process. After I switch to fp32, the training is going on smoothly.
In addition, I’m using 1cycle lr scheduler and encounter the issue, but the code works well with default step-wise lr scheduler.
But I still don’t understand, is fp16 incompatible with 1cycle lr scheduler?