The code looks generally alright, at least I cannot find any obvious errors.
Could you post your setup (hyperparameters of the optimizer, learning rate scheduler etc.) so that we could try to reproduce this issue with a dummy model?