Are you using CUDA for your training?
If so, did you set torch.backends.cudnn.deterministic = True to get deterministic behavior?
It might help to debug this issue.
Also, would it be possible to post a small code snippet reproducing this error?
I’ve never seen this behavior before.