I think gradient accumulation is the source of this problem. Seeing that the training worked for some epochs and you got the error after that.
Refer to this pytorch documentation 20 that can help if this is the case.