Check, if you are accidentally storing the computation graph, e.g. by accumulating the loss without detaching or storing it in a list.
Could you try to create a minimal code snippet to reproduce this issue?