ptrblck:

Note that I’ve replaced the .grad attributes with None to free these tensors as well instead of zeroing them out.


Excellent, thank you so much. I had not thought yet about assigning the .grad to None after the backward. I totally missed that!
Currently, I was only deleting tensors whenever they were not needed anymore, such as in X = torch.fft.fft2d(x); del x;. Deleting the model weight’s .grad will definitely improve things!
Thanks a lot!