Ah yes, that makes sense. Is there a way for the activation tensors to be overwritten or utilized ‘inplace’ (as I write this I am realizing how implausible this is). While training on the 512x512, the peak memory usage is ~6.1 GB.