Thanks for the formatting tip, I will start doing that from now on!
As for the cuda sync issue, I’ve added the torch.cuda.synchronize(); command before each memory check print command and the memory usage output is the same as before.