That is correct. I have no idea why a call to torch.cuda.memory_summary stops the out of memory errors, but I am glad it does. Basically when I see a CUDA out of memory error I call memory_summary 25 times in a row then try one more time to feed the example to the network. Most of the time this fixes the out of memory error.