Oh I see, sorry if I misunderstood. I don’t know about the exact automatisms, but I read somewhere that PyTorch uses as much memory as possible for computational throughput efficiency … So there might be some caching. There’s definitely some caching by cuDNN though, I noticed that when benchmarking. You can try setting
torch.backends.cudnn.benchmark = True

and see if it makes some difference in your comparison.