torch.cuda.max_memory_reserved() (don’t know if that function or any similar)
Shows the peak, not the real memory usage. Memory is reused on demand. When the allocator does not longer need the space it’s marked as available but not “freed” so that that memory slot can be overwritten.
It’s like hard disks, when you delete something you are not physically deleting it but marking that space as available.
That’s why there is no pairing between nvidia-smi and memory