Hi,
So if you use different GPUs, I could think of few reasons:

The cuda driver behaves differently
Different architecture lead to different size of code loaded on the device by the driver
The algorithms chosen to do the compute by cudnn are different and use different amount of memory.

You can check how much memory is used when only doing torch.rand(1).cuda(). That will tell you how much memory is used by only the driver + code.
You can use torch.cuda.memory_cached doc 1 to know how much memory pytorch hold onto for Tensor allocation.