I don’t know how you’ve installed PyTorch but given that you mention CUDA 11.4, I assume you are using source builds.
In this case, the different CUDA versions as well as cuDNN (which versions did you build with?) would most likely have a different memory footprint. If you don’t want to load e.g. the cuDNN kernels and are dynamically linking to it during the build, disable cuDNN via torch.backends.cudnn.enabled=False and check the memory usage again. Also, different GPU generations would load a different amount of kernels in libs such as cuDNN.
It’s also important what exactly you are measuring since PyTorch uses a caching allocator and can thus reuse memory so you should check the allocated and reserved memory via e.g. torch.cuda.memory_summary().