I am facing some similar issue where GPU memory consumed by the network on one platform one is 15 GB and platform 2 is 11 GB.
Platform1: NVIDIA Tesla P100 GPU 16GB, Cuda 9.1.85, Pytorch 0.4.0
Platform2: GTX 1080 GPU 12GB, Cuda 8.0.61 , Pytorch 0.4.0
Both print cudnn version to be 7102 inside python by torch,backends.cuda.version()
Strangely, when i turn torch.backends.cudnn.enabled=False in platform1 my gpu consumption becomes 11 GB. But i think gpu consumption should be lesser using cudnn as suggested in above answers. Can anyone help what could be the problem?