How are you measuring the memory usage? Are you using nvidia-smi or torch.cuda.memory_allocated() and other PyTorch methods?
Also, are you using torch.backends.cudnn.benchmark=True and what kind of GPUs are you using in those machines?