The initial increase of the memory usage might be just temporary and afterwards is cached to avoid memory allocations.
Could you check the allocated memory via torch.cuda.memory_allocated(), please?
This would give you an idea how much memory is really used to store all tensors and parameters.
Also, are you using cudnn.benchmark? If so, could you disable it for debugging purposes and also create a run with the native CUDA implementations by disabling cudnn completely via torch.backends.cudnn.enabled = False?
It would be interesting to see, how much memory is used in which part of your code.
PS: Are you using an older PyTorch version, as you are using Variables, which were deprecated in 0.4?
If so, could you update to the latest stable version (1.5)?