Hi,
This is most likely only the memory used by the cuda driver 
import torch
# No memory/process on the gpu + 180MB of cpu memory
input("Press enter to continue")

# Store a cpu Tensor
a = torch.rand(1)
# Still nothing on the GPU, still 180MB on the cpu
input("Press enter to continue")

# Store a Tensor with one element on the gpu (the Tensor takes 4bytes)
a = torch.rand(1, device="cuda")
# Cuda driver is not initialized and uses 543MB on my gpu and almost 2GB - 180MB on the cpu
input("Press enter to continue")