Pytorch uses some memory for meta-data and CUDA context management, and it’s not exposed to python API yet.
You can try:
a = torch.Tensor(1).cuda()

And then nvidia-smi to see how much extra memory pytorch requires.