The nvidia-smi number includes space the allocated by the CUDA driver when it loads PyTorch. This can be quite large because PyTorch includes many CUDA kernels. On a P100, I’ve seen an overhead of ~487 MB. On M40, I’ve seen ~303 MB.
This memory isn’t reported by the torch.cuda.xxx_memory functions because it’s not allocated by the program and there isn’t a good way to measure it outside of looking at nvidia-smi.