nvidia-smi shows all allocated memory, which is the one used by PyTorch, the cache, as well as the CUDA context etc.
You could check the used memory via: print(torch.cuda.memory_allocated()) or print(torch.cuda.memory_summary()).