PyTorch uses a caching memory allocator, which means that it might oversubscribe memory even if it is not using it and cache blocks. So in that case, nvidia-smi wont give you the exact memory used by pytorch. In out-of-memory situations, pytorch automatically frees blocks that are cached.
This might be one reason to explain the memory difference.