Besides the data and model parameters, the CUDA context will use some memory as well as the intermediate activations, which are needed to calculate the backward pass.
Also note that PyTorch uses a caching allocator, which will reuse the memory.
nvidia-smi will thus show the complete memory usage, while torch.cuda.memory_allocated() will give you the allocated memory only.