PyTorch uses a custom caching memory allocator, which will try to reuse the device memory.
Thus nvidia-smi shows you the overall memory usage including the CUDA context, the allocated and the cached memory (also from other processes), which might show the increased memory usage after a couple of steps for small model.