Note that PyTorch tries to reuse the cached memory in order to avoid cudaMalloc calls.
If you are measuring the memory usage via nvidia-smi only, you’ll see the overall used memory (allocated + cached + CUDA context + other processes).
You could check the allocated memory via torch.cuda.memory_allocated(), which would most likely go down during evaluation, if you’ve properly freed the training data.