This is expected, since PyTorch uses a caching allocator to reuse the memory instead of reallocating it (which would be slow).
If you want to reset the memory, you would have to delete all tensors, which might still be alive from the validation run, and call torch.cuda.empty_cache(). Note that this is only necessary, if you want other applications to use the memory.