There are a few things to keep in mind

PyTorch will keep GPU memory around after the tensors have been deallocated (the CUDA memory cache). nvidia-smi would see that as still in use. You can check allocated / cached memory and release cached memory using the functions in torch.cuda.
You would want to wrap things in with torch.no_grad() to instruct PyTorch to not keep the information around for backward.

While bugs in PyTorch, including memory leaks, can occur, it would seem that it is unlikely that benign use of Conv2D has it because it would affect very many people.
Best regards
Thomas