If you delete the model and no other references are pointing to it, it will be freed.
Note that PyTorch uses a custom memory caching allocator, to avoid multiple memory allocations.
nvidia-smi will thus still show the GPU memory as used by PyTorch.
I would recommend to .clone() the parameters just to make sure these parameters are not referencing each other.