ConnollyLeon:

However, the parameters memory didn’t release correctly.


I wouldn’t say that. You do remove the parameter successfully. The problem is that other part of the code are also using this Tensor and so the Tensor cannot be freed. This is actually expected.
If you want to play tricks with what the autograd saves for backward, you have hooks for that: Autograd mechanics — PyTorch 1.10.0 documentation 1. There is also a tutorial for that.