Relevant stack overflow thread on object deletion in python 3
I think in general in python, del only removes that particular reference to the object. It does not remove any other reference to that object’s child members. So when you assign the model parameters to the optimizer, the model parameters now have a reference in the optimizer. The only way to remove those would be to manually remove either the optimizer itself or all references to the relevant param groups in the optimizer, along with any other references to any of the child members of the model.
some_module = model.layer
del model
del optimizer
del some_module