Hey @mzimmer ,
again I would appreciate your help as I am trying to prune iteratively right now:
From my Conv modules, I pruned the weights using nn.utils.prune.
Now I want to finetune the model again - more specifically only the Conv modules, as the Linear modules were not pruned. Which parameters do I pass to the optimizer? weight_orig or weights?
Because model.named_parameters() returns only weight_orig and bias of each module.
The pruned weight is now accessible directly by calling module.weight.
So which parameters do I need to pass to the optimizer when I want to finetune the model again? Because I want pruned weights (which are basically just weights with the value zero) to stay zero.
My intuition says the pruned weights need to be passed now - how can I do that easily?
Maybe removing the weight_orig of the module using the remove function of the nn.utils.prune library?
Happy to hear your thoughts - maybe you can help me out.
Thanks!
UPDATE: I tried removing the weight_orig of the module using the remove function of the nn.utils.prune. Works in theory, but weights that were pruned (= set to zero) were updated during training again - resulting in non-zero weights again. So this cannot be the solution.