Hi, I have a dynamic graph that adds/removes layers after some epochs.
I realized that the optimizer is not aware of newly added layer.
Q. Is it safe to call opt.add_param_group() after a few iterations of opt.step() in runtime?
Q. How can I delete some parameters from opt? (To save the memory in runtime)
There was a recommentation to reconstruct the optimizer:

  
    
    
    Dynamically changing parameter sizes autograd
  
  
    you will have to reconstruct the optimizer if you change the shapes of the parameters. there’s no real workaround. 
Alternatively, you can simply slice self.W in the forward function and keep it’s original shape intact. 
Like: 
out = torch.mm(input, self.W[inverse_mask, :]) # just an example

Here you dont repackage or change self.W, you just enforce a mask at runtime.
  


but I think it will lose optimizer states like momentum for existing parameters. (Cloning the optimizer would be the last resort)