Hi, I am trying to manually set the gradients of the entire network during training.  Specifically I am interested to do the following:
optimizer.zero_grad()
# set gradients for all components of the network
optimizer.step()

I know that for a single module we can use following:

  
    
    
    How to set gradients manually and update weights using that? autograd
  
  
    1.Are you 100% sure that there is no difference in param after the step?, maybe you use an adaptive optimizer and the learning rate is almost zero? 
2.Are you 100% sure that the optimizer is responsible for this param update? 
I wrote this code to test this: (I am using v0.4.0): 
import torch

target = torch.randn(1, 10)
input = torch.randn(1, 20)
network = torch.nn.Linear(20, 10)
optimizer = torch.optim.SGD(network.parameters(), lr=0.01)
optimizer.zero_grad()
loss = ((network(input)-target)**2)…
  


But it would be complicated to set each module of the entire network in this way since the network may contain dozens of different modules.  I am wondering if there is a better way to do so?  I am thinking of possibilities of using something similar to state_dict when copying weights for the entire network, just wonder if there are similar things for gradients?
Thank you!