I want to inspect the growing of the params of each step. The value that I want is not exactly gradients which are possible to grab using ‘param.grad’ because if SGD with momentum is applied, the updated value will be

lr * (momentum term 90% + gradient only 10%)

, and I want to have this aggregated value.
I know that it is possible to do by calculating the different between params of current step and previous step. But in fact, this value is calculated by the optimizer function, optimizer.step(), as shown in figure below:

Screenshot from 2019-03-25 18-24-55.png762×834 67 KB

I want to get ‘d_p’ value from second last line, p.data.add_(-group[‘lr’], d_p), because it is the real value after calculated by optimizer that is updated to the network parameters. Is there any way to get it correctly? Unless I need to override the step function by myself.