csailnadi:

Do you know where add_ function is defined? However it is probably not that important for me. Just when I use PyCharm “go to declaration” it brings me to empty declaration. So the definition is lost somewhere in class hierarchy.


I’m not sure how PyCharm looks up the definition, but the method is an inplace version of torch.add 1, where alpha will be multiplied with other.



 csailnadi:

Say I have 3 layers, how do I know what weights are updated at any point?
E.g I want to do something with grad before it is used for weights update, but my action depends on a layer


In that case, I would recommend to use register_hook on the desired parameter to change the gradients before the optimizer.step() method.
Layer-dependent gradient manipulations in the optimizer would most likely break it for any other use case.