Thank you. Alright, I can now see nonzero values in
m2.multp.grad.data
after I do the optimizer steps, but the value of multp still doesn’t change during training. Will investigate. I think it might have something to do with the learning rate.
Now, how can I specify a special learning rate for this new param? I used to do something like this for my convolutional layers to make a specific smaller learning rate for layer ‘conv2’.
optimizer = optim.Adam([{‘params’: model.conv2.parameters(), ‘lr’: opt.lr 10*0.1}], lr=opt.lr 5)
Now if I do this:
optimizer = optim.Adam([{‘params’: model.multp.parameters(), ‘lr’: opt.lr 10*0.1}], lr=opt.lr 5)
I get an error File “/home/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py”, line 63, in getattr
raise AttributeError(name)
AttributeError: parameters
UPD: I replaced my ‘custom’ Adam (see above) that had different learning rates by a regular Adam which has the same learning rate for all the layers, and it worked - my multp started to become updated during the course of training (it also worked with SGD). However my question with setting a special learning rate for the nn.Parameter remains.