The docs here:   http://pytorch.org/docs/optim.html 118 explain how to set different learning rates for each layer in the model. However, I have a slightly different problem - I need to be able to set different learning rates for each weight within a layer. More specifically, I would like the weights connected to earlier output nodes to have a higher learning rate than those connected to later nodes, as is perhaps better explained by this picture:
expw.png1000×500 47.9 KB
 Any ideas as to how I might go about implementing this?