Check the Per-parameter options section here: http://pytorch.org/docs/optim.html 6.2k
Instead to feeding in a generator over all parameters, pass an iterable of  dicts, each with the key params and the value as the parameter group. It should be simple to group our model params according to the learning rates you want to apply to them.