Have a look at http://pytorch.org/docs/master/optim.html#how-to-adjust-learning-rate 4.4k
It is available starting from 0.2.0
You can still use the code you mentioned to adjust the learning rate as you want.