Adaptive learning rate
How do I change the learning rate of an optimizer during the training phase? thanks
davidenitti
trypag
If you want to change the LR we recommend reconstructing the optimizer with new parameters.
apaszke
Oh, ok sorry ! things move fast <SCODE>def adjust_learning_rate(optimizer, epoch):
    """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
    lr = args.lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

<ECODE>
trypag
ecolss
apaszke
ruotianluo
lysuhin
mderakhshani
Keras provides two functions which are fairly straightforward to implement, and everyone loves them: This one reduces LR when gradient is stuck on a plateau for past “X=patience” epochs: This one stops you from burning up your Amazon AWS $$$ credits if your model is not learning anything after “X=patience” epochs: Would PyTorch be open to adding something like this?
FuriouslyCurious
Any update on this thread about learning rate decay?
ecolss
In the models I’m training right now I see an increase in the loss when I construct a new optimizer to decay the learning rate.
will
Rongzhao_Zhan
Hi, I hope this clears things up.
chsasank
apaszke
will
How about more general situations where it is desired to adaptively adjust any parameters defined in the model? Is it practical to reconstruct optim in every loop? Maybe a mechanism like placeholder? Thanks!
lliu25
smth
lliu25
ncullen93