Jonathan_R_Williford:

dynamic learning rates


Sorry guys, but these comments seem very misleading!
Optimizers have a fixed learning rate for all parameters. param_group['lr'] would allow you to set a different LR for each layer of the network, but it’s generally not used very often, and most people have 1 single LR for the whole nn.
What Adam does is to save a running average of the gradients for each parameter (not a LR!). The learning rate is still the same throughout the whole training, unless you use a lr_scheduler like CosineAnnealingLR etc.