Solved by mmisiur in post #2 
              
              
                Read the docs: https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate 
There are many ways to do this, including plateau.