There is a paper that proposes tuning the learning rate using the gradient of the update rule with respect to the learning rate itself. Basically, it dynamically learns the learning rate during training.
If you are interested, here is the paper https://arxiv.org/abs/1703.04782 280 and my implementation https://github.com/jpeg729/pytorch_bits/blob/master/optim/adam_hd.py#L67 430