Also, try implementing the logarithmically decreasing lr with https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ExponentialLR 1. It will most likely help you get closer to where you want to be