If I’m reading it right you’re only dropping your learning rate once during the training. You should consider using a function of some sort to decrease the learning rate as your training progresses. My first guess would be that the LR you’re using is too high at the end to make meaningful progress beyond a certain point.