I tried a higher learning rate than 1e-5, which leads to a gradient explosion. if you observe up to 2k iterations the rate of decrease of error is pretty good but after that, the rate of decrease slows down, and towards 10k+ iterations it almost dead and not decreasing at all.
Yeah, I will try adapting the learning rate. Is there any guide on how to adapt?