I’ll be trying the scheduler , let’s hope for the best , as for the weight update I should leave like that the inverse square root ? I’ll be getting a larger dataset soon-ish ( the one I used has 9230 examples , this one will have about 250,000 but with 28 classes ) that’s why i’m trying to tune and squeeze a very good model hyper-paramertes so I woudln’t have to repeat the whole process again on the large dataset.
Again thank you for your help , wish you all the best.
cheers