Hi,
Thanks for the response, I haven’t got time to check the imbalance in the dataset, I was thinking of this reason as well.
Also I have played it a little bit, decreasing the learning rate does increases the validation accuracy, but there is a research paper I am trying to implement and they started with a high LR, so I did the same just to see the difference. With high LR, the accuracy has decreased but due to slow decay the accuracy is rising but not risen up to the accuracy achieved with smaller learning rate I trained with earlier yet . Upon checking the results of validation and testing, I think it might be a data imbalance but I was wondering if gradient clipping and dropouts is also contributing to this problem or not.
P.S.  I employed gradient clipping only because I was using LSTMs. And yeah, I am doing model.train() and model.eval() switches.