Update:
I have tried with different learning rates and dropout but WER did not improve much.
So far I have found learning rate 1e-4 works best for me.
Though there isn’t a huge improvement in WER, WER and validation loss does goes down with steps.
Still the best WER I have got is 0.52.
Dhivehi is a very small dataset and unfortunately there isn’t any other dataset available for Dhivehi other than Common voice.