Hi @kushalj001,
Maybe you can consider XLM-Roberta (XLM-R) instead of Roberta?
XLM-R supports 100 languages out-of-the-box, including its tokenizer which can tokenize most language in the world.
This kaggle notebook show how to finetune 3 languages simultaneously on XLM-R with TF2+TPU , which is extremely efficient (10x faster than P100)
https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm 30