Hi @amankhandelia
For Hindi, the OSCAR corpus has almost 8GB which will be good-enough for a BERT like model.  IndicCorp 4 also has a Hindi corpus.
And then there’s mC4 2 which has almost 104GB of Hindi data which should be more than enough for this project 
Also, here’s an interesting insight from CamemBERT paper