we have a dataset for Turkish language with 35GB. we’ve pre processed and cleaned the whole text. but we have no gpu, no muscle computer thats why we hope maybe colab pro can make it happen.
we want to pre-train a BERT, Longformer, BigBird and GPT-2.
When we start tokenize text to train colab collapse. We are looking for a complete guide to train theese models via checkpoints and availabe to colab pro.
No google cloud suggestions please. It costs more than 7k$'s. And we are just independent student researchers with pocket money.
So please guide us. We will share models via huggingface so it is a win win situation.
Kind regards
emre