Have no GPU to train language modelling
Sorry I open this issue, is not issue of this repository. 

I very appreciate what the authors created this repository, help us to more understand how BERT works and implement on several tasks.

So I have a problem with training because I have not GPU to train language modelling, I have Indonesian dataset (about 2GB) that trainable for language modelling using this repo, could anyone help me to train this dataset? If you could help, you have permission to open source or use the model trained. 

I hope it will be give more models provided and to make NLP community more interest in latest NLP models especially Indonesian.

You can email me directly on khaerulumam42@gmail.com or comment below.

Thank you very much
[You can train a tensorflow model using google colab for free](https://github.com/google-research/bert#using-bert-in-colab). After training it, you can [convert your tf model to pytorch](https://github.com/huggingface/pytorch-pretrained-BERT#command-line-interface). <|||||>Or use 300 usd credit for google cloud, that you get when you signup i believe.<|||||>Thank you @oliverguhr and @Oxi84 for suggestions. 

I have tried both methods, using google colabs and GPU as runtime processor, it took about 240hours for every epoch (maybe if I use apex, it will be faster but I think still hundreds of hours), i think it's impossible to run google colabs dozens of days.

I got free trial for GCP, unfortunately Google not provide GPU for free trial version. I try training use GCP with 2CPU and 13GB RAM, it take 200thousands of hours training, is sooo long time.

Maybe I should reduce corpus size?

Thanks<|||||>Or smaller vocabulary. I am pretty sure you can even use TPU on Google cloud, let someone else confirm that.<|||||>@Oxi84 
For my classification task, I noticed that training the model with just 40 mb of data will give me already pretty good results. Training with the full 1,5 GB of my dataset improves the results by just 2-3% accuracy. 
So you might start with a (random)  subset of your data and improve the size step by step and see if your scores get better. <|||||>Oh nice insight @oliverguhr , thank you. I will try to reduce training data and train.
