Hi @ivokun, nice also to see another Person interested in text generation for Indonesian.
Btw, did you mean gpt2-xl? We had experience to pretrain gpt2-large on 68GB datasets, it took around 6 days for 1 epoch using tpu v3-8. If you want to train gpt2-xl, which is twice of the size of gpt-large, on Indonesian oscar dataset (around 30GB), you would need also around 6 days for an epoch. I am just not sure if you have enough ram for training gpt2-xl on tpu v3-8.