BERT pretraining routine
Hi,

I was wondering whether the scripts for finetuning can be used to pretrain BERT from scratch on a small dataset that does not require TPUs - is there any difference with the TF pretrain code (different batch sampling or train loss evaluation) other than the TPU support?

Thank you very much in advance!


I would not advise to use them for training from scratch. See #751 for discussion and links.