MLM Loss not decreasing when pretraining Bert from scratch
I want to pretrain bert base model from scratch with en-wikipedia dataset, since I havn't found a bookcorpus copy. The code I used was adapted from pytorch-pretrained-bert and Nvidia Megatron-LM. I can finetune BERT on SQuAD and get SOTA result. But for pretraining, the MLM loss stays around 7.3. I have no idea why this happens. Can someone offer some possible solutions? Great thanks!