Bert initialization
When I train bert model from scratch, it can not convergence and the loss does not decrease. Still don't work even try different learning rate many times.

But It works when I tried the tf version. I checked code and there have no much difference except the initialization.

So does anybody have some ideas about this?
Hi, I think you'll find this [particular issue interesting](https://github.com/huggingface/pytorch-transformers/issues/202).

[Thomas Wolf's comment](https://github.com/huggingface/pytorch-transformers/issues/202#issuecomment-522613642) in particular may be of help.<|||||>> Hi, I think you'll find this [particular issue interesting](https://github.com/huggingface/pytorch-transformers/issues/202).
> 
> [Thomas Wolf's comment](https://github.com/huggingface/pytorch-transformers/issues/202#issuecomment-522613642) in particular may be of help.

Thanks. But I am not saying to train a language model from scratch, I am saying to train a glue task from scratch. So I think there has much difference between this.