LayerNorm initialization
The LayerNorm gamma and beta should be initialized by .fill_(1.0) and .zero_().

reference links:

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2298

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2308
Yes, the weights are overwritten by the loading model. But if the code is used to pretrain a model from scratch, it might affect performance. (related issue: https://github.com/huggingface/pytorch-pretrained-BERT/issues/143 )<|||||>Great, thanks @donglixp !