bug in init_bert_weights
hi ,
    there is a bug  in init_bert_weights(). 
    the BERTLayerNorm  has twice init, the first init  is in the BERTLayerNorm module __init__(). the second init in init_bert_weights().
if you want to get pre-training model that is not from google model, the second init will lead to bad convergence  in my experiment 。 gamma is variance , beta is mean, there  are usually  1 and 0. the second init change it.
first: 
        self.gamma = nn.Parameter(torch.ones(config.hidden_size))
        self.beta = nn.Parameter(torch.zeros(config.hidden_size))
second:
            elif isinstance(module, BERTLayerNorm):
                 module.beta.data.normal_(mean=0.0, std=config.initializer_range)
                module.gamma.data.normal_(mean=0.0, std=config.initializer_range)

Fixed, thanks