'Some weights of BertModel were not initialized from the model checkpoint at ./model and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']'
Hi everyone,

I ran [ run_mlm.py ](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) to continue pertaining uncased BERT directly from the examples on this repo, but once I load the newly saved pretrained Bert Model, I receive a warning - "'Some weights of BertModel were not initialized from the model checkpoint at ./model and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']'"

I'm trying to fine-tune the model on a sentiment analysis task, but I'm getting horrible results and I wonder if it has something to do with this? Thanks for your help.
maybe related to #8793 , hope could help.<|||||>> maybe related to #8793 , hope could help.

seems to be related. I get high variance in accuracy, I guessed it was probably because of the random initialization of those two weights.<|||||>If you're using the `run_mlm.py`, then you're doing masked language modeling with the `BertForMaskedLM` model. This model does not make use of the pooler, hence why those two layers are randomly initialized. They're not used for predictions or training.<|||||>@LysandreJik would it make more sense to load the saved model using BertModel.load_pretrained(saved_mlm_model) or would it be better to use BertModel.load("bert-base-uncased") and copy the weights over from the saved model? <|||||>I also met this problem. Have you solved it?<|||||>@wenHK It's actually not really relevant. The BertForMaskedLM model doesn't use the pooler layer, so thus why there are no weights assigned. You don't really need to worry about the warning.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.