MLM performance difference between bert-base-cased and Conversational BERT
## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

albert, bert, GPT2, XLM: @LysandreJik 
Authors of Conversational BERT: @dilyararimovna @yoptar from DeepPavlov


## Information

I am using `bert-base-cased` and [Conversational BERT](https://huggingface.co/DeepPavlov/bert-base-cased-conversational) for MLM using the pipeline tool. The only difference between the two is that Conversational BERT was initialized with `bert-base-cased` and further pre-trained on social media text data (Twitter and the like) with a custom vocabulary (more info on the model card). 

As I'm using models that were not yet finetuned, I understand why I'm getting an output telling me that some weights were not used and that some weights are newly initialized (cf [this issue](https://github.com/huggingface/transformers/issues/5421)). What I don't understand is that even though the two models seem to have the same architecture and were trained on similar tasks (MLM and next sentence prediction), the performance on MLM is really poor for Conversational BERT and pretty good for `bert-base-cased`. This is all the more weird that I used Conversational BERT for tweet classification and found that it performed better than `bert-base-cased` on this downstream task. 

For example, when giving '[MASK] lost his job yesterday' as input, the output for `bert-base-cased` (`pipeline_bert('[MASK] lost his job yesterday')`) is 

[{'score': 0.8036763072013855,
  'sequence': '[CLS] he lost his job yesterday [SEP]',
  'token': 2002,
  'token_str': 'he'},
 {'score': 0.005656923167407513,
  'sequence': '[CLS] i lost his job yesterday [SEP]',
  'token': 1045,
  'token_str': 'i'},
 {'score': 0.005227738991379738,
  'sequence': '[CLS] dad lost his job yesterday [SEP]',
  'token': 3611,
  'token_str': 'dad'},
 {'score': 0.0032391520217061043,
  'sequence': '[CLS] david lost his job yesterday [SEP]',
  'token': 2585,
  'token_str': 'david'},
 {'score': 0.0028738391119986773,
  'sequence': '[CLS] and lost his job yesterday [SEP]',
  'token': 1998,
  'token_str': 'and'}]

and the output for Conversational BERT (`pipeline_convbert('[MASK] lost his job yesterday')`) is:

[{'score': 0.0005837088683620095,
  'sequence': '[CLS]As lost his job yesterday [SEP]',
  'token': 23390,
  'token_str': '##As'},
 {'score': 0.0004703140293713659,
  'sequence': '[CLS] rock lost his job yesterday [SEP]',
  'token': 2067,
  'token_str': 'rock'},
 {'score': 0.0004569509474094957,
  'sequence': '[CLS] ACT lost his job yesterday [SEP]',
  'token': 21111,
  'token_str': 'ACT'},
 {'score': 0.0004535183834377676,
  'sequence': '[CLS] colour lost his job yesterday [SEP]',
  'token': 5922,
  'token_str': 'colour'},
 {'score': 0.0004286181356292218,
  'sequence': '[CLS]inas lost his job yesterday [SEP]',
  'token': 16924,
  'token_str': '##inas'}]



I think this has to do with weight initializations but I'm not grasping everything in the output. Some points that are unclear:
- `['cls.seq_relationship.weight', 'cls.seq_relationship.bias']` are not used in the case of `bert-base-cased` (see output) but are used (or missing?) in the case of Conversational BERT
- Only `['cls.predictions.decoder.bias']` is newly initialized in the case of `bert-base-cased` but all of the following weights are initialized in the case of Conversational BERT: `['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias']`


## To reproduce

Steps to reproduce the behavior:

### bert-base-cased:

#### Input:

`pipeline_bert = pipeline('fill-mask', model='bert-base-uncased', tokenizer='bert-base-uncased', config='bert-base-uncased')`

#### Output:
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['cls.predictions.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

### Conversational BERT

#### Input:

`pipeline_convbert = pipeline('fill-mask', model='DeepPavlov/bert-base-cased-conversational', tokenizer='DeepPavlov/bert-base-cased-conversational', config='DeepPavlov/bert-base-cased-conversational')`

#### Output:

Some weights of BertForMaskedLM were not initialized from the model checkpoint at DeepPavlov/bert-base-cased-conversational and are newly initialized: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
