[pretrained] model classes aren't checking the arch of the pretrained model it loads
While comparing different models trained on xsum (most of which are Bart) I made a mistake and passed "google/pegasus-xsum" to `BartForConditionalGeneration`  

```
BartForConditionalGeneration.from_pretrained("google/pegasus-xsum")
```

I got:

```
Some weights of the model checkpoint at google/pegasus-xsum were not used when initializing BartForConditionalGeneration: ['model.encoder.layer_norm.weight', 'model.encoder.layer_norm.bias', 'model.decoder.layer_norm.weight', 'model.decoder.layer_norm.bias']
- This IS expected if you are initializing BartForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BartForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BartForConditionalGeneration were not initialized from the model checkpoint at google/pegasus-xsum and are newly initialized: ['model.encoder.embed_positions.weight', 'model.encoder.layernorm_embedding.weight', 'model.encoder.layernorm_embedding.bias', 'model.decoder.embed_positions.weight', 'model.decoder.layernorm_embedding.weight', 'model.decoder.layernorm_embedding.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "./bart-summarize2.py", line 8, in <module>
    tokenizer = BartTokenizer.from_pretrained(mname)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    return cls._from_pretrained(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1860, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/roberta/tokenization_roberta.py", line 159, in __init__
    super().__init__(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/gpt2/tokenization_gpt2.py", line 179, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

Any reason why the model class doesn't check that it's being fed a wrong architecture? It could detect that and give a corresponding error message, rather than spitting random errors like above? I was pretty sure it was a bug in pegasus model until I noticed that pegasus != Bart.

Thanks.

@LysandreJik 

Ah indeed, that's a good request ! There's no reason, we could definitely raise a warning when loading the weights by checking the model type in the configuration against the arch's model type. Do you want to open a PR?<|||||>Why a warning and not an assert? If the code throws a totally unrelated long backtrace how would a user know to search for an earlier warnings?

Do you see a use-case where someone may need to load mismatching arch for the given model?<|||||>After thinking about it, you're right that an error would be better. I can't think of use-cases where that would affect someone's workflow negatively.<|||||>> Do you want to open a PR?

I could, but realistically it might not happen soon. But since it's not a complicated task perhaps asking the community to help? I guess it'd be as simple as:

1. read the config of the downloaded model as soon as the config got downloaded
2. compare `config.arch` with model's arch
3. assert if mismatch<|||||>Hi @LysandreJik Does someone work on that ? I'd like to make my first contribution to the project<|||||>Hi @ankh6, feel free to work on it! The issue is not reserved until a PR is opened with some progress made towards solving the issue.<|||||>And when you solve it, one test can be:
```
python -c 'from transformers import PegasusForConditionalGeneration; PegasusForConditionalGeneration.from_pretrained("patrickvonplaten/t5-tiny-random")'
```
but this one doesn't crash, just spits a lot of warnings. 

This one does crash:
```
python -c 'from transformers import BartForConditionalGeneration; BartForConditionalGeneration.from_pretrained("prajjwal1/bert-tiny")'
```

So it'd be a better candidate to go into the test suite. 

We want a tiny model so that it runs the test fast.<|||||>@LysandreJik  If I understand correctly we should check that the input is in the PRETRAINED_VOCAB_FILES_MAP object (for this issue). Should the assertion occur when we call is_torch_available method, i.e. in src/transformers/models/gpt2/__init__.py, ? <|||||>As soon as you retrieved the config file and you know which model's class is used, so that you have the 2 things to compare.

It definitely shouldn't happen in the specific model files, but inside the common library.

Most likely there should be one check inside one of the super-classes for model's `from_pretrained` and the same for the tokenizer. Since either may have this conflict.<|||||>Hi, 
I've made some progress on this issue. Think I've fixed it for initiating models. 
To show if my approach is fine shall I submit a PR? 

I've essentially added an assert statement in the `from_pretrained` method in the `PretrainedConfig` class.  <|||||>That sounds about right, and yes PR please - thank you!<|||||>Added a pull request #10586 