Mismatch of special token ids between config and tokenizer config
### System Info

```shell
main branch
```


### Who can help?

@SaulLu, @LysandreJik

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Example mismatch:
* OPT-* has `bos_token_id` set to 0 in `config.json` while `bos_token` set to `</s>` (which leads to `bos_token_id` as 2) in `tokenizer_config.json`. As a result, `model.config.bos_token_id != tokenizer.bos_token_id`. It might cause subtle bug during generation as the `.generate` load special token ids from `model.config` by default (I don't think it will cause bug for OPT but it might cause subtle bug if `eos_token` is mismatch)
https://github.com/huggingface/transformers/blob/8f46ac98498dd47701971064617e00d7e723a98e/src/transformers/generation_utils.py#L1123-L1134
* Others models might have similar issues.

### Expected behavior

Ideally we would have a single source of truth for special tokens id, if not, we might want to have some assertions to see if there's any mismatch or document this potential pitfall. I understand tokenizer and model are decoupled so it might be hard to introduce assertion in the library, and I am not sure how realistic to have some sort of unittests for mismatch for the model zoo.
Thanks for the report @shijie-wu ! That's indeed a bug in the model's config. Will update all of them now. The BOS token is identical to the EOS token and should therefore be 2 (=> the tokenizer has it correct here).<|||||>Also for the record, this is not a critical bug since in 99% of the times a user prompts OPT with something. This means the user passes a string through the tokenizer and then to the model:

```py
input_ids = tokenizer("some prompt", return_tensors="pt").input_ids
sequence = opt.generate(input_ids)
```

In this case the tokenizer **always** correctly prepends the EOS token. The only time when the model config would cause a bug is if the user would generate from an empty prompt:

```py
sequence = opt.generate()
```<|||||>Thanks for updating OPT!

I understand this won't cause bug in most cases. But hypothetically speaking, if users misconfig `eos_token` in `model.config` and it doesn't match `tokenizer.eos_token`, it would cause the generation to cut short silently. I understand having an assertion might not make sense but documenting it somewhere might be helpful?<|||||>> Also for the record, this is not a critical bug since in 99% of the times a user prompts OPT with something. This means the user passes a string through the tokenizer and then to the model:
> 
> ```python
> input_ids = tokenizer("some prompt", return_tensors="pt").input_ids
> sequence = opt.generate(input_ids)
> ```
> 
> In this case the tokenizer **always** correctly prepends the EOS token. The only time when the model config would cause a bug is if the user would generate from an empty prompt:
> 
> ```python
> sequence = opt.generate()
> ```

Hi, @patrickvonplaten  I notice that the tokenizer of OPT models uses `</s>` for `eos_token`, `bos_token` and `unk_token` in `special_tokens_map`.  Is it intended?
```
# transformers 4.20.1
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-1.3b') # all OPT models
print(tokenizer.special_tokens_map)
# {'bos_token': '</s>', 'eos_token': '</s>', 'unk_token': '</s>', 'pad_token': '<pad>'}
```
Another issue I found is the the vocab size in the tokenizer does not match the size of embedding module of OPT models. Tokenizer has vocab size 50265 while the embedding table in opt models has 50272.
```
# transformers 4.20.1
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained('facebook/opt-1.3b') # all OPT models
print("Embedding table: " ,model.decode.embed_tokens.weight.shape[0])
# Embedding table:  50272
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-1.3b') # all OPT models
print("Vocab size:", tokenizer.vocab_size)
# Vocab size: 50265
```

Just to confirm is it a bug or intended? Thanks.

<|||||>Hey @git-xp, 

Yes OPT indeed uses the same token for both `bos_token` and `eos_token` being `</s>`. 

The `unk_token` should actually never really be produced by the tokenizer, since the tokenizer iss based on byte-level Byte-Pair-Encoding and thus will always produce a valid token, no matter what the input (cc @SaulLu just to verify that's correct what I'm saying here)
Also note that OPT's tokenizer is fully based on GPT2's tokenizer which also uses the same token for all BOS, EOS and UNK.

Now regarding the 2nd question, yes it's expected that the OPT model has more vocab entries in the model entry than the tokenizer has tokens. The final tokens of the model are simply never used (they've just been added so that the model has a weight matrix that's a better "power of 2" matrix - *i.e.* 50272 is divisible by 2**5 whereas 50265 is not divisible by 2 at all)<|||||>Your explanation that a byte-level tokenizer has no use as an unknown token is perfect @patrickvonplaten ! :+1: 