addeing tokens to tokenizer for training and inference worsens  prediction immensely
I use the huggingface Trainer class to finetune a mT5 model. Since my data (AMR graphs) contains mayn tokens like `:ARG0`, `:op2` or `:name`, which are generally split into wordpieces I added those 219 tokens to the tokenizer with

```
from  transformers import MT5ForConditionalGeneration, MT5Tokenizer
tokenizer = MT5Tokenizer.from_pretrained("google/mt5-base")
model     = MT5ForConditionalGeneration.from_pretrained("google/mt5-base")
toks = [ ... my ist of tokens ...]
tokenizer.add_tokens(toks, special_tokens=False)
model.resize_token_embeddings(len(tokenizer))
```

and than starting the training.

Once the model is finetuned I run my test, and once again I add the same tokens in the same order to the tokenizer. But the result is catastrophic, it drops from 80% F1 to 50%, so evidently something is going wrong. I compared the tokenisation with and without added tokens  and this looks OK. But I have not the slightest idea where to check. Can you give me a hint about the error I'm committing ?

## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyTorch version (GPU): 1.9.1
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
To clarify me question: I thought (possibly erroneously) that the new tokens will have random vectors which will be "learned" during the finetuning. If this is not the case, I have my answer ...<|||||>Hey @jheinecke. The best place to ask this question would be the [forum](https://discuss.huggingface.co/). We use issues for bug reports and feature requests. Thanks! <|||||>OK, thanks, I missed that. I'll pose my question there