Adding and consequently removing tokens leads to incorrect number of input embeddings
## Environment info
- `transformers` version: 4.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Models:
- gpt2: @patrickvonplaten, @LysandreJik


Library:
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik

## Information

Using `gpt2-medium`.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I am attempting to undo `add_tokens()` and `resize_token_embeddings()` for a given, fine-tuned gpt2-medium model. I deleted the token `del tokenizer.added_tokens_encoder[token]` and `model.resize_token_embeddings(len(tokenizer))`, but there remain too many embeddings in the model and consequently, the output is corrupted.

Steps to reproduce the behavior:

```python
    from transformers import GPT2Tokenizer, GPT2LMHeadModel
    model_path = "gpt2"

    tokenizer = GPT2Tokenizer.from_pretrained(model_path)
    model = GPT2LMHeadModel.from_pretrained(model_path)
    def speak(model, tokenizer, prefix):
        input_ids = tokenizer.encode(prefix, return_tensors='pt')
        output_ids = model.generate(input_ids, max_length=15, return_dict_in_generate=True, do_sample=False).sequences
        print(tokenizer.decode(output_ids[0]))
        print(len(tokenizer), len(tokenizer.encoder))
        print(model.get_input_embeddings())
        print('\n\n')

    # out-of-the-box
    speak(model, tokenizer, 'I like cheese')

    # added token
    tokenizer.add_tokens('cheese')
    model.resize_token_embeddings(len(tokenizer))
    speak(model, tokenizer, 'I like cheese')

    # removed token
    del tokenizer.added_tokens_encoder['cheese']
    model.resize_token_embeddings(len(tokenizer))
    speak(model, tokenizer, 'I like cheese')

```

This results in
```
I like cheese.<|endoftext|>
50258 50257
Embedding(50257, 1024)

I like cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese
50259 50257
Embedding(50259, 1024)

I like<|endoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|> <|startoftext|>
50258 50257
Embedding(50258, 1024)
```

## Expected behavior
`model.get_input_embeddings()` of the third output should be equal to the first output `Embedding(50257, 1024)`. Note that I used a fine-tuned version of `gpt2-medium` and I wasn't able to recreate the issue entirely with a pretrained model, but even the deterministic output of a pretrained model will change after deleting a previously added token.

Is this expected behavior? 

Hey @doubleplusnice,

I can't really reproduce the bug. 

When running your code the output I get is:

```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
/home/patrick/python_bin/transformers/generation_utils.py:963: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
  warnings.warn(
I like cheese, but I don't like cheese. I like cheese because
50257 50257
Embedding(50257, 768)



Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
I like cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese cheese
50258 50257
Embedding(50258, 768)



Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
I like<|endoftext|>The first time I saw the new "The Walking Dead"
50257 50257
Embedding(50257, 768)
```

which seems correct to me.

Can you maybe try to update on master? Also, I'm testing with the `gpt2` checkpoint.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.