Removing tokens from the tokenizer
Any methods that I can remove unwanted tokens from the tokenizer?

Referring to #4827 , I tried to remove tokens from the tokenizer with the following code. 

First, I fetch the tokenizer from huggingface hub.
```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("klue/roberta-base")
print(len(tokenizer.vocab))
```

```
32000
```

From the fetched tokenizer, I tried to remove tokens such as `[unused363]`.
So I first extracted tokens with 'unused' and deleted afterwards.

```python
# get all tokens with "unused" in target_tokenizer
unwanted_words = []
for word in tokenizer.vocab:
    if "unused" in word:
        unwanted_words.append(word)

# remove all unwanted tokens from target_tokenizer
for word in unwanted_words:
    del tokenizer.vocab[word]

print(len(tokenizer.vocab))
```

```
32000
```

Apparently, `del` didn't do its job. 
The list `unwanted_words` has 500 elements but none of which are removed from the tokenizer.

Any other methods that I can refer to? 
@snoop2head , I will answer your question purely "technically" because in general it can be quite complicated to remove tokens from a tokenizer that has been trained with a certain algorithm. 

Indeed, it can seems a bit complicated to modify a tokenizer fast because they wrap the [tokenizers](https://huggingface.co/docs/tokenizers/python/latest/index.html) library which is not executed in python but in rust. In particular the attribute vocab cannot be modified because it is in fact just the reading of an attribute of an object which runs in rust. So when we want to modify a part of the tokenizer that runs in rust we have to recreate it.

Here are the steps to remove tokens from your vocabulary: 

1) Get your tokenizer and the list of tokens you want to remove
```python
import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("klue/roberta-base")

# get all tokens with "unused" in target_tokenizer
unwanted_words = []
for word in tokenizer.vocab:
    if "unused" in word:
        unwanted_words.append(word)
```

2) Get the arguments that allowed to initialize the "model" component of the `backend_tokenizer`.
```python
model_state = json.loads(tokenizer.backend_tokenizer.model.__getstate__())
print(len(model_state["vocab"]))
# 32000
```

3) Modify the initialization arguments, in particular the vocabulary to remove the tokens we don't want
```python
# remove all unwanted tokens from the vocabulary
for word in unwanted_words:
    del model_state["vocab"][word]

print(len(model_state["vocab"]))
# 31500
```

4) Intitialize again the "model" component of the `backend_tokenizer` with the new vocabulary
```python
from tokenizers import models

model_class = getattr(models, model_state.pop("type"))

tokenizer.backend_tokenizer.model = model_class(**model_state)

print(len(tokenizer.vocab))
# 31500
```

I take this opportunity, could you tell us more about why you want to be able to remove tokens from your vocabulary?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I will close this issue as there is not much activity. But don't hesitate to answer later if you want to bring us more information. :slightly_smiling_face: <|||||>@SaulLu  How can we remove stale terms and add newer terms? I am interested to train a model on evolving text where words go through a vocabulary and semantic shift. In such a case, I would train a new tokenizer on the latest data and create a new vocabulary of common terms between old and new tokenizer along with removal of stale tokens and adding newer ones from the new tokenizer.<|||||>@SaulLu, thanks for the example you gave above. I tried using the same approach with GPT but it fails at the point of initializing the "model" component of the backend_tokenizer with the new vocabulary


```
#1. Get your tokenizer and the list of tokens you want to remove

import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

# get all tokens with "unused" in target_tokenizer
unwanted_words = [ 'ply', 'Ġmor','Ġprovide','IC','ung','Ġparty', 'Ġexist', 'Ġmag',]


#2. Get the arguments that allowed to initialize the "model" component of the backend_tokenizer.
model_state = json.loads(tokenizer.backend_tokenizer.model.__getstate__())
print(len(model_state["vocab"]))


#3. Modify the initialization arguments, in particular the vocabulary to remove the tokens we don't want

# remove all unwanted tokens from the vocabulary
for word in unwanted_words:
    del model_state["vocab"][word]

print(len(model_state["vocab"]))


#4. Intitialize again the "model" component of the backend_tokenizer with the new vocabulary

from tokenizers import models

model_class = getattr(models, model_state.pop("type"))

tokenizer.backend_tokenizer.model = model_class(**model_state)

print(len(tokenizer.vocab))
```

The error I have is below:
```
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

[<ipython-input-21-fa908d23c419>](https://localhost:8080/#) in <module>
     30 model_class = getattr(models, model_state.pop("type"))
     31 
---> 32 tokenizer.backend_tokenizer.model = model_class(**model_state)
     33 
     34 print(len(tokenizer.vocab))

TypeError: argument 'merges': failed to extract enum PyMerges ('Merges | Filename')
- variant Merges (Merges): TypeError: failed to extract field PyMerges::Merges.0, caused by TypeError: 'str' object cannot be converted to 'PyTuple'
- variant Filename (Filename): TypeError: failed to extract field PyMerges::Filename.0, caused by TypeError: 'list' object cannot be converted to 'PyString'
```

Is there any thing I can do to get it to work?