Adding New Vocabulary Tokens to the Models
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,

How could I extend the vocabulary of the pre-trained models, e.g. by adding new tokens to the lookup table?

Any examples demonstrating this?
Hi, I believe this method does exactly what you're looking for: [add_tokens](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_tokens). There's an example right below it.<|||||>thanks @LysandreJik ! yes, that's exactly what I was looking for. A follow-up question: How could I initialize the embeddings of these "new tokens" to something I already have pre-computed? I assume currently, embedding for these new tokens will be randomly initialized.<|||||>You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.

```py
import torch
from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
model = BertModel.from_pretrained("bert-base-cased")

print(len(tokenizer))  # 28996
tokenizer.add_tokens(["NEW_TOKEN"])
print(len(tokenizer))  # 28997

model.resize_token_embeddings(len(tokenizer)) 
# The new vector is added at the end of the embedding matrix

print(model.embeddings.word_embeddings.weight[-1, :])
# Randomly generated matrix

model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])

print(model.embeddings.word_embeddings.weight[-1, :])
# outputs a vector of zeros of shape [768]
```<|||||>thanks @LysandreJik ! That should solve it quite neatly. I will reopen the issue in case I run into any issues. <|||||>Hello @LysandreJik ,

What is the difference between the following approaches?
  1. to train a tokenizer from scratch such as pointed in [hugginface blog](https://huggingface.co/blog/how-to-train#2-train-a-tokenizer); or
  2. to use [add_tokens](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_tokens) method?

Thank you in advance.



<|||||>Training a tokenizer from scratch would imply training a model from scratch as well - depending on the corpus used for the tokenizer, the tokens may be entirely different from another model's tokens trained on a similar corpus (except if you train the tokenizer using the exact same method and the exact same data).

Adding tokens adds tokens at the end of the tokenizer's vocabulary, essentially extending the vocabulary. The model's embedding matrix would need to be resized as well to take into account the new tokens, but all the other tokens would keep their representation as-is. Seeing as the new rows in the embedding matrix are randomly initialized, you would still need to fine-tune the model to a dataset containing such tokens.<|||||>@LysandreJik 
I have a dutch medical dataset (for Namen Entity Recognition) which contains a lot of domain-specific words. The dutch BERT tokenizer therefor outputs a lot of [UNK] tokens when it tokenizes. 
Given that I dispose over a corpus of 60k labelled tokens, and right now I have also a relatively small annotated corpus of 185k tokens, would it be best to:
- just add the most frequent out of vocab words to the vocab of the tokenizer
- start from a BERT checkpoint and do further pretraining on the unlabeled dataset (which is now of size 185k which is pretty small I assume..). There might be a possibility for me to obtain a much larger unannotated dataset of potentially millions of (unlabelled) tokens, but I was wondering if even millions of tokens is enough to do some meaningful further pretraining?
 
Thanks!<|||||>> Training a tokenizer from scratch would imply training a model from scratch as well - depending on the corpus used for the tokenizer, the tokens may be entirely different from another model's tokens trained on a similar corpus (except if you train the tokenizer using the exact same method and the exact same data).
> 
> Adding tokens adds tokens at the end of the tokenizer's vocabulary, essentially extending the vocabulary. The model's embedding matrix would need to be resized as well to take into account the new tokens, but all the other tokens would keep their representation as-is. Seeing as the new rows in the embedding matrix are randomly initialized, you would still need to fine-tune the model to a dataset containing such tokens.

Hey I would like to fine-tune the model as you suggested at the end to the dataset containing such tokens. Can you help me out on how I can do that?<|||||>If I add unknown tokens to the tokenizer and train the model on, say sentence pair similarity, while I suppose the new tokens embeddings will not have the correct relationship with other tokens, will the model output still be able to find similarity correctly given sufficient training on the model?<|||||>@LysandreJik Thank you for your suggestion. However, I run into trouble because altering the embedding turns the embedding tensor into a non-leaf tensor and hence cannot be optimized i.e. 
``` python 
model.embeddings.word_embeddings.weight.is_leaf # False
```
I cannot figure out how to fix this (I am torch beginner; sorry). Do you have any suggestions? <|||||>facing same issue; getting false for is_leaf<|||||>`BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True).get_vocab()` not return added token. How can I check if the new token is properly added to vocab dictionary? <|||||>> You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> 
> ```python
> import torch
> from transformers import BertTokenizer, BertModel
> 
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> model = BertModel.from_pretrained("bert-base-cased")
> 
> print(len(tokenizer))  # 28996
> tokenizer.add_tokens(["NEW_TOKEN"])
> print(len(tokenizer))  # 28997
> 
> model.resize_token_embeddings(len(tokenizer)) 
> # The new vector is added at the end of the embedding matrix
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # Randomly generated matrix
> 
> model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # outputs a vector of zeros of shape [768]
> ```

Hi,
I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
it may have lag or problem...
what should I do?<|||||>> 
> 
> > You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> > ```python
> > import torch
> > from transformers import BertTokenizer, BertModel
> > 
> > tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> > model = BertModel.from_pretrained("bert-base-cased")
> > 
> > print(len(tokenizer))  # 28996
> > tokenizer.add_tokens(["NEW_TOKEN"])
> > print(len(tokenizer))  # 28997
> > 
> > model.resize_token_embeddings(len(tokenizer)) 
> > # The new vector is added at the end of the embedding matrix
> > 
> > print(model.embeddings.word_embeddings.weight[-1, :])
> > # Randomly generated matrix
> > 
> > model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> > 
> > print(model.embeddings.word_embeddings.weight[-1, :])
> > # outputs a vector of zeros of shape [768]
> > ```
> 
> Hi,
> I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
> it may have lag or problem...
> what should I do?

Have you solved the problem? If so, can you share it with us?<|||||>> > > You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> > > ```python
> > > import torch
> > > from transformers import BertTokenizer, BertModel
> > > 
> > > tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> > > model = BertModel.from_pretrained("bert-base-cased")
> > > 
> > > print(len(tokenizer))  # 28996
> > > tokenizer.add_tokens(["NEW_TOKEN"])
> > > print(len(tokenizer))  # 28997
> > > 
> > > model.resize_token_embeddings(len(tokenizer)) 
> > > # The new vector is added at the end of the embedding matrix
> > > 
> > > print(model.embeddings.word_embeddings.weight[-1, :])
> > > # Randomly generated matrix
> > > 
> > > model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> > > 
> > > print(model.embeddings.word_embeddings.weight[-1, :])
> > > # outputs a vector of zeros of shape [768]
> > > ```
> > 
> > 
> > Hi,
> > I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
> > it may have lag or problem...
> > what should I do?
> 
> Have you solved the problem? If so, can you share it with us?

yes, it was because it takes a very long time to add all tokens. and I installed transformers from source:
pip install -U git+https://github.com/huggingface/transformers ,due to recently it was merged a PR that should speed this up dramatically and my problem solved.<|||||>thank you!




------------------&nbsp;原始邮件&nbsp;------------------
发件人: ***@***.***&gt;; 
发送时间: 2021年5月10日(星期一) 下午2:11
收件人: ***@***.***&gt;; 
抄送: "Patrick ***@***.***&gt;; ***@***.***&gt;; 
主题: Re: [huggingface/transformers] Adding New Vocabulary Tokens to the Models (#1413)





    
You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings weight. Here's an example with the BertModel.
 import torch from transformers import BertTokenizer, BertModel tokenizer = BertTokenizer.from_pretrained("bert-base-cased") model = BertModel.from_pretrained("bert-base-cased") print(len(tokenizer))  # 28996 tokenizer.add_tokens(["NEW_TOKEN"]) print(len(tokenizer))  # 28997 model.resize_token_embeddings(len(tokenizer))  # The new vector is added at the end of the embedding matrix print(model.embeddings.word_embeddings.weight[-1, :]) # Randomly generated matrix model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size]) print(model.embeddings.word_embeddings.weight[-1, :]) # outputs a vector of zeros of shape [768]
  
Hi,
 I tried this, but my code still stop in tokenizing the sentences section and doesn't pass it.
 it may have lag or problem...
 what should I do?
  
Have you solved the problem? If so, can you share it with us?
  
yes, it was because it takes a very long time to add all tokens. and I installed transformers from source:
 pip install -U git+https://github.com/huggingface/transformers ,due to recently it was merged a PR that should speed this up dramatically and my problem solved.
 
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.<|||||>> Training a tokenizer from scratch would imply training a model from scratch as well - depending on the corpus used for the tokenizer, the tokens may be entirely different from another model's tokens trained on a similar corpus (except if you train the tokenizer using the exact same method and the exact same data).
> 
> Adding tokens adds tokens at the end of the tokenizer's vocabulary, essentially extending the vocabulary. The model's embedding matrix would need to be resized as well to take into account the new tokens, but all the other tokens would keep their representation as-is. Seeing as the new rows in the embedding matrix are randomly initialized, you would still need to fine-tune the model to a dataset containing such tokens.

Why can't we repurpose the existing 999 unused tokens [UNK] instead of extending the vocab size? 
https://github.com/google-research/bert/issues/9#issuecomment-434796704<|||||>> You are right, these tokens will be randomly initialized. What I would do if I wanted to assign new values to this embedding (as an initialization), is to directly change the Embeddings `weight`. Here's an example with the `BertModel`.
> 
> ```python
> import torch
> from transformers import BertTokenizer, BertModel
> 
> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
> model = BertModel.from_pretrained("bert-base-cased")
> 
> print(len(tokenizer))  # 28996
> tokenizer.add_tokens(["NEW_TOKEN"])
> print(len(tokenizer))  # 28997
> 
> model.resize_token_embeddings(len(tokenizer)) 
> # The new vector is added at the end of the embedding matrix
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # Randomly generated matrix
> 
> model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
> 
> print(model.embeddings.word_embeddings.weight[-1, :])
> # outputs a vector of zeros of shape [768]
> ```

@LysandreJik when I ran your code the following error popped up. please help

**RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.**<|||||>> RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

You can fix that error by temporarily disabling gradient calculation. (Because initializing the weights is not an operation that needs to be accounted for in backpropagation.)

```python
with torch.no_grad():
    model.embeddings.word_embeddings.weight[-1, :] = torch.zeros([model.config.hidden_size])
```<|||||>why hidden_size? Is that specific to just Bert model? For Albert it should be different right?<|||||>How do we initialise the pre-existing embeddings for new tokens from old partitioned tokens?<|||||>> why hidden_size? Is that specific to just Bert model? For Albert it should be different right?

Hi, yes, I do believe the name can vary from model to model. For T5 model it seems to be `d_model`<|||||>> How do we initialise the pre-existing embeddings for new tokens from old partitioned tokens?

If I understand you correctly, we can initialise new tokens from already pre-trained ones with taking a mean of them: 
```
with torch.no_grad():
    for i, token in enumerate(reversed(added_tokens), start=1):
        tokenized = tokenizer.tokenize(token)
        tokenized_ids = tokenizer.convert_tokens_to_ids(tokenized)
        model.embeddings.word_embeddings.weight[-i, :] = model.embeddings.word_embeddings.weight[tokenized_ids].mean(axis=0)
```<|||||>> > How do we initialise the pre-existing embeddings for new tokens from old partitioned tokens?
> 
> If I understand you correctly, we can initialise new tokens from already pre-trained ones with taking a mean of them:
> 
> ```
> with torch.no_grad():
>     for i, token in enumerate(reversed(added_tokens), start=1):
>         tokenized = tokenizer.tokenize(token)
>         tokenized_ids = tokenizer.convert_tokens_to_ids(tokenized)
>         model.embeddings.word_embeddings.weight[-i, :] = model.embeddings.word_embeddings.weight[tokenized_ids].mean(axis=0)
> ```

Ok. Thank you. Is this also correct?

```
model.resize_token_embeddings(len(tokenizer))
weights = model.roberta.embeddings.word_embeddings.weight
    
# initialize new embedding weights as mean of original tokens
with torch.no_grad():
    emb = []
    for i in range(len(joined_keywords)):
        word = joined_keywords[i]
        # first & last tokens are just string start/end; don't keep
        tok_ids = tokenizer_org(word)["input_ids"][1:-1]
        tok_weights = weights[tok_ids]

        # average over tokens in original tokenization
        weight_mean = torch.mean(tok_weights, axis=0)
        emb.append(weight_mean)
    weights[-len(joined_keywords):,:] = torch.vstack(emb).requires_grad_()
```<|||||>How should I save new tokenizer to use it in downstream model?


```
tokenizer_org = tr.BertTokenizer.from_pretrained("/home/pc/bert_base_multilingual_uncased")
tokenizer.add_tokens(joined_keywords)
model = tr.BertForMaskedLM.from_pretrained("/home/pc/bert_base_multilingual_uncased", return_dict=True)

# prepare input
text = ["Replace me by any text you'd like"]
encoded_input = tokenizer(text, truncation=True, padding=True, max_length=512, return_tensors="pt")
print(encoded_input)


# add embedding params for new vocab words
model.resize_token_embeddings(len(tokenizer))
weights = model.bert.embeddings.word_embeddings.weight
    
# initialize new embedding weights as mean of original tokens
with torch.no_grad():
    emb = []
    for i in range(len(joined_keywords)):
        word = joined_keywords[i]
        # first & last tokens are just string start/end; don't keep
        tok_ids = tokenizer_org(word)["input_ids"][1:-1]
        tok_weights = weights[tok_ids]

        # average over tokens in original tokenization
        weight_mean = torch.mean(tok_weights, axis=0)
        emb.append(weight_mean)
    weights[-len(joined_keywords):,:] = torch.vstack(emb).requires_grad_()

model.to(device)

```

`trainer.save_model("/home/pc/Bert_multilingual_exp_TCM/model_mlm_exp1")`

**It saves model, config, training_args. How to save the new tokenizer as well??**<|||||>I am not sure if anyone can help to answer this here but I cannot seems to be able to find an answer from anywhere:
what exactly is the difference between "token" and a "special token"?

I understand the following:
* what is a typical token
* what is a typical special token: MASK, UNK, SEP, etc
* when do you add a token (when you want to expand your vocab)

What I don't understand is, under what kind of capacity will you want to create a new special token, any examples what we need it for and when we want to create a special token other than those default special tokens? If an example uses a special token, why can't a normal token achieve the same objective?

```
tokenizer.add_tokens(['[EOT]'], special_tokens=True)
```

And I also dont quite understand the following description in the source documentation.
what difference does it do to our model if we set add_special_tokens to False? 

```
add_special_tokens (bool, optional, defaults to True) — Whether or not to encode the sequences with the special tokens relative to their model.
```<|||||>> I am not sure if anyone can help to answer this here but I cannot seems to be able to find an answer from anywhere: what exactly is the difference between "token" and a "special token"?
> 
> I understand the following:
> 
> * what is a typical token
> * what is a typical special token: MASK, UNK, SEP, etc
> * when do you add a token (when you want to expand your vocab)
> 
> What I don't understand is, under what kind of capacity will you want to create a new special token, any examples what we need it for and when we want to create a special token other than those default special tokens? If an example uses a special token, why can't a normal token achieve the same objective?
> 
> ```
> tokenizer.add_tokens(['[EOT]'], special_tokens=True)
> ```
> 
> And I also dont quite understand the following description in the source documentation. what difference does it do to our model if we set add_special_tokens to False?
> 
> ```
> add_special_tokens (bool, optional, defaults to True) — Whether or not to encode the sequences with the special tokens relative to their model.
> ```

When you add a "special token" it will not be replaced by the "[MASK]" or replaced by a random word in the pre-training procedure.