Hugging Face GPT-2 Tokenizer
Hello,

I know that if I choose to add any new "special token" onto the pre-made GPT-2 tokenizer, and if I want to use the pre-trained GPT-2 model for my analysis, I will need to re-train the pre-trained GPT-2 to make the model learn that new special token.

But what if I just add an extra non-special token? for example, a word "paradox" is not included in the existing GPT-2 tokenizer, so say I add the word "paradox" to the existing set of GPT-2 vocabulary, like below:

```python
# load the pre-trained GPT2-tokenizer
gpt2_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# adding a new word (not special token) to the existing vocabulary,
# but I am not making any changes to the pre-assigned special tokens
gpt2_tokenizer.add_tokens("paradox")

# get the pre-trained HuggingFace GPT2DoubleHeadsModel 
model_gpt2DoubleHeadsModel = GPT2DoubleHeadsModel.from_pretrained('gpt2', output_hidden_states = True)

# resize the token embeddings
# (not sure what this function does)
model_gpt2DoubleHeadsModel.resize_token_embeddings(len(gpt2_tokenizer))
```
Given that I didn't make any changes to the special tokens in the GPT-2-tokenizer, do I still need to train the already pre-trained `GPT2DoubleHeadsModel` before I start using it,  just because I added a new word to the set of vocabulary?

Thank you,
You would need to fine-tune your GPT-2 model on a dataset containing the word, yes. The reason being that your model needs to understand in which context is the word used, what it means, etc.<|||||>Hello,

Thank you for your reply.
So I have a set of multiple choice questions, and when I use the `add_tokens` function to add whichever the tokens from the dataset that are not originally included in the GPT-2 tokenizer, the length of my GPT-2 tokenizer jumps up by ~3000 (so my dataset contains 3000 new tokens)

even if I fine tune the pre-trained GPT-2 model on a portion of my dataset (say), I won't be able to train the pre-trained model on all of the new tokens. So I am not sure how I will go about this.

What I am trying to do is though, I want the pre-trained `GPT2DoubleHeadsModel` to solve a set of multiple-choice questions, and I want to compare the error rates generated by the hidden outputs of each of the 12 layers when they are fed directly into the multiple-choice head of the model. That is, my goal is not to minimize the overall error rate of the GPT-2 model, my goal is to simply compare the error rates generated by the different layers of the model. Given this information, do I still need to fine-tune my GPT-2 model on all of the new tokens that I am adding?

Thank you,<|||||>If you don't fine-tune your model on the new tokens you're adding, then when the model sees it at inference it will be a completely unknown token, and the model probably won't handle it correctly.

If you don't have any training data to fine-tune the model, why don't you keep the tokens as they are? The GPT-2 tokenizer should be able to correctly tokenize them, as it's a byte level BPE. <|||||>Hello,

Thank you again for your reply.
I think I want to add the word as a new token mainly because I do not want the word to be treated as the mere `<unk>` token.

I am not sure what byte level BPE means, but if I do not add the new words as extra tokens, would it really work just because the tokenizer is a byte level BPE?

Thank you :s<|||||>Byte level BPEs should be able to tokenize everything. The GPT-2 tokenizer has no unknown token for that reason.

You should try to tokenize your tokens to see if some come back as unknown, but it shouldn't with GPT-2 (and RoBERTa for that matter)!<|||||>Thank you! I was able to confirm that what you mentioned in your previous post also works for my case.