When using the Hugging Face Transformer, if I set my pad_token to be a different token then the default, do I need to train my model on that new pad_token as well?
Hello,

For using the GPT2DoubleHeadsModel, I used ``<eos>`` as the last token of my text sequence, which my model will use to make predictions for multiple-choice questions.
I also set ``<pad>`` as the padding token in my tokenizer, which is different than the default.

When using the Hugging Face pre-trained GPT2DoubleHeadsModel, do I need to train the already pre-trained Transformer because the two tokens I mentioned above are new?

Thank you,

Hi @h56cho, if you add new tokens then yes, you'll need to train the model. If you just want `eos`  and `pad` token then its a good idea to use the model defaults. `eos` and  `pad`  tokens are already available in GPT2 model <|||||>Hi,

Thank you for your reply.
So what I am getting is that if I add any new "special token" onto the existing pre-trained tokenizer, I will need to re-train the pre-trained Transformer to make it learn that new special token.

 But what if I just add extra non-special tokens? for example, a word "paradox" is not included in the existing GPT-2 tokenizer, so say I add the word "paradox" to the existing set of GPT-2 vocabulary. If I didn't make any changes to the special tokens in the GPT-2 tokenizer, do I still need to train the pre-trained GPT-2 because I added a new word to a set of vocabulary?

Thanks, 