@ rgwatwormhill I just want to add couple of hundreds tokens, representing technical terms in the text. As far as I can see from the answer here: Adding New Vocabulary Tokens to the Models · Issue #1413 · huggingface/transformers · GitHub 57, adding tokens does not require training from scratch, but it does require fine tuning the embedding layer (as the embeddings of these tokens are initialized randomly). So I was wondering if the embedding layer is tuned automatically as part of the training process. From your answer it seems like it is indeed the case?