Thank you for the answer. I also think that to actually achieve better results, the model needs to be trained from scratch (with specific vocab). But I wanted to give it a try. Do you have any experience with how long it takes to train from scratch to get similar results to pre-trained models?
I know that with
for param in model.base_model.parameters():
    param.requires_grad = False

one can gradually unfreeze/freeze some layers but don’t know how to implement that with the huggingface trainer, since I do not manually define the loops for the epochs and cannot choose and adapt the number of layers to freeze in each epoch.