It is definitely an interesting experiment to try and an interesting benchmark to have.
It would be great if I could use my own vocabulary while also leveraging the knowledge of this pretrained model. This would allow me to keep idioms and remove useless tokens from the vocab. Do you know if there is any way to change the vocabulary before finetuning? Haven’t found it in these 1 scripts.