Please see the updated code snippet. Sorry for not being explicit previously - what I am planning to do is:

collecting domain specific data.
adding new words to the vocab
then, finetuning RoBerta for MLM (say with 20% masking probability)
finally, extract word embeddings for some important words to be used in a down-stream task.

Will this not yield good word embeddings which are usable down-stream?