[Question] When pretraining a language model, can I choose to mask specific words?
Hi there,

I apologize if this is answered anywhere.  I need to pretrain a language model with some specific words masked.  I was wondering if this is currently supported?  Since language models are trained in an unsupervised way, I saw in the examples that the provided datasets don't need any labels.  However, I was thinking if it would be possible to create my own (sentence_with_masks, masked_words) pairs.  If library isn't currently supporting that, may anyone point me to a file so that I can make my modifications?  Thanks in advance!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.