XLMForMaskedLM
Hi, I am currently training a BERT model using facebook XLM framework. I use the script in this repo to convert XLM format to PyTorch format. Is it possible to implement an `XLMForMaskedLM` which is just like `BertForMaskedLM` but use XLM trained BERT instead?
You can do that using `XLNetLMHeadModel` and custom masks as shown in the [`run_generation` example](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_generation.py#L115-L121).

But note that XLNet is rather bad on short text input completions as I discussed in https://github.com/huggingface/pytorch-transformers/issues/846#issuecomment-514228565<|||||>> #846 (comment)

Thanks for your reply! However, I mean the `XLM` version of BERT, instead of XLNet. Is it also convenient to do that for XLM?

Thanks!<|||||>Oh, right! You can just use `XLMWithLMHeadModel` with an input sequence containing XLM masked token `tokenizer.mask_token` (which is `<special1>` for XLM).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
