I would advise you to choose option 1



 huberl:

# Option 1
loss = F.cross_entropy(out[mask], pseudo_label[mask])



This is an option I’ve always used to train my language models with the Mask Language Modeling objective: once I mask the tokens in my sentence and pass it to the BERT encoder, I use a mask to select the representations of the masked tokens that I then project to get probabilities.
As you can also see in the original code of XLM (GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining. 1) : here 7 for the mask selection and here 6 for the loss computation.