Thanks for the answer @VictorSanh! I was able to adapt your implementation to work with a custom Trainer and the latest version of transformers using the approach you suggested. Nevertheless, I would certainly be interested in seeing how the mapping of BERT → MaskedBERT can be done on the fly