we mask them using the special_tokens_mask passed to DataCollatorForLanguageModeling .

Ah that makes sense.
Is it problematic that there will also be some sequences that do not start with a BOS token or end in an EOS token?
Edit: I think this is a bad question because it looks like all special tokens are masked, regardless of location.
New question: Does MLM not require special tokens at all? If so, why not tokenize without them?
Edit 2: It looks like the special_tokens_mask prevents special tokens from being replaced with a [MASK] token, but I don’t think they are masked (using the attention_mask) when they are used as inputs to the model. Am I wrong?