Why run_language_modelling.py does not use segment embeddings or language embeddings?
I am trying to train Bert and XLM on my own data. But I found that it seems run_language_modelling.py only tokenizes input text into ids, but it doesn't create token_type_ids or langs for Bert or XLM.

`batch_encoding = tokenizer.batch_encode_plus(lines, add_special_tokens=True, max_length=block_size)`
`self.examples = batch_encoding["input_ids"]`
As you can see from the codes, the examples only contain input_ids. 

I set breakpoints in the modeling_bert.py and modeling_xlm.py, and there is no token_type_ids or langs as a part of input to Bert or XLM. For the downstream tasks, Bert and XLM always use segment embeddings or language embeddings as a part of their input. Why don't we use them in the pretraining step?
If we do not use segment embeddings or language embeddings during pretraining, but use them in fine-tuning, shouldn't that cause bias?
Sorry if misunderstood something, but I am quite confused now.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
