Can pytorch-transformers be used to get XLM sentence embeddings for multiple languages?
## ❓ Questions & Help

I tried to create a class to get the XLM sentence embeddings in multiple languages.
```
class XLMSentenceEmbeddings(pt.XLMPreTrainedModel):
    def __init__(self, config):
        super(XLMSentenceEmbeddings, self).__init__(config)
        self.transformer = pt.XLMModel(config)
    def forward(self, input_ids, lengths=None, position_ids=None, langs=None, token_type_ids=None,
                attention_mask=None, cache=None, labels=None, head_mask=None):
        transformer_outputs = self.transformer(input_ids, lengths=lengths, position_ids=position_ids, token_type_ids=token_type_ids, langs=langs, attention_mask=attention_mask, cache=cache, head_mask=head_mask)
    return transformer_outputs[0][:, 0, :]
```

But if I try this code I get the same result if I set the language ids to Dutch or if I don't.
```
tokenizer = pt.XLMTokenizer.from_pretrained('xlm-mlm-100-1280')
xlm_model = XLMSentenceEmbeddings.from_pretrained('xlm-mlm-100-1280')

sentence = 'een nederlandstalige zin'
lang = 'nl'
input_ids = torch.tensor(tokenizer.encode(sentence, lang=lang)).unsqueeze(0)
with torch.no_grad():
    xlm_model.eval()
    output_without_lang = xlm_model(input_ids).numpy()
    output_with_lang = xlm_model(input_ids, langs=xlm_model.transformer.config.lang2id[lang]*torch.ones(input_ids.size(), dtype=int)).numpy()

np.sum(output_without_lang - output_with_lang)
```
The output of this sum at the end always returns zero.

If I change the config to use the language embeddings like so: `config.use_lang_emb = True` my results are random each time which seems to suggest the embeddings are not included in the model. Should I change a different config? Or is the only way to get the sentence embeddings to train the model again?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
