XLMForSequenceClassification - help with zero-shot cross-lingual classification
## ❓ Questions & Help
Hi guys
According to XLM description (https://github.com/facebookresearch/XLM?fbclid=IwAR0-ZJpmWmIVfR20fA2KCHgrUU3k0cMUyx2n_V9-9C8g857-nhavrfBnVSI#pretrained-cross-lingual-language-models), we could potentially do XLNI by training in `en` dataset and do inference in other language:
```
XLMs can be used to build cross-lingual classifiers. After fine-tuning an XLM model 
on an English training corpus for instance (e.g. of sentiment analysis, natural language 
inference),  the model is still able to make accurate predictions at test time in other 
languages, for which there is very little or no training data.  This approach is usually 
referred to as "zero-shot cross-lingual classification".
```

I'm doing some tests w/ a dataset here at work, and I'm able to train the model with *XLMForSequenceClassification* and when I test it in `en`, performance looks great!
However, when I try to pass another language and do some inference, performance on other language (the so-called zero shot XLNI) does not look fine.

Here are the configurations I'm using:
```python
config = XLMConfig.from_pretrained('xlm-mlm-tlm-xnli15-1024')
config.num_labels = len(list(label_to_ix.values()))
config.n_layers = 6
tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-tlm-xnli15-1024')
model = XLMForSequenceClassification(config)
```
I trained the forward pass passing the `lang` vector with `en` language:
```python
langs = torch.full(ids.shape, fill_value = language_id, dtype = torch.long).cuda()
output = model.forward(ids.cuda(), token_type_ids=tokens.cuda(), langs=langs, head_mask=None)[0]
```
For inference, I'm passing now other languages, by using following function:
```python
def get_reply(msg, language = 'en'):
    features = prepare_features(msg, zero_pad = False)
    language_id = tokenizer.lang2id[language]
    ids = torch.tensor(features['input_ids']).unsqueeze(0).cuda()
    langs = torch.full(ids.shape, fill_value = language_id, dtype = torch.long).cuda()
    tokens = torch.tensor(features['token_type_ids']).unsqueeze(0).cuda()
    output = model.forward(ids,token_type_ids=tokens, langs=langs)[0]
    _, predicted = torch.max(output.data, 1)
    return list(label_to_ix.keys())[predicted]
```
BTW, my `prepare_features` function is quite simple, just use the `encode_plus` method and then zero pad for a given sentence size:
```python
def prepare_features(seq_1, zero_pad = False, max_seq_length = 120):
    enc_text = tokenizer.encode_plus(seq_1, add_special_tokens=True, max_length=300)
    if zero_pad:
        while len(enc_text['input_ids']) < max_seq_length:
            enc_text['input_ids'].append(0)
            enc_text['token_type_ids'].append(0)
    return enc_text
```
I've noticed in file (https://github.com/huggingface/transformers/blob/master/transformers/modeling_xlm.py) that the forward loop for `XLMModel` does add `lang` vector as sort of an offset for the embeddings:
```python
tensor = tensor + self.lang_embeddings(langs)
```
I'm wondering if I'm passing the `lang` vector in right format. I choose the model with *tlm* language model on purpose because I was guessing that it benefits of the translation language model of the XLM. Have you guys experiment already with zero-shot text classification? Any clue my inference in other languages is not working?
Any insight will be greatly appreciated!
Cheers,
Roberto
Hi, we've added some details on the multi-modal models here: https://huggingface.co/transformers/multilingual.html
And an XNLI example here: https://github.com/huggingface/transformers/tree/master/examples#xnli