XLM model masked word prediction Double Language 
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am trying to generate in-context word translations. 

For instance, if the target language is french and "well" is the word to translate.
- I walked to the well. -> the translation for "well" should be "puit" 
- I am doing well. -> the translation for "well" should be "bien"

I have a simple solution inspired from https://github.com/huggingface/transformers/issues/1842#issuecomment-555734728 and https://github.com/qiang2100/BERT-LS

I am basically concatenating a sentence clone, masking the second target word and changing its language inside the "langs" tensor.

The code is something like this :

```
from transformers import XLMTokenizer, XLMWithLMHeadModel
import torch

# model
model_string = "xlm-mlm-tlm-xnli15-1024"

# load tokenizer
tokenizer = XLMTokenizer.from_pretrained(model_string)

# encode sentence with a masked token in the middle
encoded_array = tokenizer.encode(
    "That is a well. That is a " + tokenizer.mask_token + ".")
sentence = torch.tensor([encoded_array])

# Identify the masked token position
masked_index = torch.where(sentence == tokenizer.mask_token_id)[1].tolist()[0]

# Load model
model = XLMWithLMHeadModel.from_pretrained(model_string)

# Load languages
language_id_from = tokenizer.lang2id['en']  # 0
language_id_to = tokenizer.lang2id['fr']  # 0
languages_array = [language_id_from] * len(encoded_array)
languages_array[masked_index] = language_id_to

langs = torch.tensor(languages_array)
langs = langs.view(1, -1)

# Get the five top answers
result = model(input_ids=sentence, langs=langs)
prediction_scores = result[0]

result = prediction_scores[:, masked_index].topk(20).indices
result = result.tolist()[0]
print(tokenizer.decode(result))

```

I've been getting some positive results, and not so positive results (with longer sentences). For example:

"I walked to the well" : well -> Puit is the 3rd result. [easily identifiable as correct!]
"I felt well." : well -> Bien is the 6th result. [easily identifiable as correct!]
"If the sentence is too convoluted, it won't translate well." -> The results are all parts of correct answers but wrong individually. Like "correct" and "ly" in french.  [very hard to piece together answer]
"After a hard day's work like yesterday, I really like to go jump in the well to cool down." -> all results are in English for some reason.  [no french at all!]
Sentences with the animal meaning of "bat" don't work at all. I don't think it knows a bat is an animal.

My questions. 
- Should I be looking at something else than xlm; is there a better way of doing this already out there, or existing solution? I would rather avoid a cloud service like Yandex translate, as I will want to do this a lot!
- If this can be achieved with xlm, is there a way to force a particular language, and full words?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
