How to get masked word prediction probabilities
Original sentence: i love apples. there are a lot of fruits in the world that i like, but apples would be my favorite fruit.
Masked sentence: i love apples . there are a lot of fruits in the world that i [MASK] , but apples would be my favorite fruit .

When I run through the pytorch version of bert, I get the following representations of probabilities:

Best predicted word: ['love'] tensor(12.7276, grad_fn=)
Other words along with their probabilities:
['like'] tensor(10.2872, grad_fn=)
['miss'] tensor(8.8226, grad_fn=)
['know'] tensor(8.5971, grad_fn=)
['am'] tensor(7.9407, grad_fn=)
['hate'] tensor(7.9209, grad_fn=)
['mean'] tensor(7.8873, grad_fn=)
['enjoy'] tensor(7.8813, grad_fn=)
['want'] tensor(7.6885, grad_fn=)
['prefer'] tensor(7.5712, grad_fn=)

I am quite sure that this does not mean that probability for word "love" is proportional to 12.7276 and for word "like" is 10.2872.
I also know that the summ of all func(this number) thought the whole vocabulary is 1. But I do not know what the func is?

Thanks
I'm interested in an answer, too. A score/probability would help to select the best word for a masked token.<|||||>You are looking for the softmax function: https://pytorch.org/docs/stable/nn.html?highlight=softmax#torch.nn.functional.softmax<|||||>Thanks Thomas, I'll give it a try.<|||||>Thanks,
So you say that for score x1 (where all the scores are x1,x2,.. xn) :
probability_x1 = (exp(^x1)/(exp(^x1) + exp(^x2)  +  .. exp(^xn))

<|||||>@Oxi84 could you share how you obtained the masked word probabilities? I have been trying to do that on my custom data. That is, I want to pretrain my own model and then do masked word prediction on new data.<|||||>@rvoak The quickstart guide [here](https://github.com/huggingface/pytorch-transformers/blob/master/docs/source/quickstart.md#bert-example) shows a nice example of how to do masked word prediction. 
Replace 
```
# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
```
with something like this (e.g. if you want the top k predicted tokens):

```
top_k = 10
probs = torch.nn.functional.softmax(predictions[0, mask_idx], dim=-1)
top_k_weights, top_k_indices = torch.topk(probs, top_k, sorted=True)

for i, pred_idx in enumerate(top_k_indicies):
    predicted_token = tokenizer.convert_ids_to_tokens([pred_idx])[0]
    token_weight = top_k_weights[i]
```

<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> @rvoak The quickstart guide [here](https://github.com/huggingface/pytorch-transformers/blob/master/docs/source/quickstart.md#bert-example) shows a nice example of how to do masked word prediction.
> Replace
> 
> ```
> # confirm we were able to predict 'henson'
> predicted_index = torch.argmax(predictions[0, masked_index]).item()
> predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
> ```
> 
> with something like this (e.g. if you want the top k predicted tokens):
> 
> ```
> top_k = 10
> probs = torch.nn.functional.softmax(predictions[0, mask_idx], dim=-1)
> top_k_weights, top_k_indices = torch.topk(probs, top_k, sorted=True)
> 
> for i, pred_idx in enumerate(top_k_indicies):
>     predicted_token = tokenizer.convert_ids_to_tokens([pred_idx])[0]
>     token_weight = top_k_weights[i]
> ```

very good example thanks! is there a version for RoBERTa and other models?<|||||>Hi @yuchenlin, you can use the recently added `fill-mask` pipeline to do so:

```py
>>> from transformers import pipeline
>>> nlp = pipeline("fill-mask", model="roberta-base")
>>> nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")
[
    {'sequence': "<s> This is the best thing I've done in my life.</s>", 'score': 0.8024354577064514, 'token': 626}, 
    {'sequence': "<s> This is the best thing I've heard in my life.</s>", 'score': 0.031355079263448715, 'token': 1317}, 
    {'sequence': "<s> This is the best thing I've learned in my life.</s>", 'score': 0.027319395914673805, 'token': 2435}, 
    {'sequence': "<s> This is the best thing I've seen in my life.</s>", 'score': 0.026892054826021194, 'token': 450}, 
    {'sequence': "<s> This is the best thing I've experienced in my life.</s>", 'score': 0.02160099521279335, 'token': 2984}
]
```

We're in the process of adding example usage for common tasks (question answering, sequence classification, mask filling etc), you can follow the progress in https://github.com/huggingface/transformers/pull/2850. There already is an example for mask filling.<|||||>Hey @LysandreJik, does the fill-mask also support whole word mask prediction, or does it only work on subword level?<|||||>> Hi @yuchenlin, you can use the recently added `fill-mask` pipeline to do so:
> 
> ```python
> >>> from transformers import pipeline
> >>> nlp = pipeline("fill-mask", model="roberta-base")
> >>> nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")
> [
>     {'sequence': "<s> This is the best thing I've done in my life.</s>", 'score': 0.8024354577064514, 'token': 626}, 
>     {'sequence': "<s> This is the best thing I've heard in my life.</s>", 'score': 0.031355079263448715, 'token': 1317}, 
>     {'sequence': "<s> This is the best thing I've learned in my life.</s>", 'score': 0.027319395914673805, 'token': 2435}, 
>     {'sequence': "<s> This is the best thing I've seen in my life.</s>", 'score': 0.026892054826021194, 'token': 450}, 
>     {'sequence': "<s> This is the best thing I've experienced in my life.</s>", 'score': 0.02160099521279335, 'token': 2984}
> ]
> ```
> 
> We're in the process of adding example usage for common tasks (question answering, sequence classification, mask filling etc), you can follow the progress in #2850. There already is an example for mask filling.

Is it possible to give this an input word for the mask and get probabilities back for that specific word?<|||||>Also is it possible to request the top N sentences rather than the default returned?

Edit: Never mind on this specific question! I found out by setting:

`nlp.topk = 20`

before doing:

` nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")`

It now returns 20.<|||||>Sure, you can do that using the recently added `targets` (in `v3.1.0`):

```py
>>> from transformers import pipeline
>>> nlp = pipeline("fill-mask", model="roberta-base")
>>> nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.", targets=[' experienced'])
[
    {
        'sequence': "<s>This is the best thing I've experienced in my life.</s>", 
        'score': 0.022622672840952873, 
        'token': 2984, 
        'token_str': 'Ġexperienced'
    }
]
```

Please note the space before the word, because we're using the [RoBERTa tokenizer](https://huggingface.co/transformers/model_doc/roberta.html#robertatokenizer) which is a Byte-level BPE tokenizer that has a different behaviour according to the spaces before tokens.<|||||>@LysandreJik So very helpful! Thank you so much!<|||||>@LysandreJik If a word is at the start of a sentence, should it also have a space in front of it?:

```
nlp(f"{nlp.tokenizer.mask_token} talk about the rules of the game first.", targets=[' We\'ll'])
```
Which gives me:
```
The specified target token ` We'll` does not exist in the model vocabulary. Replacing with `ĠWe`.
[{'sequence': '<s> We talk about the rules of the game first</s>', 'score': 8.493712812196463e-06, 'token': 166, 'token_str': 'ĠWe'}]
```

Or

```
nlp(f"{nlp.tokenizer.mask_token} talk about the rules of the game first.", targets=['We\'ll'])
```
Which gives me:
```
The specified target token `We'll` does not exist in the model vocabulary. Replacing with `We`.
[{'sequence': '<s>We talk about the rules of the game first</s>', 'score': 0.12082401663064957, 'token': 170, 'token_str': 'We'}]
```<|||||>How to predict a word that is seperated into several tokens. For example, DOTA2(name for a popular game)?