On masked-lm labels and computing the loss
Recently I was using bert for my own project, and going through the function mask_tokens I found this line of code 
`labels[~masked_indices] = -100 # We only compute loss on masked tokens`
I wonder why we do this?
like i get the part where we do
```
    indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
    inputs[indices_replaced] = tokenizer.convert_tokens_to_ids(tokenizer.mask_token)
```

To mask the input tokens but is it necessary for labels? 
Like if I had a constant -100 as ground truth and the actual id maybe say 1000 the loss may never converge

And I've found two contradictory comments ie
`labels[~masked_indices] = -100 # We only compute loss on masked tokens`
and 
```
```(run_language_modeling)
masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):
            Labels for computing the masked language modeling loss.
            Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring)
            Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels
            in ``[0, ..., config.vocab_size]
```(modeling_bert)

```

One says loss will be computed on masked and another says will be ignored...
Could anyone please let me know about it... Thanks.
> `labels[~masked_indices] = -100 # We only compute loss on masked tokens`

-100 is the default value that gets ignored by the PyTorch `CrossEntropyLoss` method. When doing masked language modeling, we only compute the loss on the masked tokens, as is said in the comment. 

> `Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels in ``[0, ..., config.vocab_size]`

The wording could be improved here, but it means the same thing. The tokens with indices set to `-100` are seen as masked from the loss point of view, which means they will not be computed. These are not masked indices in the sense of masked language modeling.<|||||><img width="748" alt="Screenshot 2020-02-22 at 3 22 40 PM" src="https://user-images.githubusercontent.com/45225143/75090283-441e3e00-5587-11ea-92da-0411b0752fa5.png">

I opened up a collab and tried to simulate what is going on thanks to your comment I followed through...
What I understand from this is we do label -100 to ignore some values that the `torch.bernoulli` has given probability of being `False`
and the ones which are left are then again fed into `torch.bernoulli`  and given 0.8 percent to be masked ie we convert their ids as `tokenzer.mask_tokens`
and now as seen in the screenshot 80% was the chance to be masked hence both of the ids where masked and we have built our labels tensor in such a way that it will compute the masked tokens and leave -100 be as cross-entropy will simply ignore these value, in some way we assert that these values (-100) are already right and they are used in self-attention and hence dont compute their loss which also will be simply expensive
Pls review on this<|||||>@LysandreJik I agree that the wording could be improved. I'm still not totally sure about the utility of the `masked_lm_labels` argument. I presume it is mainly for ignoring common words and special tokens during the MLM pretraining?<|||||>The `masked_lm_labels` are the labels used for computing the masked language modeling loss. There are examples in the documentation showing how to use these, for example [at the end of the `BertForMaskedLM` documentation.](https://huggingface.co/transformers/model_doc/bert.html#bertformaskedlm)<|||||>@LysandreJik Isn't the example mentioned in the official documentation missing the following line of code before feeding _labels_ into model?

 `labels[inputs.input_ids != tokenizer.mask_token_id] = -100 ` 

I believe, with this we calculate the negative log likelihood, just for the masked token which is `Paris' in the given example.


<|||||>> @LysandreJik Isn't the example mentioned in the official documentation missing the following line of code before feeding _labels_ into model?
> 
> `labels[inputs.input_ids != tokenizer.mask_token_id] = -100 `
> 
> I believe, with this we calculate the negative log likelihood, just for the masked token which is `Paris' in the given example.

Yes, I was wondering why this is missing as well. There doesn't seem to be any documentation indicating that this is happening automatically before the loss is computed. And, based on some limited testing on my end I get different values for the loss when I do this.