Mask probability in run_lm_finetuning.py
   Hi:

 I don't understand why 0.5 is used when replacing masked input tokens with a random word. I think the probability should be 0.1? Or, the positions replaced with [MASK] shall already be stripped out before using 0.5. I think it is a small bug here? Thanks!

     # 80% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK])
    indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
    inputs[indices_replaced] = tokenizer.convert_tokens_to_ids(tokenizer.mask_token)

    # 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random]

    # The rest of the time (10% of the time) we keep the masked input tokens unchanged
    return inputs, labels

Best Regards,
Qi

The lines:

```
indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
```
and
```
indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
```
make it that `indices_random` has a 50% chance of being true when `indices_replaced`, which has a 80% chance of being active, is not active, which is: 100% - 80% = 20%. 50% of 20% is 10%, so the chance that indices_random is true \*is\* 10%<|||||>Thanks for the reply. Yes. You are right! I missed the  ~indices_replaced when reading the code. Thanks!