Implementation of 15% words masking in pretraining
In the BERT paper, they randomly mask 15% words for pretraining, and that's exactly what they do in the TF version. 

https://github.com/google-research/bert/blob/0fce551b55caabcfba52c61e18f34b541aef186a/create_pretraining_data.py#L342

However, the implementation here is a little bit different, instead of randomly select 15% tokens, it assigns a probability of 15% to each token, that is, each token has a probability of 15% to be masked. That means, each time we might have less than or more than 15% tokens masked.

So, is it correct to mask tokens with the expectation of 0.15 rather than fixed 15%?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/f9cde97b313c3218e1b29ea73a42414dfefadb40/examples/lm_finetuning/simple_lm_finetuning.py#L276-L301
It should be fine. A bit of randomness in the pre-processing of the inputs is never bad when training a deep learning model.<|||||>> It should be fine. A bit of randomness in the pre-processing of the inputs is never bad when training a deep learning model.

I found the same problem that the implementation is different from tensorflow. But the key point is not the fixed 15% prob of all the token. If  we use the implementation of pytorch will produce two extreme case especially for short sentences like article title,usually 10-20 characters.
case 1. sentence with too much '[MASK]'
case 2. sentence with none '[MASK]'
both case1 and case2 would cause the drop of performance. case 1 make the model difficult to predict and case2 would not produce the loss.
Given a corpus with an average sentence length of 10. The implementation of tensorflow would generate 1 '[MASK]' for the sentences, but the implementation of pytorch would have :
0.85^10                      = 0.19 to generate 0 '[MASK]'
0.15 * 0.85^9 * 10      =0.34 to generate 1 '[MASK]'
0.15^2 * 0.85^8  * 45 =0.27 to generate 2 '[MASK]'
0.15^2 * 0.85^7 * 120 =0.12 to generate 3 '[MAKS]'
... 

If we roughly consider the sentence with 15% '[MASK]' is  appropriate, only 1/2 '[MASK]' is useful for training models. So only 0.34 + 0.27  = 0.61 training case is useful.

And we found it is this is a very serious problem for short text.



<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
