<p>I've found the answer in the original <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">BERT git repo</a></p>
<blockquote>
<p>***** New May 31st, 2019: Whole Word Masking Models *****</p>
<p>This is a release of several new models which were the result of an
improvement the pre-processing code.</p>
<p>In the original pre-processing code, we randomly select WordPiece
tokens to mask. For example:</p>
</blockquote>
<pre><code>Input Text: 
the man jumped up , put his basket on phil ##am ##mon ' s
head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
phil [MASK] ##mon ' s head
</code></pre>
<blockquote>
<p>The new technique is called Whole Word Masking. In this case, we
always mask all of the the tokens corresponding to a word at once. The
overall masking rate remains the same.</p>
</blockquote>
<pre><code>Whole Word Masked Input: 
the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head
</code></pre>
<blockquote>
<p>The training is identical -- we still predict each masked WordPiece
token independently. The improvement comes from the fact that the
original prediction task was too 'easy' for words that had been split
into multiple WordPieces.</p>
<p>This can be enabled during data generation by passing the flag</p>
</blockquote>
<pre><code>--do_whole_word_mask=True  
</code></pre>
<blockquote>
<p>to create_pretraining_data.py.</p>
</blockquote>

CC BY-SA 4.0
2020-11-02T15:25:07.997Z
2020-11-02T15:25:07.997Z
Stephen Philip
<p>I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.</p>
<pre><code>Original : &quot;Mountain Dew is an energetic drink&quot;
Masked : &quot;[MASK] is an energetic drink&quot;
</code></pre>
<p>But BERT MLM task doesn't consider two tokens at a time for the MASK. I strongly think that there should be some sort of work around that I'm unable to find other than fine-tuning.</p>

<natural-language-processing><bert><language-model>
Is there a way to provide multiple masks to BERT in MLM task?
Stephen Philip
# Is there a way to provide multiple masks to BERT in MLM task?

(Asked by: username_0 on 2020-10-31)

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.
Original : "Mountain Dew is an energetic drink"
Masked : "[MASK] is an energetic drink"

But BERT MLM task doesn't consider two tokens at a time for the MASK. I strongly think that there should be some sort of work around that I'm unable to find other than fine-tuning.

**Tags:** natural-language-processing, bert, language-model


---

## 1 answers

### Answer by username_0 on 2020-11-02. Score: 0 | Accepted answer

I've found the answer in the original BERT git repo

***** New May 31st, 2019: Whole Word Masking Models *****
This is a release of several new models which were the result of an
improvement the pre-processing code.
In the original pre-processing code, we randomly select WordPiece
tokens to mask. For example:

Input Text: 
the man jumped up , put his basket on phil ##am ##mon ' s
head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
phil [MASK] ##mon ' s head


The new technique is called Whole Word Masking. In this case, we
always mask all of the the tokens corresponding to a word at once. The
overall masking rate remains the same.

Whole Word Masked Input: 
the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head


The training is identical -- we still predict each masked WordPiece
token independently. The improvement comes from the fact that the
original prediction task was too 'easy' for words that had been split
into multiple WordPieces.
This can be enabled during data generation by passing the flag

--do_whole_word_mask=True  


to create_pretraining_data.py.


---

