Answers.Body: <p>I've found the answer in the original <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">BERT git repo</a></p>
  <blockquote>
  <p>***** New May 31st, 2019: Whole Word Masking Models *****</p>
  <p>This is a release of several new models which were the result of an
  improvement the pre-processing code.</p>
  <p>In the original pre-processing code, we randomly select WordPiece
  tokens to mask. For example:</p>
  </blockquote>
  <pre><code>Input Text: 
  the man jumped up , put his basket on phil ##am ##mon ' s
  head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
  phil [MASK] ##mon ' s head
  </code></pre>
  <blockquote>
  <p>The new technique is called Whole Word Masking. In this case, we
  always mask all of the the tokens corresponding to a word at once. The
  overall masking rate remains the same.</p>
  </blockquote>
  <pre><code>Whole Word Masked Input: 
  the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head
  </code></pre>
  <blockquote>
  <p>The training is identical -- we still predict each masked WordPiece
  token independently. The improvement comes from the fact that the
  original prediction task was too 'easy' for words that had been split
  into multiple WordPieces.</p>
  <p>This can be enabled during data generation by passing the flag</p>
  </blockquote>
  <pre><code>--do_whole_word_mask=True  
  </code></pre>
  <blockquote>
  <p>to create_pretraining_data.py.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-02T15:25:07.997Z
Answers.LastActivityDate: 2020-11-02T15:25:07.997Z
Answers.UserDisplayName: Stephen Philip
Body: <p>I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.</p>
  <pre><code>Original : &quot;Mountain Dew is an energetic drink&quot;
  Masked : &quot;[MASK] is an energetic drink&quot;
  </code></pre>
  <p>But BERT MLM task doesn't consider two tokens at a time for the MASK. I strongly think that there should be some sort of work around that I'm unable to find other than fine-tuning.</p>
  
Tags: <natural-language-processing><bert><language-model>
Title: Is there a way to provide multiple masks to BERT in MLM task?
UserDisplayName: Stephen Philip
ThreadText: # Is there a way to provide multiple masks to BERT in MLM task?
  
  (Asked by: username_0 on 2020-10-31)
  
  I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.
  
  ```
  Original : "Mountain Dew is an energetic drink"
  Masked : "[MASK] is an energetic drink"
  
  ```
  
  But BERT MLM task doesn't consider two tokens at a time for the MASK. I strongly think that there should be some sort of work around that I'm unable to find other than fine-tuning.
  
  **Tags:** natural-language-processing, bert, language-model
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-11-02. Score: 0 | Accepted answer
  
  I've found the answer in the original [BERT git repo](https://github.com/google-research/bert)
  
  > \*\*\*\*\* New May 31st, 2019: Whole Word Masking Models \*\*\*\*\*
  >
  > This is a release of several new models which were the result of an
  > improvement the pre-processing code.
  >
  > In the original pre-processing code, we randomly select WordPiece
  > tokens to mask. For example:
  
  ```
  Input Text: 
  the man jumped up , put his basket on phil ##am ##mon ' s
  head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
  phil [MASK] ##mon ' s head
  
  ```
  
  > The new technique is called Whole Word Masking. In this case, we
  > always mask all of the the tokens corresponding to a word at once. The
  > overall masking rate remains the same.
  
  ```
  Whole Word Masked Input: 
  the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head
  
  ```
  
  > The training is identical -- we still predict each masked WordPiece
  > token independently. The improvement comes from the fact that the
  > original prediction task was too 'easy' for words that had been split
  > into multiple WordPieces.
  >
  > This can be enabled during data generation by passing the flag
  
  ```
  --do_whole_word_mask=True  
  
  ```
  
  > to create\_pretraining\_data.py.
  
  
  ---
  
  