Answers.Body: <p>I've found the answer in the original <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">BERT git repo</a></p>
  <blockquote>
  <p>***** New May 31st, 2019: Whole Word Masking Models *****</p>
  <p>This is a release of several new models which were the result of an
  improvement the pre-processing code.</p>
  <p>In the original pre-processing code, we randomly select WordPiece
  tokens to mask. For example:</p>
  </blockquote>
  <pre><code>Input Text: 
  the man jumped up , put his basket on phil ##am ##mon ' s
  head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
  phil [MASK] ##mon ' s head
  </code></pre>
  <blockquote>
  <p>The new technique is called Whole Word Masking. In this case, we
  always mask all of the the tokens corresponding to a word at once. The
  overall masking rate remains the same.</p>
  </blockquote>
  <pre><code>Whole Word Masked Input: 
  the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head
  </code></pre>
  <blockquote>
  <p>The training is identical -- we still predict each masked WordPiece
  token independently. The improvement comes from the fact that the
  original prediction task was too 'easy' for words that had been split
  into multiple WordPieces.</p>
  <p>This can be enabled during data generation by passing the flag</p>
  </blockquote>
  <pre><code>--do_whole_word_mask=True  
  </code></pre>
  <blockquote>
  <p>to create_pretraining_data.py.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-02T15:25:07.997Z
Answers.LastActivityDate: 2020-11-02T15:25:07.997Z
Answers.UserDisplayName: Stephen Philip
Body: <p>I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.</p>
  <pre><code>Original : &quot;Mountain Dew is an energetic drink&quot;
  Masked : &quot;[MASK] is an energetic drink&quot;
  </code></pre>
  <p>But BERT MLM task doesn't consider two tokens at a time for the MASK. I strongly think that there should be some sort of work around that I'm unable to find other than fine-tuning.</p>
  
Tags: <natural-language-processing><bert><language-model>
Title: Is there a way to provide multiple masks to BERT in MLM task?
UserDisplayName: Stephen Philip
ThreadText: # Is there a way to provide multiple masks to BERT in MLM task?
  
  (Asked by: username_0 on 2020-10-31)
  
  I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.
  Original : "Mountain Dew is an energetic drink"
  Masked : "[MASK] is an energetic drink"
  
  But BERT MLM task doesn't consider two tokens at a time for the MASK. I strongly think that there should be some sort of work around that I'm unable to find other than fine-tuning.
  
  **Tags:** natural-language-processing, bert, language-model
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-11-02. Score: 0 | Accepted answer
  
  I've found the answer in the original BERT git repo
  
  ***** New May 31st, 2019: Whole Word Masking Models *****
  This is a release of several new models which were the result of an
  improvement the pre-processing code.
  In the original pre-processing code, we randomly select WordPiece
  tokens to mask. For example:
  
  Input Text: 
  the man jumped up , put his basket on phil ##am ##mon ' s
  head Original Masked Input: [MASK] man [MASK] up , put his [MASK] on
  phil [MASK] ##mon ' s head
  
  
  The new technique is called Whole Word Masking. In this case, we
  always mask all of the the tokens corresponding to a word at once. The
  overall masking rate remains the same.
  
  Whole Word Masked Input: 
  the man [MASK] up , put his basket on [MASK] [MASK] [MASK] ' s head
  
  
  The training is identical -- we still predict each masked WordPiece
  token independently. The improvement comes from the fact that the
  original prediction task was too 'easy' for words that had been split
  into multiple WordPieces.
  This can be enabled during data generation by passing the flag
  
  --do_whole_word_mask=True  
  
  
  to create_pretraining_data.py.
  
  
  ---
  
  