Answers.Body: <p>This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. It is a convention that the input sequences are zero-padded, but in theory, it does not have to be so. In the Huggingface implementation, you use a different tokenizer that would pad the sequences with different numbers and still get valid masking.</p>
  <p>You are right that you can infer the mask from the input IDs at the very beginning (if you know the pad ID), but you need to explicitly use the mask in every single layer. Each layer returns a 3D tensor of floats from which you cannot say what the padded positions are, you need to have the explicit mask when calling the next layer. I guess that having the mask everywhere makes the API more consistent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-12T08:50:34.830Z
Answers.LastActivityDate: 2021-10-12T08:50:34.830Z
Answers.UserDisplayName: Jindřich
Answers.Body: <p>In huggingface implementation, they have made the Bert implementation generic. So that It could be used as both an encoder or an decoder.</p>
  <p>Example, for a transformer to act as decoder, only two things that's required is :</p>
  <ol>
  <li>masking future tokens.</li>
  <li>Also do cross attention based on &quot;supplied encoder representations&quot;.</li>
  </ol>
  <p>Check the code for  transformers.BertLMHeadModel where Bert is used as decoder while transformers.BertforMaskedLM uses it as our regular encoder.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T03:25:38.987Z
Answers.LastActivityDate: 2023-04-20T03:25:38.987Z
Answers.UserDisplayName: bad programmer
Answers.Body: <p>It makes sense to me that BERT does not require attention mask. BERT is a bi-directional encoder. Each word in a sequence is allowed to &quot;attend&quot; other words from both left and right sides. Attention mask would only make sense if the encoder is uni-directional, or in case of a decoder, where each word is only allowed to attend the words before it.</p>
  <p>I also think that this is related to Huggingface implementation. BERT is an encoder. The pytorch implementation of an encoder can be found <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html" rel="nofollow noreferrer">here</a>. As you can see in the &quot;forward&quot; function there, the argument &quot;mask&quot; (in this case it refers to attention mask) is only optional.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T20:40:17.917Z
Answers.LastActivityDate: 2023-04-03T20:40:17.917Z
Answers.UserDisplayName: hoang tran
Body: <p>I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial <a href="https://huggingface.co/transformers/glossary.html" rel="nofollow noreferrer">https://huggingface.co/transformers/glossary.html</a>:</p>
  <pre><code>from transformers import BertTokenizer
  tokenizer = BertTokenizer.from_pretrained(&quot;bert-base-cased&quot;)
  
  sequence_a = &quot;This is a short sequence.&quot;
  sequence_b = &quot;This is a rather long sequence. It is at least longer than the sequence A.&quot;
  
  encoded_sequence_a = tokenizer(sequence_a)[&quot;input_ids&quot;]
  encoded_sequence_b = tokenizer(sequence_b)[&quot;input_ids&quot;]
  
  padded_sequences = tokenizer([sequence_a, sequence_b], padding=True)
  </code></pre>
  <p>Output of padded sequences input ids:</p>
  <pre><code>padded_sequences[&quot;input_ids&quot;]
  
  [[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]
  </code></pre>
  <p>Output of padded sequence attention mask:</p>
  <pre><code>padded_sequences[&quot;attention_mask&quot;]
  [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]
  </code></pre>
  <p>In the tutorial, it clearly states that an attention mask is needed to tell the model (BERT) which input ids need to be attended and which not (if an element in attention mask is 1 then the model will pay attention to that index, if it is 0 then model will not pay attention).</p>
  <p>The thing I don't get is: why does BERT have an attention mask in the first place? Doesn't model need only input ids because you can clearly see that attention_mask has zeros on the same indices as the input_ids. Why does the model need to have an additional layer of difficulty added?</p>
  <p>I know that BERT was created in google's &quot;super duper laboratories&quot;, so I think the creators had something in their minds and had a strong reason for creating an attention mask as a part of the input.</p>
  
Tags: <deep-learning><natural-language-processing><attention><bert>
Title: Isn't attention mask for BERT model useless?
UserDisplayName: artas2357
ThreadText: # Isn't attention mask for BERT model useless?
  
  (Asked by: username_0 on 2021-07-25)
  
  I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial <https://huggingface.co/transformers/glossary.html>:
  
  ```
  from transformers import BertTokenizer
  tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
  
  sequence_a = "This is a short sequence."
  sequence_b = "This is a rather long sequence. It is at least longer than the sequence A."
  
  encoded_sequence_a = tokenizer(sequence_a)["input_ids"]
  encoded_sequence_b = tokenizer(sequence_b)["input_ids"]
  
  padded_sequences = tokenizer([sequence_a, sequence_b], padding=True)
  
  ```
  
  Output of padded sequences input ids:
  
  ```
  padded_sequences["input_ids"]
  
  [[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]
  
  ```
  
  Output of padded sequence attention mask:
  
  ```
  padded_sequences["attention_mask"]
  [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]
  
  ```
  
  In the tutorial, it clearly states that an attention mask is needed to tell the model (BERT) which input ids need to be attended and which not (if an element in attention mask is 1 then the model will pay attention to that index, if it is 0 then model will not pay attention).
  
  The thing I don't get is: why does BERT have an attention mask in the first place? Doesn't model need only input ids because you can clearly see that attention\_mask has zeros on the same indices as the input\_ids. Why does the model need to have an additional layer of difficulty added?
  
  I know that BERT was created in google's "super duper laboratories", so I think the creators had something in their minds and had a strong reason for creating an attention mask as a part of the input.
  
  **Tags:** deep-learning, natural-language-processing, attention, bert
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2021-10-12. Score: 7
  
  This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. It is a convention that the input sequences are zero-padded, but in theory, it does not have to be so. In the Huggingface implementation, you use a different tokenizer that would pad the sequences with different numbers and still get valid masking.
  
  You are right that you can infer the mask from the input IDs at the very beginning (if you know the pad ID), but you need to explicitly use the mask in every single layer. Each layer returns a 3D tensor of floats from which you cannot say what the padded positions are, you need to have the explicit mask when calling the next layer. I guess that having the mask everywhere makes the API more consistent.
  
  
  ---
  
  ### Answer by username_2 on 2023-04-20. Score: 0
  
  In huggingface implementation, they have made the Bert implementation generic. So that It could be used as both an encoder or an decoder.
  
  Example, for a transformer to act as decoder, only two things that's required is :
  
  1. masking future tokens.
  2. Also do cross attention based on "supplied encoder representations".
  
  Check the code for transformers.BertLMHeadModel where Bert is used as decoder while transformers.BertforMaskedLM uses it as our regular encoder.
  
  
  ---
  
  ### Answer by username_3 on 2023-04-03. Score: 0
  
  It makes sense to me that BERT does not require attention mask. BERT is a bi-directional encoder. Each word in a sequence is allowed to "attend" other words from both left and right sides. Attention mask would only make sense if the encoder is uni-directional, or in case of a decoder, where each word is only allowed to attend the words before it.
  
  I also think that this is related to Huggingface implementation. BERT is an encoder. The pytorch implementation of an encoder can be found [here](https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html). As you can see in the "forward" function there, the argument "mask" (in this case it refers to attention mask) is only optional.
  
  
  ---
  
  