Answers.Body: <p>In BERT, the context is considered bidirectional because there is no (causal) masking applied to the input sequence, allowing the transformer to process the entire sequence simultaneously. In contrast, GPT employs <strong>causal masking</strong> to prevent the model from accessing future tokens in the sequence during both training and inference. The causal mask is defined as follows:
  <span class="math-container">$$
  M_{i,j} = 
  \begin{cases}
  0 &amp; \text{if $j \leq i$}\\
  -\infty &amp; \text{if $j &gt; i$}
  \end{cases}
  $$</span>
  The self-attention mechanism is then updated as:
  <span class="math-container">$$
  Y = \operatorname{softmax}\left( \frac{QK^T}{\sqrt{d_k}} + M \right)V
  $$</span></p>
  <p>This masking is applied when computing the key (<span class="math-container">$K$</span>), query (<span class="math-container">$Q$</span>), and value (<span class="math-container">$V$</span>) matrices during the self-attention mechanism. <strong>The causal mask ensures that each position in the sequence can only attend to previous positions</strong>, thereby preserving the autoregressive property of the model. Specifically, this masking matrix is used to mask out (<em>i.e.</em>, set to negative infinity) the attention scores for positions that should not be attended to. This ensures that the softmax function only considers the allowed positions, effectively masking out future tokens.</p>
  <p>In contrast, <strong>BERT applies no causal masking</strong> but does employ padding masking. Padding masking ensures that padding tokens, which do not carry meaningful information, are not attended to during attention computation. In other words, BERT can attend to every token in the input sequence, while GPT can only attend to tokens up to <span class="math-container">$i−1$</span> when predicting token <span class="math-container">$i$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T01:22:48.053Z
Answers.Text: @fred Thank you for your answer. I understand that the difference between BERT and GPT can be represented by masking, which defines the scope of attention.
Answers.UserDisplayName: XYJ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T23:11:20.447Z
Answers.LastActivityDate: 2024-07-10T23:11:20.447Z
Answers.UserDisplayName: frad
Body: <p>I have read the paper &quot;BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding&quot; by Jacob Devlin et al. (2018) and &quot;Improving Language Understanding by Generative Pre-training&quot; by Alec Radford et al. (2018).</p>
  <p>I have understand that the largest difference between BERT and GPT is the direction of the context. BERT is bidirectional, while GPT is unidirectional. I have understand the difference visually thorugh Figure 3 of the BERT paper.
  However, I have question about how the difference is achieved in more detailed level, such as implementation level. I have understand that the shortest way is to look at the code, but I am wondering if there is any explanation in the paper.
  Especially, I am wondering how multiple outputs from previous transformer layers are combined as the input of the next layer in BERT (or so does GPT).</p>
  <p>I have also looked at the paper &quot;Attention is All You Need&quot; by Vaswani et al. (2017) and found the following equation
  <span class="math-container">$$ Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V $$</span>
  , but I am not sure about where does it implies the unidirectional context in the transformer architecture. (I think original transformer is unidirectional.)</p>
  <p>I would appreciate it if you could explain how the bidirectional context is achieved in BERT.</p>
  
Tags: <transformer><attention><gpt><bert><language-model>
Title: How is the bidirectional context achieved in BERT?
UserDisplayName: XYJ
ThreadText: # How is the bidirectional context achieved in BERT?
  
  (Asked by: username_0 on 2024-06-23)
  
  I have read the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by Jacob Devlin et al. (2018) and "Improving Language Understanding by Generative Pre-training" by Alec Radford et al. (2018).
  
  I have understand that the largest difference between BERT and GPT is the direction of the context. BERT is bidirectional, while GPT is unidirectional. I have understand the difference visually thorugh Figure 3 of the BERT paper.
  However, I have question about how the difference is achieved in more detailed level, such as implementation level. I have understand that the shortest way is to look at the code, but I am wondering if there is any explanation in the paper.
  Especially, I am wondering how multiple outputs from previous transformer layers are combined as the input of the next layer in BERT (or so does GPT).
  
  I have also looked at the paper "Attention is All You Need" by Vaswani et al. (2017) and found the following equation
  $$ Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d\_k}})V $$
  , but I am not sure about where does it implies the unidirectional context in the transformer architecture. (I think original transformer is unidirectional.)
  
  I would appreciate it if you could explain how the bidirectional context is achieved in BERT.
  
  **Tags:** transformer, attention, gpt, bert, language-model
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-10. Score: 0 | Accepted answer
  
  In BERT, the context is considered bidirectional because there is no (causal) masking applied to the input sequence, allowing the transformer to process the entire sequence simultaneously. In contrast, GPT employs **causal masking** to prevent the model from accessing future tokens in the sequence during both training and inference. The causal mask is defined as follows:
  $$
  M\_{i,j} =
  \begin{cases}
  0 & \text{if $j \leq i$}\\
  -\infty & \text{if $j > i$}
  \end{cases}
  $$
  The self-attention mechanism is then updated as:
  $$
  Y = \operatorname{softmax}\left( \frac{QK^T}{\sqrt{d\_k}} + M \right)V
  $$
  
  This masking is applied when computing the key ($K$), query ($Q$), and value ($V$) matrices during the self-attention mechanism. **The causal mask ensures that each position in the sequence can only attend to previous positions**, thereby preserving the autoregressive property of the model. Specifically, this masking matrix is used to mask out (*i.e.*, set to negative infinity) the attention scores for positions that should not be attended to. This ensures that the softmax function only considers the allowed positions, effectively masking out future tokens.
  
  In contrast, **BERT applies no causal masking** but does employ padding masking. Padding masking ensures that padding tokens, which do not carry meaningful information, are not attended to during attention computation. In other words, BERT can attend to every token in the input sequence, while GPT can only attend to tokens up to $i−1$ when predicting token $i$.
  
  > **Comments:**
  > **username_0**: @fred Thank you for your answer. I understand that the difference between BERT and GPT can be represented by masking, which defines the scope of attention.  
  
  ---
  
  