Answers.Body: <ol>
  <li><p>&quot;BertBase&quot; is basically what's inside the ellipses labelled &quot;Trm&quot; (attention heads + FF) - both for BERT and GPT.</p>
  </li>
  <li><p>The main difference is that BERT pays bidirectional attention while GPT is unidirectional (from left to right) or &quot;causal&quot;.</p>
  </li>
  <li><p>T_N of GPT is the contextualized last token (embedding) which is used to calculate the next token.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-20T14:19:13.003Z
Answers.Text: So in general each `Trm` block outputs something of shape `(Batch, Seqlen, d)`, except in the output where each `T_i` selects the final sequence element?
Answers.UserDisplayName: statusfailed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-31T16:06:32.650Z
Answers.LastActivityDate: 2024-01-31T23:27:04.250Z
Answers.LastEditDate: 2024-01-31T23:27:04.250Z
Answers.UserDisplayName: Hans-Peter Stricker
Body: <p>The <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">BERT paper</a> has the following diagram (Figure 3):</p>
  <p><a href="https://i.sstatic.net/UA8hj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UA8hj.png" alt="enter image description here" /></a></p>
  <p>It's captioned &quot;Differences in pre-training model architectures&quot;.
  However, I thought the BERT architecture was just a stack of attention blocks like this (from <a href="https://huggingface.co/blog/bert-101" rel="nofollow noreferrer">huggingface</a>):</p>
  <p><a href="https://i.sstatic.net/LZja3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LZja3.png" alt="enter image description here" /></a></p>
  <p>My question: <strong>What is the BERT diagram supposed to mean?</strong></p>
  <p>My best guess is that it represents the process of predicting an output sequence from a given input sequence. In other words, in the GPT model, one first samples with a sequence length of 1 to get next token T1, then we embed T1 via E2 and sample with a sequence of length 2 to get T2, and so on.</p>
  <p>In BERT, I assume we mask all input tokens but the first, then sample the whole thing.</p>
  <p>The problem is that I don't understand what the <code>Trm</code> blocks mean, or how this relates to the BERT architecture.</p>
  
Tags: <deep-learning><transformer><bert>
Title: What does Figure 3 in the BERT paper represent?
UserDisplayName: statusfailed
ThreadText: # What does Figure 3 in the BERT paper represent?
  
  (Asked by: username_0 on 2024-01-31)
  
  The [BERT paper](https://arxiv.org/pdf/1810.04805.pdf) has the following diagram (Figure 3):
  
  [![enter image description here](https://i.sstatic.net/UA8hj.png)](https://i.sstatic.net/UA8hj.png)
  
  It's captioned "Differences in pre-training model architectures".
  However, I thought the BERT architecture was just a stack of attention blocks like this (from [huggingface](https://huggingface.co/blog/bert-101)):
  
  [![enter image description here](https://i.sstatic.net/LZja3.png)](https://i.sstatic.net/LZja3.png)
  
  My question: **What is the BERT diagram supposed to mean?**
  
  My best guess is that it represents the process of predicting an output sequence from a given input sequence. In other words, in the GPT model, one first samples with a sequence length of 1 to get next token T1, then we embed T1 via E2 and sample with a sequence of length 2 to get T2, and so on.
  
  In BERT, I assume we mask all input tokens but the first, then sample the whole thing.
  
  The problem is that I don't understand what the `Trm` blocks mean, or how this relates to the BERT architecture.
  
  **Tags:** deep-learning, transformer, bert
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-31. Score: 1
  
  1. "BertBase" is basically what's inside the ellipses labelled "Trm" (attention heads + FF) - both for BERT and GPT.
  2. The main difference is that BERT pays bidirectional attention while GPT is unidirectional (from left to right) or "causal".
  3. T\_N of GPT is the contextualized last token (embedding) which is used to calculate the next token.
  
  > **Comments:**
  > **username_0**: So in general each `Trm` block outputs something of shape `(Batch, Seqlen, d)`, except in the output where each `T_i` selects the final sequence element?  
  
  ---
  
  