<ol>
<li><p>&quot;BertBase&quot; is basically what's inside the ellipses labelled &quot;Trm&quot; (attention heads + FF) - both for BERT and GPT.</p>
</li>
<li><p>The main difference is that BERT pays bidirectional attention while GPT is unidirectional (from left to right) or &quot;causal&quot;.</p>
</li>
<li><p>T_N of GPT is the contextualized last token (embedding) which is used to calculate the next token.</p>
</li>
</ol>

CC BY-SA 4.0
2024-02-20T14:19:13.003Z
So in general each `Trm` block outputs something of shape `(Batch, Seqlen, d)`, except in the output where each `T_i` selects the final sequence element?
statusfailed
CC BY-SA 4.0
2024-01-31T16:06:32.650Z
2024-01-31T23:27:04.250Z
2024-01-31T23:27:04.250Z
Hans-Peter Stricker
<p>The <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">BERT paper</a> has the following diagram (Figure 3):</p>
<p><a href="https://i.sstatic.net/UA8hj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UA8hj.png" alt="enter image description here" /></a></p>
<p>It's captioned &quot;Differences in pre-training model architectures&quot;.
However, I thought the BERT architecture was just a stack of attention blocks like this (from <a href="https://huggingface.co/blog/bert-101" rel="nofollow noreferrer">huggingface</a>):</p>
<p><a href="https://i.sstatic.net/LZja3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LZja3.png" alt="enter image description here" /></a></p>
<p>My question: <strong>What is the BERT diagram supposed to mean?</strong></p>
<p>My best guess is that it represents the process of predicting an output sequence from a given input sequence. In other words, in the GPT model, one first samples with a sequence length of 1 to get next token T1, then we embed T1 via E2 and sample with a sequence of length 2 to get T2, and so on.</p>
<p>In BERT, I assume we mask all input tokens but the first, then sample the whole thing.</p>
<p>The problem is that I don't understand what the <code>Trm</code> blocks mean, or how this relates to the BERT architecture.</p>

<deep-learning><transformer><bert>
What does Figure 3 in the BERT paper represent?
statusfailed
# What does Figure 3 in the BERT paper represent?

(Asked by: username_0 on 2024-01-31)

The [BERT paper](https://arxiv.org/pdf/1810.04805.pdf) has the following diagram (Figure 3):

[![enter image description here](https://i.sstatic.net/UA8hj.png)](https://i.sstatic.net/UA8hj.png)

It's captioned "Differences in pre-training model architectures".
However, I thought the BERT architecture was just a stack of attention blocks like this (from [huggingface](https://huggingface.co/blog/bert-101)):

[![enter image description here](https://i.sstatic.net/LZja3.png)](https://i.sstatic.net/LZja3.png)

My question: **What is the BERT diagram supposed to mean?**

My best guess is that it represents the process of predicting an output sequence from a given input sequence. In other words, in the GPT model, one first samples with a sequence length of 1 to get next token T1, then we embed T1 via E2 and sample with a sequence of length 2 to get T2, and so on.

In BERT, I assume we mask all input tokens but the first, then sample the whole thing.

The problem is that I don't understand what the `Trm` blocks mean, or how this relates to the BERT architecture.

**Tags:** deep-learning, transformer, bert


---

## 1 answers

### Answer by username_1 on 2024-01-31. Score: 1

1. "BertBase" is basically what's inside the ellipses labelled "Trm" (attention heads + FF) - both for BERT and GPT.
2. The main difference is that BERT pays bidirectional attention while GPT is unidirectional (from left to right) or "causal".
3. T\_N of GPT is the contextualized last token (embedding) which is used to calculate the next token.

> **Comments:**
> **username_0**: So in general each `Trm` block outputs something of shape `(Batch, Seqlen, d)`, except in the output where each `T_i` selects the final sequence element?  

---

