<p>Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT:</p>
<ul>
<li>First, a <code>[SEP]</code> token is put between them</li>
<li>Second, a learned embedding <span class="math-container">$E_A$</span> is added to every token of the first sentence, and another learned vector <span class="math-container">$E_B$</span> to every token of the second one</li>
</ul>
<p>That is, there are just <strong>two</strong> possible &quot;segment embeddings&quot;: <span class="math-container">$E_A$</span> and <span class="math-container">$E_B$</span>.</p>
<p>Positional embeddings are learned vectors for every possible position between 0 and 512-1. Transformers don't have a sequential nature as recurrent neural networks, so some information about the order of the input is needed; if you disregard this, your output will be permutation-invariant.</p>

CC BY-SA 4.0
2023-10-05T15:37:26.740Z
Segment embeddings aren't concatenated to the sentence, they are added, just like positional encodings
Lock-not-gimbal
CC BY-SA 4.0
2023-10-09T17:31:54.353Z
Thanks @Lock-not-gimbal, you're right. Edited
finiteautomata
CC BY-SA 4.0
2020-02-17T13:44:26.610Z
2023-10-09T17:31:40.073Z
2023-10-09T17:31:40.073Z
finiteautomata
<p>These embeddings are nothing more than token embeddings.</p>

<p>You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.</p>

CC BY-SA 4.0
2021-07-10T22:06:46.043Z
Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/127410/discussion-on-answer-by-soloice-what-are-the-segment-embeddings-and-position-emb).
nbro
CC BY-SA 4.0
2019-02-17T09:16:51.480Z
2019-02-17T09:16:51.480Z
soloice
<p><a href="https://i.sstatic.net/thmqC.png" rel="noreferrer"><img src="https://i.sstatic.net/thmqC.png" alt="enter image description here"></a></p>

<p>They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.</p>

<p>ELMo paper - <a href="https://arxiv.org/pdf/1802.05365.pdf" rel="noreferrer">https://arxiv.org/pdf/1802.05365.pdf</a></p>

<p>BERT paper - <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="noreferrer">https://arxiv.org/pdf/1810.04805.pdf</a></p>

<machine-learning><deep-learning><natural-language-processing><bert>
What are the segment embeddings and position embeddings in BERT?
Skinish
# What are the segment embeddings and position embeddings in BERT?

(Asked by: username_0 on 2019-01-22)

[![enter image description here](https://i.sstatic.net/thmqC.png)](https://i.sstatic.net/thmqC.png)

They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.

ELMo paper - <https://arxiv.org/pdf/1802.05365.pdf>

BERT paper - <https://arxiv.org/pdf/1810.04805.pdf>

**Tags:** machine-learning, deep-learning, natural-language-processing, bert


---

## 2 answers

### Answer by username_3 on 2019-02-17. Score: 3 | Accepted answer

These embeddings are nothing more than token embeddings.

You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.

> **Comments:**
> **username_4**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/127410/discussion-on-answer-by-username_3-what-are-the-segment-embeddings-and-position-emb).  

---

### Answer by username_1 on 2020-02-17. Score: 4

Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT:

* First, a `[SEP]` token is put between them
* Second, a learned embedding $E\_A$ is added to every token of the first sentence, and another learned vector $E\_B$ to every token of the second one

That is, there are just **two** possible "segment embeddings": $E\_A$ and $E\_B$.

Positional embeddings are learned vectors for every possible position between 0 and 512-1. Transformers don't have a sequential nature as recurrent neural networks, so some information about the order of the input is needed; if you disregard this, your output will be permutation-invariant.

> **Comments:**
> **username_2**: Segment embeddings aren't concatenated to the sentence, they are added, just like positional encodings  
> **username_1**: Thanks @username_2, you're right. Edited  

---

