Answers.Body: <p>Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT:</p>
  <ul>
  <li>First, a <code>[SEP]</code> token is put between them</li>
  <li>Second, a learned embedding <span class="math-container">$E_A$</span> is added to every token of the first sentence, and another learned vector <span class="math-container">$E_B$</span> to every token of the second one</li>
  </ul>
  <p>That is, there are just <strong>two</strong> possible &quot;segment embeddings&quot;: <span class="math-container">$E_A$</span> and <span class="math-container">$E_B$</span>.</p>
  <p>Positional embeddings are learned vectors for every possible position between 0 and 512-1. Transformers don't have a sequential nature as recurrent neural networks, so some information about the order of the input is needed; if you disregard this, your output will be permutation-invariant.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-05T15:37:26.740Z
Answers.Text: Segment embeddings aren't concatenated to the sentence, they are added, just like positional encodings
Answers.UserDisplayName: Lock-not-gimbal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-09T17:31:54.353Z
Answers.Text: Thanks @Lock-not-gimbal, you're right. Edited
Answers.UserDisplayName: finiteautomata
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-17T13:44:26.610Z
Answers.LastActivityDate: 2023-10-09T17:31:40.073Z
Answers.LastEditDate: 2023-10-09T17:31:40.073Z
Answers.UserDisplayName: finiteautomata
Answers.Body: <p>These embeddings are nothing more than token embeddings.</p>
  
  <p>You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-10T22:06:46.043Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/127410/discussion-on-answer-by-soloice-what-are-the-segment-embeddings-and-position-emb).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-17T09:16:51.480Z
Answers.LastActivityDate: 2019-02-17T09:16:51.480Z
Answers.UserDisplayName: soloice
Body: <p><a href="https://i.sstatic.net/thmqC.png" rel="noreferrer"><img src="https://i.sstatic.net/thmqC.png" alt="enter image description here"></a></p>
  
  <p>They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.</p>
  
  <p>ELMo paper - <a href="https://arxiv.org/pdf/1802.05365.pdf" rel="noreferrer">https://arxiv.org/pdf/1802.05365.pdf</a></p>
  
  <p>BERT paper - <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="noreferrer">https://arxiv.org/pdf/1810.04805.pdf</a></p>
  
Tags: <machine-learning><deep-learning><natural-language-processing><bert>
Title: What are the segment embeddings and position embeddings in BERT?
UserDisplayName: Skinish
ThreadText: # What are the segment embeddings and position embeddings in BERT?
  
  (Asked by: username_0 on 2019-01-22)
  
  [![enter image description here](https://i.sstatic.net/thmqC.png)](https://i.sstatic.net/thmqC.png)
  
  They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.
  
  ELMo paper - <https://arxiv.org/pdf/1802.05365.pdf>
  
  BERT paper - <https://arxiv.org/pdf/1810.04805.pdf>
  
  **Tags:** machine-learning, deep-learning, natural-language-processing, bert
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-02-17. Score: 3 | Accepted answer
  
  These embeddings are nothing more than token embeddings.
  
  You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.
  
  > **Comments:**
  > **username_4**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/127410/discussion-on-answer-by-username_3-what-are-the-segment-embeddings-and-position-emb).  
  
  ---
  
  ### Answer by username_1 on 2020-02-17. Score: 4
  
  Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT:
  
  * First, a `[SEP]` token is put between them
  * Second, a learned embedding $E\_A$ is added to every token of the first sentence, and another learned vector $E\_B$ to every token of the second one
  
  That is, there are just **two** possible "segment embeddings": $E\_A$ and $E\_B$.
  
  Positional embeddings are learned vectors for every possible position between 0 and 512-1. Transformers don't have a sequential nature as recurrent neural networks, so some information about the order of the input is needed; if you disregard this, your output will be permutation-invariant.
  
  > **Comments:**
  > **username_2**: Segment embeddings aren't concatenated to the sentence, they are added, just like positional encodings  
  > **username_1**: Thanks @username_2, you're right. Edited  
  
  ---
  
  