Answers.Body: <p>While an input with <span class="math-container">$n$</span> tokens generates an output with <span class="math-container">$n$</span> vectors, there is a lot of cross mixing of information. One word, or even one word sense, may have many different representations.  The embeddings are very sensitive to other words in the sentence, even when the sense of the word does not change. To measure how the embeddings change, we could use the standard deviation of the variations in the vector entries. For example, the difference between the word vectors for <em>run</em> in &quot;Run fast&quot; and &quot;Run fast.&quot; (notice the period) is 0.24, which is only about half the difference among random uses of the word <em>run</em>.</p>
  <p>The authors of the <a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">BERT paper</a> explored the possibility of generating contextualized word embeddings using BERT. Table 7 in the BERT paper explains different ways they tried for extracting word representations that would work well in a named entity extraction task. They concluded that the weighted sum of the last four hidden layers provides good contextualized word representations. However, there are still many other choices to make when it comes to developing contextualized word embeddings, such as how to go from the wordpieces that BERT uses to a single word vector. If you're looking to learn more about BERT-related topics, I found the paper on <a href="https://doi.org/10.1162/tacl_a_00349" rel="nofollow noreferrer">BERTology</a> to be a great starting point (section 4.1 discusses word embeddings).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T12:35:18.297Z
Answers.LastActivityDate: 2023-04-03T12:35:18.297Z
Answers.UserDisplayName: Hbar
Body: <p>It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is &quot;cold&quot;, then the 5th BERT embedding is a representation of that word, using context to disambiguate the word (e.g. determine whether it's to do with the illness or temperature).</p>
  <p>However, because of the self-attention encoder layers, this embedding can in theory incorporate information from any of the other words in the text. BERT is trained using masked language modelling (MLM), which would encourage each embedding to learn enough to predict the corresponding word. But why wouldn't it contain additional information from other words? In other words, is there any reason to believe the BERT embeddings for different words contain well-separated information?</p>
  
Tags: <natural-language-processing><word-embedding><bert><embeddings>
Title: Why are BERT embeddings interpreted as representations of the corresponding words?
UserDisplayName: EmbeddingEnthusiast
ThreadText: # Why are BERT embeddings interpreted as representations of the corresponding words?
  
  (Asked by: username_0 on 2021-06-02)
  
  It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a representation of that word, using context to disambiguate the word (e.g. determine whether it's to do with the illness or temperature).
  
  However, because of the self-attention encoder layers, this embedding can in theory incorporate information from any of the other words in the text. BERT is trained using masked language modelling (MLM), which would encourage each embedding to learn enough to predict the corresponding word. But why wouldn't it contain additional information from other words? In other words, is there any reason to believe the BERT embeddings for different words contain well-separated information?
  
  **Tags:** natural-language-processing, word-embedding, bert, embeddings
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-03. Score: 1
  
  While an input with $n$ tokens generates an output with $n$ vectors, there is a lot of cross mixing of information. One word, or even one word sense, may have many different representations. The embeddings are very sensitive to other words in the sentence, even when the sense of the word does not change. To measure how the embeddings change, we could use the standard deviation of the variations in the vector entries. For example, the difference between the word vectors for *run* in "Run fast" and "Run fast." (notice the period) is 0.24, which is only about half the difference among random uses of the word *run*.
  
  The authors of the [BERT paper](https://arxiv.org/abs/1810.04805) explored the possibility of generating contextualized word embeddings using BERT. Table 7 in the BERT paper explains different ways they tried for extracting word representations that would work well in a named entity extraction task. They concluded that the weighted sum of the last four hidden layers provides good contextualized word representations. However, there are still many other choices to make when it comes to developing contextualized word embeddings, such as how to go from the wordpieces that BERT uses to a single word vector. If you're looking to learn more about BERT-related topics, I found the paper on [BERTology](https://doi.org/10.1162/tacl_a_00349) to be a great starting point (section 4.1 discusses word embeddings).
  
  
  ---
  
  