Answers.Body: <p>For the purpose of downstream tasks BERT does not use static word embeddings in the traditional sense as seen in models like Word2Vec or GloVe, it starts with an initial embedding layer where each token is mapped to a fixed-length vector. This initial mapping is learned during pre-training, and it is further contextualized throughout the layers of its bidirectional transformer. Here each input token is mapped to a pre-trained word embedding vector of size <span class="math-container">$d_\text{model}=768$</span> which is learned during BERT pre-training on massive corpora using the Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks, therefore tokens that appear in similar contexts during pre-training must tend to have embeddings that are closer to each other in the embedding space, which is shown in your two example pairwise Euclidean distances.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-17T06:44:34.070Z
Answers.LastActivityDate: 2024-12-17T08:32:26.617Z
Answers.LastEditDate: 2024-12-17T08:32:26.617Z
Answers.UserDisplayName: cinch
Body: <p>We know BERT's output will have contextual embeddings for each input token. This means the token &quot;log&quot; will have different output embeddings when input texts &quot;wood log&quot; and &quot;error log&quot; are processed through BERT.</p>
  <p>But each token receives a static embedding learned during BERT's pre-training. These static embeddings (in combination with position and segment embeddings) are used in BERT's embedding layer.</p>
  <p>Do semantically similar tokens receive similar static embeddings? For example, is &quot;copper&quot; closer to &quot;bronze&quot; than &quot;beef&quot;?</p>
  <p>The output of the following code snippet indicate that the answer is yes. Distance between &quot;copper&quot; and &quot;bronze&quot; is 1.1040, while it is 1.3561 between &quot;copper&quot; and &quot;beef.&quot;</p>
  <p>But is that true in general? Can you explain why?</p>
  <pre><code>from transformers import BertModel, BertTokenizer
  import torch
  
  tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
  model = BertModel.from_pretrained('bert-base-uncased')
  
  input_text = &quot;copper bronze beef&quot;
  tokens = tokenizer.tokenize(input_text)
  print(f'Tokens: {tokens}')
  
  token_ids = tokenizer(input_text, return_tensors='pt')['input_ids'][0][1:-1]
  static_embeddings = model.embeddings.word_embeddings(token_ids)
  print('Distance between static embeddings:')
  print(torch.cdist(static_embeddings, static_embeddings, p=2))
  </code></pre>
  
Tags: <large-language-models><word-embedding><bert>
Title: Do the static token embeddings learned by BERT have semantic meaning?
UserDisplayName: Fijoy Vadakkumpadan
ThreadText: # Do the static token embeddings learned by BERT have semantic meaning?
  
  (Asked by: username_0 on 2024-12-15)
  
  We know BERT's output will have contextual embeddings for each input token. This means the token "log" will have different output embeddings when input texts "wood log" and "error log" are processed through BERT.
  But each token receives a static embedding learned during BERT's pre-training. These static embeddings (in combination with position and segment embeddings) are used in BERT's embedding layer.
  Do semantically similar tokens receive similar static embeddings? For example, is "copper" closer to "bronze" than "beef"?
  The output of the following code snippet indicate that the answer is yes. Distance between "copper" and "bronze" is 1.1040, while it is 1.3561 between "copper" and "beef."
  But is that true in general? Can you explain why?
  from transformers import BertModel, BertTokenizer
  import torch
  
  tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
  model = BertModel.from_pretrained('bert-base-uncased')
  
  input_text = "copper bronze beef"
  tokens = tokenizer.tokenize(input_text)
  print(f'Tokens: {tokens}')
  
  token_ids = tokenizer(input_text, return_tensors='pt')['input_ids'][0][1:-1]
  static_embeddings = model.embeddings.word_embeddings(token_ids)
  print('Distance between static embeddings:')
  print(torch.cdist(static_embeddings, static_embeddings, p=2))
  
  **Tags:** large-language-models, word-embedding, bert
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-17. Score: 1
  
  For the purpose of downstream tasks BERT does not use static word embeddings in the traditional sense as seen in models like Word2Vec or GloVe, it starts with an initial embedding layer where each token is mapped to a fixed-length vector. This initial mapping is learned during pre-training, and it is further contextualized throughout the layers of its bidirectional transformer. Here each input token is mapped to a pre-trained word embedding vector of size $d\_\text{model}=768$ which is learned during BERT pre-training on massive corpora using the Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks, therefore tokens that appear in similar contexts during pre-training must tend to have embeddings that are closer to each other in the embedding space, which is shown in your two example pairwise Euclidean distances.
  
  
  ---
  
  