Answers.Body: <p>It makes sense to provide a more context-sensitive embedding via weighted average of last hidden layer tokens to create a smoother gradient across the input sequence for capturing subtle trends over the entire text better, in addition to the multihead-attention mechanism which already implicitly does some <em>relevance</em> weighting of all previous tokens for prediction, especially for autoregressive models like Llama.</p>
  <p>There're similar models to implement your general ideas, for instance, &quot;<em>Investigating Layer-wise BERT Representations</em>&quot; by Jawahar et al (2019). This study provides insights into what each layer in BERT learns, which has motivated approaches that assign learned weights to layer outputs based on their contribution to specific tasks. This approach could inspire your proposed token-based adaptive weighting strategies. In document modeling, hierarchical attention layers often assign varying levels of importance across tokens or even sentences. This is somewhat similar to weighted averaging, but instead of a simple 1,2,3 weighting schedule, it assigns learned weights to tokens based on importance, see paper by Yang, Yang et al (2016) &quot;<em>Hierarchical Attention Networks for Document Classification</em>&quot;. Also see  Shaw et al (2018) &quot;<em>Self-Attention with Relative Position Representations</em>&quot;. This paper adapts self-attention by adding <em>relative</em> position information, which is crucial for document-level models where the position and structure of sentences contribute to the model’s understanding of importance across the text.</p>
  <p>In some NLP tasks, however, earlier tokens may carry meaningful more information such as topic-setting terms or with a longer prompt context. So your proposed straightforward absolute positional weighting model may likely encounter performance issues in these common cases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-05T20:02:28.717Z
Answers.LastActivityDate: 2024-11-05T20:02:28.717Z
Answers.UserDisplayName: cinch
Body: <p>I would like to do a smart text embedding from models such as Llama3 or similar. Two commonly approaches are</p>
  <p>(A) either average last hidden layers over all tokens, or</p>
  <p>(B) just take the last hidden layer above the last &lt;|end-of-text|&gt; token</p>
  <p>Would it make sense to do a weighted average in which latter tokens have higher weights than former tokens (with weights 1,2,3,4,... maybe)? The idea is that latter tokens can &quot;see&quot; the former context in autoregressive models, but the initial tokens are less information rich.</p>
  <p>(B) can be considered as the most extreme case with weights (0,0,...,0, 1).</p>
  <p>Any experience with this type of averaging? Is it used / does it make sense?</p>
  
Tags: <large-language-models><embeddings>
Title: Text embedding from Llamas
UserDisplayName: Peter Franek
ThreadText: # Text embedding from Llamas
  
  (Asked by: username_0 on 2024-11-05)
  
  I would like to do a smart text embedding from models such as Llama3 or similar. Two commonly approaches are
  
  (A) either average last hidden layers over all tokens, or
  
  (B) just take the last hidden layer above the last <|end-of-text|> token
  
  Would it make sense to do a weighted average in which latter tokens have higher weights than former tokens (with weights 1,2,3,4,... maybe)? The idea is that latter tokens can "see" the former context in autoregressive models, but the initial tokens are less information rich.
  
  (B) can be considered as the most extreme case with weights (0,0,...,0, 1).
  
  Any experience with this type of averaging? Is it used / does it make sense?
  
  **Tags:** large-language-models, embeddings
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-05. Score: 1 | Accepted answer
  
  It makes sense to provide a more context-sensitive embedding via weighted average of last hidden layer tokens to create a smoother gradient across the input sequence for capturing subtle trends over the entire text better, in addition to the multihead-attention mechanism which already implicitly does some *relevance* weighting of all previous tokens for prediction, especially for autoregressive models like Llama.
  
  There're similar models to implement your general ideas, for instance, "*Investigating Layer-wise BERT Representations*" by Jawahar et al (2019). This study provides insights into what each layer in BERT learns, which has motivated approaches that assign learned weights to layer outputs based on their contribution to specific tasks. This approach could inspire your proposed token-based adaptive weighting strategies. In document modeling, hierarchical attention layers often assign varying levels of importance across tokens or even sentences. This is somewhat similar to weighted averaging, but instead of a simple 1,2,3 weighting schedule, it assigns learned weights to tokens based on importance, see paper by Yang, Yang et al (2016) "*Hierarchical Attention Networks for Document Classification*". Also see Shaw et al (2018) "*Self-Attention with Relative Position Representations*". This paper adapts self-attention by adding *relative* position information, which is crucial for document-level models where the position and structure of sentences contribute to the model’s understanding of importance across the text.
  
  In some NLP tasks, however, earlier tokens may carry meaningful more information such as topic-setting terms or with a longer prompt context. So your proposed straightforward absolute positional weighting model may likely encounter performance issues in these common cases.
  
  
  ---
  
  