Answers.Body: <p>Without positional embeddings, the self-attention mechanism even with causal mask is <em>permutation-invariant</em>, meaning it simply treats the causally attended input sequence portion as a bag of words without inherent order which is necessary for many common NLP tasks where sequence matters such as understanding syntax or more high-level semantic temporal relations. For example, without positional encoding information in the embeddings, LLM would struggle to differentiate sequences between &quot;the lion chased the tiger&quot; and &quot;the tiger chased the lion&quot; both of which are outside the mask.</p>
  <p>Of course you could use <a href="https://arxiv.org/pdf/1803.02155" rel="nofollow noreferrer">relative positional encoding</a> or adding additional bias to model the same in the self-attention mechanism directly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-25T22:32:52.237Z
Answers.LastActivityDate: 2024-11-25T22:32:52.237Z
Answers.UserDisplayName: cinch
Body: <p>I'm learning about transformers and their components, and I understand that:</p>
  <p><strong>Masked attention</strong> ensures that each token can only attend to previous tokens (or itself) in causal language models, effectively encoding the order of tokens.</p>
  <p><strong>Positional embeddings</strong> are added to token embeddings to incorporate position information explicitly.</p>
  <p>This makes me wonder: if masked attention already respects the sequence of tokens by enforcing causality, why do we still need positional embeddings? Doesn't the masking already capture the token order implicitly?</p>
  <p>Can we get away without using the positional embedding if the masked attention already attends to the need of positioning of token ?</p>
  
Tags: <transformer><attention><positional-encoding>
Title: If masked attention tells which token precedes which token, why are positional embeddings required?
UserDisplayName: Saurabh Patel
ThreadText: # If masked attention tells which token precedes which token, why are positional embeddings required?
  
  (Asked by: username_0 on 2024-11-25)
  
  I'm learning about transformers and their components, and I understand that:
  
  **Masked attention** ensures that each token can only attend to previous tokens (or itself) in causal language models, effectively encoding the order of tokens.
  
  **Positional embeddings** are added to token embeddings to incorporate position information explicitly.
  
  This makes me wonder: if masked attention already respects the sequence of tokens by enforcing causality, why do we still need positional embeddings? Doesn't the masking already capture the token order implicitly?
  
  Can we get away without using the positional embedding if the masked attention already attends to the need of positioning of token ?
  
  **Tags:** transformer, attention, positional-encoding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-25. Score: 0
  
  Without positional embeddings, the self-attention mechanism even with causal mask is *permutation-invariant*, meaning it simply treats the causally attended input sequence portion as a bag of words without inherent order which is necessary for many common NLP tasks where sequence matters such as understanding syntax or more high-level semantic temporal relations. For example, without positional encoding information in the embeddings, LLM would struggle to differentiate sequences between "the lion chased the tiger" and "the tiger chased the lion" both of which are outside the mask.
  
  Of course you could use [relative positional encoding](https://arxiv.org/pdf/1803.02155) or adding additional bias to model the same in the self-attention mechanism directly.
  
  
  ---
  
  