Answers.Body: <p>Usually the embeddings are not zero for the padding, since they are part fo the lookup table of the embedding layer, however in addition to the input, a mask is forwarded through the forward pass of the network to mask them out when it's relevant (in transformers encoder used in CLIP, in the cross-attention)</p>
  <p>So the actual transformer attention is something like:
  <span class="math-container">$$
  A = softmax(\frac{QK^T}{\sqrt{d_k}} \cdot (MM^T\cdot -\infty))V
  $$</span></p>
  <p>where <span class="math-container">$M$</span> is a vector <span class="math-container">$1xN$</span> (<span class="math-container">$N$</span> being the length of the sequence you have inputted, with <span class="math-container">$0$</span> for actual tokens, <span class="math-container">$1$</span> for masking)</p>
  <p>By doing so, you are adding <span class="math-container">$-\infty$</span> to the padding tokens, thus the attention will be 0, which means that in the last product (<span class="math-container">$softmax \cdot V$</span>) those embeddings that you are seeing not being zero are not considered</p>
  <p>However, I want to highlight that if the padding tokens were zeros, this still is needed, as after normalizatio, attention or FFNN step, they might not be zero anymore, thus contributing to the whole next attention step, which would make the network no more length invariant</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-04T22:26:47.933Z
Answers.LastActivityDate: 2023-08-04T22:26:47.933Z
Answers.UserDisplayName: Alberto
Body: <p>I'm working with OpenAI's CLIP model and trying to understand the output of the text encoder. When I input a short prompt like &quot;cat&quot;, the output is a tensor of shape [77, 1024]. My understanding is that the 1024 represents the dimensionality of the embeddings, and the 77 represents the maximum sequence length that the model can handle.</p>
  <p>Given that &quot;apple&quot; would be tokenized into far fewer than 77 tokens, I'm assuming that the remaining tokens are padding tokens. However, when I inspect the tensor, I don't see any zero values. I was expecting the embeddings for the padding tokens to be zero vectors, but this doesn't seem to be the case.</p>
  <p>My current hypothesis is that only the first few 1024-dimensional vectors in the tensor (corresponding to the tokens in my input) are significant, and the remaining vectors (corresponding to padding tokens) do not carry meaningful information about my input. Is this understanding correct?</p>
  <p>Also, could someone explain why the embeddings for the padding tokens are not zero vectors? How does the model ensure that these padding tokens do not contribute to the output?</p>
  
Tags: <transformer><embeddings>
Title: Understanding embedding outputs in transformer models like CLIP
UserDisplayName: hanugm
ThreadText: # Understanding embedding outputs in transformer models like CLIP
  
  (Asked by: username_0 on 2023-08-04)
  
  I'm working with OpenAI's CLIP model and trying to understand the output of the text encoder. When I input a short prompt like "cat", the output is a tensor of shape [77, 1024]. My understanding is that the 1024 represents the dimensionality of the embeddings, and the 77 represents the maximum sequence length that the model can handle.
  
  Given that "apple" would be tokenized into far fewer than 77 tokens, I'm assuming that the remaining tokens are padding tokens. However, when I inspect the tensor, I don't see any zero values. I was expecting the embeddings for the padding tokens to be zero vectors, but this doesn't seem to be the case.
  
  My current hypothesis is that only the first few 1024-dimensional vectors in the tensor (corresponding to the tokens in my input) are significant, and the remaining vectors (corresponding to padding tokens) do not carry meaningful information about my input. Is this understanding correct?
  
  Also, could someone explain why the embeddings for the padding tokens are not zero vectors? How does the model ensure that these padding tokens do not contribute to the output?
  
  **Tags:** transformer, embeddings
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-04. Score: 1
  
  Usually the embeddings are not zero for the padding, since they are part fo the lookup table of the embedding layer, however in addition to the input, a mask is forwarded through the forward pass of the network to mask them out when it's relevant (in transformers encoder used in CLIP, in the cross-attention)
  
  So the actual transformer attention is something like:
  $$
  A = softmax(\frac{QK^T}{\sqrt{d\_k}} \cdot (MM^T\cdot -\infty))V
  $$
  
  where $M$ is a vector $1xN$ ($N$ being the length of the sequence you have inputted, with $0$ for actual tokens, $1$ for masking)
  
  By doing so, you are adding $-\infty$ to the padding tokens, thus the attention will be 0, which means that in the last product ($softmax \cdot V$) those embeddings that you are seeing not being zero are not considered
  
  However, I want to highlight that if the padding tokens were zeros, this still is needed, as after normalizatio, attention or FFNN step, they might not be zero anymore, thus contributing to the whole next attention step, which would make the network no more length invariant
  
  
  ---
  
  