Answers.Body: <p>The GPT models (as manifested using the decoder block in the original Transformer architecture) are not generating the embedding.  However, the weights from the GPT are being used as the initial weights in a new model, dubbed CPT [<a href="https://openai.com/research/clip" rel="nofollow noreferrer">1</a>], that can create embeddings.</p>
  <p>The key lies in Contrastive Modeling techniques - acquire pair data and train a new model to minimize loss on the known pairs. Look at the <a href="https://openai.com/research/clip" rel="nofollow noreferrer">CLIP model</a> for a good review of contrastive loss.  The CLIP model embeds the text and image into the same vector space. Thus, CLIP and contrastive models know how to create embeddings.</p>
  <p>Pair data, luckily, is prevalent on the web and easy to acquire:  [image, caption], [docstring, code], [podcast titles, podcast descriptions], etc.</p>
  <p>One key ingredient: you get way better results when you start with an initialized pre-trained model. That's where the GPT leverage comes in.  They are using the GPT models (at various sizes like Ada, Babbage, Curie, DaVinci...) as the initial weights on a new constrastive model.</p>
  <p>Thus, the formula is:</p>
  <ol>
  <li>A pre-trained GPT for initial weights</li>
  <li>Pair data for training</li>
  <li>Contrastive loss training</li>
  </ol>
  <p>The resulting model is what OpenAI calls a <a href="https://cdn.openai.com/papers/Text_and_Code_Embeddings_by_Contrastive_Pre_Training.pdf" rel="nofollow noreferrer">&quot;cpt&quot; model.</a></p>
  <p>It's unclear how well these contrastive loss embedding models work compared to a more traditional encoder-based LLM block (like universal-sentence-encode). <a href="https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9" rel="nofollow noreferrer">See this blog post for more.</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-18T23:28:23.337Z
Answers.Text: That paper uses encoder only models. How does it make sense to initialize encoders with decoder weights?
Answers.UserDisplayName: JobHunter69
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T14:36:41.100Z
Answers.LastActivityDate: 2023-07-17T14:36:41.100Z
Answers.UserDisplayName: sghael
Body: <p>GPT3 and 4 are both examples of decoder-only models. However OpenAI offers an text embedding API endpoint based on these models. This begs the general question how can one obtain text embeddings from a decoder-only transformer model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-07T22:19:42.827Z
Comments.Text: Would you add web link of example? Also regarding accepted opinion of the user sgheal, as the user JobHunter69 indicated, it is about an encoder. Your question is about the decoder, isn't it?
Comments.UserDisplayName: Cloud Cho
Tags: <transformer>
Title: How can an decoder-only transformer be used for document embedding?
UserDisplayName: Goods
ThreadText: # How can an decoder-only transformer be used for document embedding?
  
  (Asked by: username_0 on 2023-07-05)
  
  GPT3 and 4 are both examples of decoder-only models. However OpenAI offers an text embedding API endpoint based on these models. This begs the general question how can one obtain text embeddings from a decoder-only transformer model?
  
  **Tags:** transformer
  > **Comments:**
  > **username_1**: Would you add web link of example? Also regarding accepted opinion of the user sgheal, as the user JobHunter69 indicated, it is about an encoder. Your question is about the decoder, isn't it?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-17. Score: 2 | Accepted answer
  
  The GPT models (as manifested using the decoder block in the original Transformer architecture) are not generating the embedding. However, the weights from the GPT are being used as the initial weights in a new model, dubbed CPT [[1](https://openai.com/research/clip)], that can create embeddings.
  
  The key lies in Contrastive Modeling techniques - acquire pair data and train a new model to minimize loss on the known pairs. Look at the [CLIP model](https://openai.com/research/clip) for a good review of contrastive loss. The CLIP model embeds the text and image into the same vector space. Thus, CLIP and contrastive models know how to create embeddings.
  
  Pair data, luckily, is prevalent on the web and easy to acquire: [image, caption], [docstring, code], [podcast titles, podcast descriptions], etc.
  
  One key ingredient: you get way better results when you start with an initialized pre-trained model. That's where the GPT leverage comes in. They are using the GPT models (at various sizes like Ada, Babbage, Curie, DaVinci...) as the initial weights on a new constrastive model.
  
  Thus, the formula is:
  
  1. A pre-trained GPT for initial weights
  2. Pair data for training
  3. Contrastive loss training
  
  The resulting model is what OpenAI calls a ["cpt" model.](https://cdn.openai.com/papers/Text_and_Code_Embeddings_by_Contrastive_Pre_Training.pdf)
  
  It's unclear how well these contrastive loss embedding models work compared to a more traditional encoder-based LLM block (like universal-sentence-encode). [See this blog post for more.](https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9)
  
  > **Comments:**
  > **username_3**: That paper uses encoder only models. How does it make sense to initialize encoders with decoder weights?  
  
  ---
  
  