Body: <p>There is OpenAI embedding API <a href="https://platform.openai.com/docs/guides/embeddings" rel="nofollow noreferrer">https://platform.openai.com/docs/guides/embeddings</a>. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input embeddings part, which is the summation of word embedding and position embedding, at the end of model training?</p>
  <p>Any tutorials or guidance will be helpful.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-04T00:00:57.250Z
Comments.Text: Please, one question per post and focus on the theoretical aspects, otherwise, this could be off-topic.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-04T00:17:45.430Z
Comments.Text: Thanks for the suggestion. Edited the question.
Comments.UserDisplayName: Salty Gold Fish
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-04T01:13:43.563Z
Comments.Text: It generally represents the output from the last layer of the encoder architecture. However, multiple methods are used to obtain the embeddings by designers. Some take the embeddings from a specific hidden layer, some average out the outputs from multiple layers, some get the embedding from the last layer.
Comments.UserDisplayName: Chinmay
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-04T17:33:33.897Z
Comments.Text: Thanks! This is helpful. Are the optimization goals for 'model used for inference purpose' and 'model used for embedding purpose' the same?
Comments.UserDisplayName: Salty Gold Fish
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-04T19:07:08.560Z
Comments.Text: Yes, kind of. So, for embeddings, you use the encoder part of the architecture, but for inference, you also need the decoder part. The training process or optimization goals mostly are the same for both Transformers and BERTs.
Comments.UserDisplayName: Chinmay
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-14T11:38:14.887Z
Comments.Text: I can't seem to find GPT3.5 amongst the models used for embeddings so it's hard to say how these embeddings are obtained.
Comments.UserDisplayName: Daviiid
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-04T01:44:05.117Z
Comments.Text: I share a similar curiosity. I'm pondering whether they employ a Siamese network architecture, akin to what is utilized in Sentence Transformers. The Sentence Transformer incorporates additional layers post-BERT, which are trained on a corpus of analogous sentences and necessitates a corpus comprising pairs of similar sentences. It's quite plausible that they may not be employing a Siamese architecture. Unfortunately, there doesn't seem to be an abundance of details available on this matter.
Comments.UserDisplayName: user1264641
Tags: <open-ai><embeddings><gpt-3>
Title: How is OpenAI embeddings obtained
UserDisplayName: Salty Gold Fish
ThreadText: # How is OpenAI embeddings obtained
  
  (Asked by: username_0 on 2023-07-03)
  
  There is OpenAI embedding API <https://platform.openai.com/docs/guides/embeddings>. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input embeddings part, which is the summation of word embedding and position embedding, at the end of model training?
  
  Any tutorials or guidance will be helpful.
  
  **Tags:** open-ai, embeddings, gpt-3
  > **Comments:**
  > **username_1**: Please, one question per post and focus on the theoretical aspects, otherwise, this could be off-topic.  
  > **username_0**: Thanks for the suggestion. Edited the question.  
  > **username_2**: It generally represents the output from the last layer of the encoder architecture. However, multiple methods are used to obtain the embeddings by designers. Some take the embeddings from a specific hidden layer, some average out the outputs from multiple layers, some get the embedding from the last layer.  
  > **username_0**: Thanks! This is helpful. Are the optimization goals for 'model used for inference purpose' and 'model used for embedding purpose' the same?  
  > **username_2**: Yes, kind of. So, for embeddings, you use the encoder part of the architecture, but for inference, you also need the decoder part. The training process or optimization goals mostly are the same for both Transformers and BERTs.  
  > **username_3**: I can't seem to find GPT3.5 amongst the models used for embeddings so it's hard to say how these embeddings are obtained.  
  > **username_4**: I share a similar curiosity. I'm pondering whether they employ a Siamese network architecture, akin to what is utilized in Sentence Transformers. The Sentence Transformer incorporates additional layers post-BERT, which are trained on a corpus of analogous sentences and necessitates a corpus comprising pairs of similar sentences. It's quite plausible that they may not be employing a Siamese architecture. Unfortunately, there doesn't seem to be an abundance of details available on this matter.  
  
  ---
  
  ## 0 answers
  
  