Answers.Body: <p>Embedding functions can differ in several ways. Additionally, some of the jargon can be confusing in this area.</p>
  <p>Starting simply, PyTorch's nn.Embedding is a simple lookup table. The catch is that it also carries its learnable weights so input embedding can be learned during back-propagation.</p>
  <p>From there, it gets more complex. The output of any function in a network are considered &quot;embeddings&quot;, but my understanding is that usable embeddings are understood to be the output of the last hidden layer of the network.</p>
  <p>There are variety of functions. You can see many different types of functions at <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noreferrer">https://pytorch.org/docs/stable/nn.html</a> . Right now, self-attention is very popular and is utilized in scaled dot-product attention in transformers.</p>
  <p>OpenAI's embedding API may be advantageous as you don't need to build or maintain a network. Additionally, their models have been trained on very large amounts of texts, given that they have a very powerful cluster. These are likely more informative embeddings than you can generate on your own, and so wind up being very useful for informational retrieval or comparison.</p>
  <p>I think a question implied, but not asked here is why is generating vectors and then comparing them useful? To illustrate, given a random point in a square, the chance of the point being near an edge is 0.1%. If choice a random point in a 1000D object, the chance of that point being near the edge is 99.9%. The high dimensionality allows us to find extremes in samples much more easily than low dimension objects.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-01T15:46:41.463Z
Answers.LastActivityDate: 2023-06-01T14:08:53.137Z
Answers.LastEditDate: 2023-06-01T14:08:53.137Z
Answers.UserDisplayName: Thomas K
Body: <p>Embeddings are just arrays of numbers that represent some data (sentences or images). How do embedding functions differ. For example what advantage does OpenAI's embeddings API have over others?</p>
  
Tags: <embeddings>
Title: What do different embedding functions do?
UserDisplayName: felipe57387
ThreadText: # What do different embedding functions do?
  
  (Asked by: username_0 on 2023-05-01)
  
  Embeddings are just arrays of numbers that represent some data (sentences or images). How do embedding functions differ. For example what advantage does OpenAI's embeddings API have over others?
  
  **Tags:** embeddings
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-01. Score: 1
  
  Embedding functions can differ in several ways. Additionally, some of the jargon can be confusing in this area.
  
  Starting simply, PyTorch's nn.Embedding is a simple lookup table. The catch is that it also carries its learnable weights so input embedding can be learned during back-propagation.
  
  From there, it gets more complex. The output of any function in a network are considered "embeddings", but my understanding is that usable embeddings are understood to be the output of the last hidden layer of the network.
  
  There are variety of functions. You can see many different types of functions at <https://pytorch.org/docs/stable/nn.html> . Right now, self-attention is very popular and is utilized in scaled dot-product attention in transformers.
  
  OpenAI's embedding API may be advantageous as you don't need to build or maintain a network. Additionally, their models have been trained on very large amounts of texts, given that they have a very powerful cluster. These are likely more informative embeddings than you can generate on your own, and so wind up being very useful for informational retrieval or comparison.
  
  I think a question implied, but not asked here is why is generating vectors and then comparing them useful? To illustrate, given a random point in a square, the chance of the point being near an edge is 0.1%. If choice a random point in a 1000D object, the chance of that point being near the edge is 99.9%. The high dimensionality allows us to find extremes in samples much more easily than low dimension objects.
  
  
  ---
  
  