Answers.Body: <p>To answer your one question: Are embeddings model-specific? YES! They are. I am not going to invoke math or other techniques here. My explanation is going to be in a intuitive perspective. I don't know if the current literature and jargons will agree to my usage. But I get your question.</p>
  <p>Take a scenario where you trained a CNN to classify smileys or emjoies based on their positivity or negativity. So it's a binary classification problem. Say you achieved a very good model. The penultimate layer for this model will give you a higher dimensional &quot;embedding&quot; vector for any new emojie or smiley pic you are feeding to this CNN.</p>
  <p>Can you compare these embedding with this another ANN model's results arising from the same training and testing data? Technically you can't.</p>
  <p>But by coincidence it might be very related. How do you verify it? You can use the argument of @postnubilaphoebus to do a PCA on the embedding data so that you can compare these two embeddings (by reducing the dimensions to match between both embeddings results). And then if all inputs which you feed to both models and then PCA-ing the corresponding intermediate penultimate layer outputs gives you pairs of &quot;embeddings&quot;. Do a dot product analysis to compare them. You will know if they say the same thing or are very different.</p>
  <p>I am not a NLP or LLM person, but this philosophy will work for any models. Thank you. I hope you got an idea. If not I am willing to explain it further.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-28T18:57:43.290Z
Answers.Text: AI noob here. What's a PCA?
Answers.UserDisplayName: darKnight
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T06:19:50.143Z
Answers.Text: PCA = Principal Component Analysis. https://en.wikipedia.org/wiki/Principal_component_analysis#:~:text=Principal%20component%20analysis%20(PCA)%20is,the%20visualization%20of%20multidimensional%20data.
Answers.UserDisplayName: dexterdev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-11T07:10:12.237Z
Answers.LastActivityDate: 2023-06-11T07:10:12.237Z
Answers.UserDisplayName: dexterdev
Answers.Body: <p>Vector Embedding is always model-specific. Model to Model embedding may change.</p>
  <p>Please look at this video: <a href="https://youtu.be/PR7xz5vQKGg?t=367" rel="nofollow noreferrer">https://youtu.be/PR7xz5vQKGg?t=367</a> and this page which has a diagram showing how the model generates its embedding: <a href="https://www.pinecone.io/learn/vector-database/" rel="nofollow noreferrer">https://www.pinecone.io/learn/vector-database/</a></p>
  <p>The idea is to first create an embedding of your content using a specific model and store it in Vector DB.</p>
  <p>Then you will use the model again to convert your query (questions you might want to ask to search in your content) to generate embedding using the same model and use that embedding query matrix to search vector DB.</p>
  <p>The model controls how to generate embedding. It depends on what that model is supposed to do. It will create embedding accordingly on content as well as query.</p>
  <p>Remember this &quot;Embeddings differ from other machine learning techniques in that they are learned through training a model on a large dataset rather than being explicitly defined by a human expert. This allows the model to learn complex patterns and relationships in the data that may be difficult or impossible for a human to identify.&quot;.  In short, each model may generate different embeddings. We don't control how the model will generate its embeddings. The model evolves as it is trained on a dataset. Once the Model is finalized, it is settled on a specific type of embedding.    <a href="https://encord.com/blog/embeddings-machine-learning/" rel="nofollow noreferrer">https://encord.com/blog/embeddings-machine-learning/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-31T17:03:49.777Z
Answers.LastActivityDate: 2023-12-31T17:17:40.023Z
Answers.LastEditDate: 2023-12-31T17:17:40.023Z
Answers.UserDisplayName: Bakul Brahmbhatt
Answers.Body: <p>An embedding layer is a linear layer that is used to convert a discrete input into a vector of a fixed size, d. Learned embedding layers are often used in natural language processing (NLP). Common learned embeddings are GloVe, GoogleNews, or word2vec. <strong>These embeddings have often been trained on huge amounts of data (eg 3 billion words for GoogleNews), and as such can be applied over a wide variety of contexts (although you might need specific embeddings for specific tasks)</strong>. These embeddings were all obtained by training neural networks such that words that occur in similar contexts have a similar embedding (you may look up the individual training algorithms for a better understanding).</p>
  <p>Words that are similar to each other are closer to each other in d-dimensional space (where d is the vector size), whereas dissimilar words are far apart. This distance can be measured by cosine-similarity. Research has shown that using pre-trained embeddings usually improves model performance in many NLP tasks (<a href="https://arxiv.org/pdf/1804.06323.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1804.06323.pdf</a>).</p>
  <p>If you do use pre-trained word embeddings, make sure that the vocabulary of your training data is present in the embeddings. During neural network training when using pre-trained embeddings, it makes sense to freeze the weights of the embedding to prevent superfluous gradient computation (and ruin your embeddings). However, if some words present in your training data are not present in the embedding, you may consider only partially freezing the weights of your embedding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-20T22:44:28.450Z
Answers.Text: I think you misunderstood my question, I am asking that how embeddings learned in one model can be used in another? shouldnt embeddings be model specific?
Answers.UserDisplayName: Oculu 
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-20T22:54:14.163Z
Answers.Text: I'm afraid I don't understand what you mean. An embedding is simply a linear layer, you can just load its weights. See: https://keras.io/examples/nlp/pretrained_word_embeddings/ under the section "Load pre-trained word embeddings" for an example in Keras.
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-20T23:54:36.720Z
Answers.Text: I have put in the quotation that as per the the website it is mentioned `It can be used alone to learn a word embedding that can be saved and used in another model later` this is what my concern is how it can be used in another model? Shouldnt the embeddings learned is model-specific i.e. it can be used only in that model?
Answers.UserDisplayName: Oculu 
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-21T00:03:04.863Z
Answers.Text: No, embeddings aren't model-specific. Why would they be?
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-31T13:54:14.133Z
Answers.Text: Because different models require different input vector size and the way vector algebra works, if we change model then different parameters are needed for it to function, so if your model uses tanh as activation function and if you change to sigmoid, I am sure the embeddings need to be changed
Answers.UserDisplayName: Oculu 
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-31T14:57:02.803Z
Answers.Text: I see your issue now. You can just use PCA to adjust that (at least when your embedding dimension of your model is lower than the emb dimension of the pretrained embeddings). When it is the other way around, consider adapting the input dimension of your model. And I'm not sure why you think activation functions matter here. Maybe it's a good idea you explain how you think it works and we can spot any misconceptions.
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-31T22:58:55.837Z
Answers.Text: So basically my understanding is that embedding vectors are representation of the input learned by training some specific model, so if the model is to change say just only the activation function changed from tanh -1 to +1 to sigmoid 0 - 1 so the response by using embeddings learned by tanh-based model will be different by using same embeddings in sigmoid-based model which will affect the output. I think what the original text mentioned is wrong.
Answers.UserDisplayName: Oculu 
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-01T01:11:31.923Z
Answers.Text: The activation function is not relevant here. An embedding is the first layer of a neural network. So if you train a neural network to have a certain embedding, it doesn't matter what activation functions it has in later layers. The output also doesn't matter here. Maybe reading up on how word2vec (the model) got trained would help, specifically CBOW and skip-gram.
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-01T12:07:00.807Z
Answers.Text: Isnt embeddings learned by backpropogation? So the type of activations used affects the outcome?
Answers.UserDisplayName: Oculu 
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-01T16:40:10.903Z
Answers.Text: Backpropagation simply means to calculate the gradient of a loss function with respect to a model's variables. Upon obtaining the gradients, gradient descent is used for the model to step into the direction of the gradients, i.e. to change the model weights at each layer according to the weight update. This happens regardless of the activation functions used. What matters is that the outcome, i.e. the trained embedding, meaningfully separates words in d-dimensional space (where d is the vector size).
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-01T16:42:09.503Z
Answers.Text: I recommend you talk to a professor to understand this better, and explain your understanding of how embeddings are trained as detailed as possible. That way, they can quickly help point out what you are missing, because embeddings are definitely model-independent.
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T12:41:59.303Z
Answers.Text: I'm on the same page of @Oculu , I don't understand how embeddings can be so model friendly. I get how back propagation works, the point is that any model at the end of the training will generate different weights based on their trainingset and specification, so even if the layer size of different model matches, the weights in that layer doesn't, so how can an embedding (so as you say the first layer of the model) be compatible with the first level of another model?
  Did we have any reference to a specific explaination about this?
Answers.UserDisplayName: Not Important
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T12:49:59.817Z
Answers.Text: fun fact, I asked to ChatGPT 4: response was too long to post here but clarified me some concepts (need to check hallucinations). 
  
  Basically there are different approaches with pros and cons, here a ref to go deeper:
  
  Embeddings learned by one model can be used in another through transfer learning techniques such as pre-trained embeddings, fine-tuning, embedding layer sharing, and knowledge distillation.
  
  I think that the doubt me and @Oculu were sharing was on point, it's the implementation of embeddings and the models that matters
Answers.UserDisplayName: Not Important
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T12:58:14.467Z
Answers.Text: Also was point of doubt could arise about how these embeddings are used, because usually you use them to index external document sources and then search a user query against a vector DB containing these indexes, so it doesn't matter from where these embeddings are coming from as long as you index all documents using the same embeddings. Embeddings resoults aren't something you give back to the model
Answers.UserDisplayName: Not Important
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-20T16:58:56.903Z
Answers.LastActivityDate: 2022-10-20T16:58:56.903Z
Answers.UserDisplayName: postnubilaphoebus
Body: <p>In the <a href="https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/" rel="nofollow noreferrer">website</a> the following explanation is provided about Embedding layer:</p>
  <blockquote>
  <p>The Embedding layer is initialized with random weights and will learn
  an embedding for all of the words in the training dataset.</p>
  <p>It is a flexible layer that can be used in a variety of ways, such as:</p>
  <p>It can be used alone to learn a word embedding that can be saved and
  used in another model later. It can be used as part of a deep learning
  model where the embedding is learned along with the model itself. It
  can be used to load a pre-trained word embedding model, a type of
  transfer learning.</p>
  </blockquote>
  <p>Isnt embeddings model specific? I mean to learn a representation of something we need the model that something was used to represent! so how can embeddings learned in one model can be used in another?</p>
  
Tags: <keras><word-embedding><embeddings>
Title: How embeddings learned from one model can be used in another?
UserDisplayName: Oculu 
ThreadText: # How embeddings learned from one model can be used in another?
  
  (Asked by: username_0 on 2022-10-20)
  
  In the [website](https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/) the following explanation is provided about Embedding layer:
  
  > The Embedding layer is initialized with random weights and will learn
  > an embedding for all of the words in the training dataset.
  >
  > It is a flexible layer that can be used in a variety of ways, such as:
  >
  > It can be used alone to learn a word embedding that can be saved and
  > used in another model later. It can be used as part of a deep learning
  > model where the embedding is learned along with the model itself. It
  > can be used to load a pre-trained word embedding model, a type of
  > transfer learning.
  
  Isnt embeddings model specific? I mean to learn a representation of something we need the model that something was used to represent! so how can embeddings learned in one model can be used in another?
  
  **Tags:** keras, word-embedding, embeddings
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2022-10-20. Score: 2 | Accepted answer
  
  An embedding layer is a linear layer that is used to convert a discrete input into a vector of a fixed size, d. Learned embedding layers are often used in natural language processing (NLP). Common learned embeddings are GloVe, GoogleNews, or word2vec. **These embeddings have often been trained on huge amounts of data (eg 3 billion words for GoogleNews), and as such can be applied over a wide variety of contexts (although you might need specific embeddings for specific tasks)**. These embeddings were all obtained by training neural networks such that words that occur in similar contexts have a similar embedding (you may look up the individual training algorithms for a better understanding).
  
  Words that are similar to each other are closer to each other in d-dimensional space (where d is the vector size), whereas dissimilar words are far apart. This distance can be measured by cosine-similarity. Research has shown that using pre-trained embeddings usually improves model performance in many NLP tasks (<https://arxiv.org/pdf/1804.06323.pdf>).
  
  If you do use pre-trained word embeddings, make sure that the vocabulary of your training data is present in the embeddings. During neural network training when using pre-trained embeddings, it makes sense to freeze the weights of the embedding to prevent superfluous gradient computation (and ruin your embeddings). However, if some words present in your training data are not present in the embedding, you may consider only partially freezing the weights of your embedding.
  
  > **Comments:**
  > **username_0**: I think you misunderstood my question, I am asking that how embeddings learned in one model can be used in another? shouldnt embeddings be model specific?  
  > **username_4**: I'm afraid I don't understand what you mean. An embedding is simply a linear layer, you can just load its weights. See: https://keras.io/examples/nlp/pretrained_word_embeddings/ under the section "Load pre-trained word embeddings" for an example in Keras.  
  > **username_0**: I have put in the quotation that as per the the website it is mentioned `It can be used alone to learn a word embedding that can be saved and used in another model later` this is what my concern is how it can be used in another model? Shouldnt the embeddings learned is model-specific i.e. it can be used only in that model?  
  > **username_4**: No, embeddings aren't model-specific. Why would they be?  
  > **username_0**: Because different models require different input vector size and the way vector algebra works, if we change model then different parameters are needed for it to function, so if your model uses tanh as activation function and if you change to sigmoid, I am sure the embeddings need to be changed  
  > **username_4**: I see your issue now. You can just use PCA to adjust that (at least when your embedding dimension of your model is lower than the emb dimension of the pretrained embeddings). When it is the other way around, consider adapting the input dimension of your model. And I'm not sure why you think activation functions matter here. Maybe it's a good idea you explain how you think it works and we can spot any misconceptions.  
  > **username_0**: So basically my understanding is that embedding vectors are representation of the input learned by training some specific model, so if the model is to change say just only the activation function changed from tanh -1 to +1 to sigmoid 0 - 1 so the response by using embeddings learned by tanh-based model will be different by using same embeddings in sigmoid-based model which will affect the output. I think what the original text mentioned is wrong.  
  > **username_4**: The activation function is not relevant here. An embedding is the first layer of a neural network. So if you train a neural network to have a certain embedding, it doesn't matter what activation functions it has in later layers. The output also doesn't matter here. Maybe reading up on how word2vec (the model) got trained would help, specifically CBOW and skip-gram.  
  > **username_0**: Isnt embeddings learned by backpropogation? So the type of activations used affects the outcome?  
  > **username_4**: Backpropagation simply means to calculate the gradient of a loss function with respect to a model's variables. Upon obtaining the gradients, gradient descent is used for the model to step into the direction of the gradients, i.e. to change the model weights at each layer according to the weight update. This happens regardless of the activation functions used. What matters is that the outcome, i.e. the trained embedding, meaningfully separates words in d-dimensional space (where d is the vector size).  
  > **username_4**: I recommend you talk to a professor to understand this better, and explain your understanding of how embeddings are trained as detailed as possible. That way, they can quickly help point out what you are missing, because embeddings are definitely model-independent.  
  > **username_5**: I'm on the same page of @Oculu , I don't understand how embeddings can be so model friendly. I get how back propagation works, the point is that any model at the end of the training will generate different weights based on their trainingset and specification, so even if the layer size of different model matches, the weights in that layer doesn't, so how can an embedding (so as you say the first layer of the model) be compatible with the first level of another model? Did we have any reference to a specific explaination about this?  
  > **username_5**: fun fact, I asked to ChatGPT 4: response was too long to post here but clarified me some concepts (need to check hallucinations).   Basically there are different approaches with pros and cons, here a ref to go deeper:  Embeddings learned by one model can be used in another through transfer learning techniques such as pre-trained embeddings, fine-tuning, embedding layer sharing, and knowledge distillation.  I think that the doubt me and @Oculu were sharing was on point, it's the implementation of embeddings and the models that matters  
  > **username_5**: Also was point of doubt could arise about how these embeddings are used, because usually you use them to index external document sources and then search a user query against a vector DB containing these indexes, so it doesn't matter from where these embeddings are coming from as long as you index all documents using the same embeddings. Embeddings resoults aren't something you give back to the model  
  
  ---
  
  ### Answer by username_1 on 2023-06-11. Score: 3
  
  To answer your one question: Are embeddings model-specific? YES! They are. I am not going to invoke math or other techniques here. My explanation is going to be in a intuitive perspective. I don't know if the current literature and jargons will agree to my usage. But I get your question.
  
  Take a scenario where you trained a CNN to classify smileys or emjoies based on their positivity or negativity. So it's a binary classification problem. Say you achieved a very good model. The penultimate layer for this model will give you a higher dimensional "embedding" vector for any new emojie or smiley pic you are feeding to this CNN.
  
  Can you compare these embedding with this another ANN model's results arising from the same training and testing data? Technically you can't.
  
  But by coincidence it might be very related. How do you verify it? You can use the argument of @postnubilaphoebus to do a PCA on the embedding data so that you can compare these two embeddings (by reducing the dimensions to match between both embeddings results). And then if all inputs which you feed to both models and then PCA-ing the corresponding intermediate penultimate layer outputs gives you pairs of "embeddings". Do a dot product analysis to compare them. You will know if they say the same thing or are very different.
  
  I am not a NLP or LLM person, but this philosophy will work for any models. Thank you. I hope you got an idea. If not I am willing to explain it further.
  
  > **Comments:**
  > **username_2**: AI noob here. What's a PCA?  
  > **username_1**: PCA = Principal Component Analysis. https://en.wikipedia.org/wiki/Principal_component_analysis#:~:text=Principal%20component%20analysis%20(PCA)%20is,the%20visualization%20of%20multidimensional%20data.  
  
  ---
  
  ### Answer by username_3 on 2023-12-31. Score: 0
  
  Vector Embedding is always model-specific. Model to Model embedding may change.
  
  Please look at this video: <https://youtu.be/PR7xz5vQKGg?t=367> and this page which has a diagram showing how the model generates its embedding: <https://www.pinecone.io/learn/vector-database/>
  
  The idea is to first create an embedding of your content using a specific model and store it in Vector DB.
  
  Then you will use the model again to convert your query (questions you might want to ask to search in your content) to generate embedding using the same model and use that embedding query matrix to search vector DB.
  
  The model controls how to generate embedding. It depends on what that model is supposed to do. It will create embedding accordingly on content as well as query.
  
  Remember this "Embeddings differ from other machine learning techniques in that they are learned through training a model on a large dataset rather than being explicitly defined by a human expert. This allows the model to learn complex patterns and relationships in the data that may be difficult or impossible for a human to identify.". In short, each model may generate different embeddings. We don't control how the model will generate its embeddings. The model evolves as it is trained on a dataset. Once the Model is finalized, it is settled on a specific type of embedding. <https://encord.com/blog/embeddings-machine-learning/>
  
  
  ---
  
  