Answers.Body: <p>Either by building embeddings yourself or loading pretrained embeddings.</p>
  <p>For building yourself, this is typically done with an auto-regressive model.  It can be as simple as creating numeric representations of all of the words in a corpus and generating random embeddings of whatever dimensionality you choose that are randomly initialized.  The auto-regressive process could be to process your corpus word by word with the objective of predicting the next word in the corpus with some selected window size.</p>
  <p>Alternatively, you could take your corpus as a chunk of any arbitrary size you choose, representing it as the equivalent vector made up of the word numbers (which is ultimately converted to a vector of embeddings for the words).  Your training process would randomly (or progressively) exclude one of the words in the vector of embeddings with the objective of predicting the missing word.</p>
  <p>Over time, the embeddings emerge.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-13T17:45:03.407Z
Answers.Text: It's bit confusing. Would be helpful if I can see explanation with some examples 
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-13T20:27:11.040Z
Answers.Text: embeddings are just learned weights like every other parameter
  
  https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html
Answers.UserDisplayName: Karl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T04:23:13.720Z
Answers.Text: @Karl so before training a transformer model, we first need to train another model to get word to embedding?
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T10:44:53.023Z
Answers.Text: like I said, embeddings are just learned weights like every other parameter. you train them at the same time as every other parameter. there's nothing special about the embedding weights.
Answers.UserDisplayName: Karl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T12:33:01.547Z
Answers.Text: Ok let me be more clear. Given an input sentence, let's say below steps happens inputs -> tokens -> word embedding -> encoders. Here how is token to word embedding created?
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T15:30:49.067Z
Answers.Text: You can pretrain the embeddings or you can use pretrained embeddings and use them to encode data before they go to your transformer.  In the latter case you would usually freeze the embeddings.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-15T05:17:49.640Z
Answers.Text: @DavidHoelzer Which technique is used to pretrain?. Let's say when BERT model was initially built how would have they might have mapped tokens to word embeddings?
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-15T07:40:42.677Z
Answers.Text: Ok, I understood that one of the two techniques mentioned above can be used, but want to know what they are called as... Are there any code implementation which I can refer to understand in detail?
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-15T17:43:21.037Z
Answers.Text: WHen they were first built they did *nothing* other than training.  They were not pre-initialized in any way other than stochastic initialization.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T17:58:16.790Z
Answers.Text: @DavidHoelzer any code implementation from scratch to understand how auto-regressive model can be used to generate initial word embeddings
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T22:08:16.730Z
Answers.Text: No, I generally do not share code.  However, it is simple as fitting where the target is the next word through a model that begins with the embeddings to be trained.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T18:52:01.930Z
Answers.Text: @DavidHoelzer Is masking necessary for the embedding layer like in the attention layer where we mask the padded places to be excluded from attention?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-13T15:42:49.977Z
Answers.LastActivityDate: 2023-10-13T15:42:49.977Z
Answers.UserDisplayName: David Hoelzer
Answers.Body: <p>Embeddings are essentially just a lookup table from integers to vector representations. These representations are learned during the pretraining process like any other parameter in the language model.</p>
  <p>That is, they are initialized randomly then, as the model trains during pretraining (typically with some next-token prediction or denoising objective), gradients are propagated back to the embedding lookup table. Just as the regular parameters change to allow the model to predict tokens effectively, the token representations will also change to become more &quot;meaningful&quot; for the model.</p>
  <p>If you further finetune the model, you use the existing pretrained parameters of the model, <em>including</em> the token embedding representations. Like any other parameter in the model, you can then choose to freeze or finetune the token embeddings as well.</p>
  <p><strong>You should just think of token embeddings like any other parameter in the model.</strong> Although there are other <em>kinds</em> of token embeddings (e.g., word2vec), in the context of large language model training, I'm not aware of any other initialization method other than what I described above, other than some niche scenarios (e.g., cross-lingual transfer: <a href="https://arxiv.org/pdf/2301.09626.pdf" rel="nofollow noreferrer">1</a>, <a href="https://aclanthology.org/2022.naacl-main.293.pdf" rel="nofollow noreferrer">2</a>, <a href="https://arxiv.org/abs/2305.14481" rel="nofollow noreferrer">3</a>).</p>
  <p>If you want a code example, you can take a look at the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html" rel="nofollow noreferrer">Embedding</a> class in PyTorch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T17:56:41.513Z
Answers.Text: How can next word prediction kind of architecture helps in generating initial word embeddings?
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T21:01:56.767Z
Answers.Text: @Swastik They are learned through backpropagation like any other parameters in the model. As the model trains to be better at next token prediction, the embeddings need to become more meaningful, just as the parameters become more meaningful during training.
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-18T01:20:36.340Z
Answers.Text: is there any code implementation which I can refer to understand more. Or if I want to build that from scratch just to learn how it works
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-18T01:34:39.570Z
Answers.Text: @Swastik check out this tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T02:44:38.473Z
Answers.LastActivityDate: 2023-10-16T02:44:38.473Z
Answers.UserDisplayName: Alexander Wan
Body: <p>In Transformers Encoders, from where do the Encoders get Input Embedding from?</p>
  <p>So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with some integer in vocabulary. After that how does those tokens (integer) is converted to Input Embeddings?</p>
  <p>If initial embeddings are obtained from some pre-trained model, how did that model was able to generate embeddings for a word?</p>
  <p><a href="https://i.sstatic.net/KNx4U.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KNx4U.png" alt="enter image description here" /></a></p>
  
Tags: <transformer><autoencoders><word-embedding><embeddings>
Title: From where do the Encoders in Transformers gets Input Embedding from?
UserDisplayName: Swastik
ThreadText: # From where do the Encoders in Transformers gets Input Embedding from?
  
  (Asked by: username_0 on 2023-10-13)
  
  In Transformers Encoders, from where do the Encoders get Input Embedding from?
  
  So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with some integer in vocabulary. After that how does those tokens (integer) is converted to Input Embeddings?
  
  If initial embeddings are obtained from some pre-trained model, how did that model was able to generate embeddings for a word?
  
  [![enter image description here](https://i.sstatic.net/KNx4U.png)](https://i.sstatic.net/KNx4U.png)
  
  **Tags:** transformer, autoencoders, word-embedding, embeddings
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-10-13. Score: 2
  
  Either by building embeddings yourself or loading pretrained embeddings.
  
  For building yourself, this is typically done with an auto-regressive model. It can be as simple as creating numeric representations of all of the words in a corpus and generating random embeddings of whatever dimensionality you choose that are randomly initialized. The auto-regressive process could be to process your corpus word by word with the objective of predicting the next word in the corpus with some selected window size.
  
  Alternatively, you could take your corpus as a chunk of any arbitrary size you choose, representing it as the equivalent vector made up of the word numbers (which is ultimately converted to a vector of embeddings for the words). Your training process would randomly (or progressively) exclude one of the words in the vector of embeddings with the objective of predicting the missing word.
  
  Over time, the embeddings emerge.
  
  > **Comments:**
  > **username_0**: It's bit confusing. Would be helpful if I can see explanation with some examples   
  > **username_2**: embeddings are just learned weights like every other parameter  https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html  
  > **username_0**: @username_2 so before training a transformer model, we first need to train another model to get word to embedding?  
  > **username_2**: like I said, embeddings are just learned weights like every other parameter. you train them at the same time as every other parameter. there's nothing special about the embedding weights.  
  > **username_0**: Ok let me be more clear. Given an input sentence, let's say below steps happens inputs -> tokens -> word embedding -> encoders. Here how is token to word embedding created?  
  > **username_1**: You can pretrain the embeddings or you can use pretrained embeddings and use them to encode data before they go to your transformer.  In the latter case you would usually freeze the embeddings.  
  > **username_0**: @username_1 Which technique is used to pretrain?. Let's say when BERT model was initially built how would have they might have mapped tokens to word embeddings?  
  > **username_0**: Ok, I understood that one of the two techniques mentioned above can be used, but want to know what they are called as... Are there any code implementation which I can refer to understand in detail?  
  > **username_1**: WHen they were first built they did *nothing* other than training.  They were not pre-initialized in any way other than stochastic initialization.  
  > **username_0**: @username_1 any code implementation from scratch to understand how auto-regressive model can be used to generate initial word embeddings  
  > **username_1**: No, I generally do not share code.  However, it is simple as fitting where the target is the next word through a model that begins with the embeddings to be trained.  
  > **username_3**: @username_1 Is masking necessary for the embedding layer like in the attention layer where we mask the padded places to be excluded from attention?  
  
  ---
  
  ### Answer by username_4 on 2023-10-16. Score: 0
  
  Embeddings are essentially just a lookup table from integers to vector representations. These representations are learned during the pretraining process like any other parameter in the language model.
  
  That is, they are initialized randomly then, as the model trains during pretraining (typically with some next-token prediction or denoising objective), gradients are propagated back to the embedding lookup table. Just as the regular parameters change to allow the model to predict tokens effectively, the token representations will also change to become more "meaningful" for the model.
  
  If you further finetune the model, you use the existing pretrained parameters of the model, *including* the token embedding representations. Like any other parameter in the model, you can then choose to freeze or finetune the token embeddings as well.
  
  **You should just think of token embeddings like any other parameter in the model.** Although there are other *kinds* of token embeddings (e.g., word2vec), in the context of large language model training, I'm not aware of any other initialization method other than what I described above, other than some niche scenarios (e.g., cross-lingual transfer: [1](https://arxiv.org/pdf/2301.09626.pdf), [2](https://aclanthology.org/2022.naacl-main.293.pdf), [3](https://arxiv.org/abs/2305.14481)).
  
  If you want a code example, you can take a look at the [Embedding](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html) class in PyTorch.
  
  > **Comments:**
  > **username_0**: How can next word prediction kind of architecture helps in generating initial word embeddings?  
  > **username_4**: @username_0 They are learned through backpropagation like any other parameters in the model. As the model trains to be better at next token prediction, the embeddings need to become more meaningful, just as the parameters become more meaningful during training.  
  > **username_0**: is there any code implementation which I can refer to understand more. Or if I want to build that from scratch just to learn how it works  
  > **username_4**: @username_0 check out this tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html  
  
  ---
  
  