Answers.Body: <p>No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms.  Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.</p>
  <p>The embeddings are trained from scratch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-05T21:11:35.623Z
Answers.Text: Maybe you should provide more details about the embeddings used by the transformers (apart from saying that they are learned from scratch).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-06T00:09:00.287Z
Answers.Text: Embeddings in Word2Vec are also trained from scratch
Answers.UserDisplayName: Alex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-06T12:35:04.600Z
Answers.Text: And how are they trained from scratch then? In the tutorials I have read yet that part is never mentioned. Every time just the attention part is focussed and the embedding part is skipped.
Answers.UserDisplayName: Bert Gayus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-11T13:52:12.937Z
Answers.Text: Re. static (Word2Vec) and dynamic/contextual (Transformer) embeddings, for the Transformer you would still use PyTorch nn.Embedding ("static") but the attention mechanism disambiguates homonyms (words with different meaning but spelled the same) using the context of the  words.
Answers.UserDisplayName: 0dB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-04T16:44:13.700Z
Answers.Text: @BertGayus yeah, so true. seem like they treat it as highest secret.
Answers.UserDisplayName: Wuaner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T00:09:01.937Z
Answers.Text: This answer is deceptive. After an input sequence has been tokenized, the transformer's first step will be to use a STATIC matrix to find that token's embedding. Bert Gayus' answer gives some insight into how that static matrix is trained.
Answers.UserDisplayName: David Skarbrevik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-05T20:25:00.707Z
Answers.LastActivityDate: 2021-02-05T21:06:54.713Z
Answers.LastEditDate: 2021-02-05T21:06:54.713Z
Answers.UserDisplayName: Brian O'Donnell
Answers.Body: <p>As &quot;initial&quot;  word embeddings (those without any positional or context information for each word or sub word)  are used from the very beginning  It seems to me that someone has to provide a trained embedding for each word at the very beginning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-15T18:20:23.603Z
Answers.Text: They can and will be trained from random starting values.
Answers.UserDisplayName: 0dB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-01T07:06:32.290Z
Answers.LastActivityDate: 2021-05-01T12:43:56.187Z
Answers.LastEditDate: 2021-05-01T12:43:56.187Z
Answers.UserDisplayName: Raul Alvarez
Answers.Body: <p>I have found a good answer in this blog post <a href="https://glassboxmedicine.com/2019/08/15/the-transformer-attention-is-all-you-need/" rel="noreferrer">The Transformer: Attention Is All You Need</a>:</p>
  <blockquote>
  <p>we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight matrix W.</p>
  </blockquote>
  <blockquote>
  <p>There are two options for dealing with the Pytorch nn.Embedding weight matrix. One option is to initialize it with pre-trained embeddings and keep it fixed, in which case it’s really just a lookup table. Another option is to initialize it randomly, or with pre-trained embeddings, but keep it trainable. In that case the word representations will get refined and modified throughout training because the weight matrix will get refined and modified throughout training.</p>
  <p>The Transformer uses a random initialization of the weight matrix and refines these weights during training – i.e. it learns its own word embeddings.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T15:46:54.467Z
Answers.Text: This response is confusing because it basically says it could be either trained or not trained. But I assume there is a single overall approach taken by the initial transformers paper, gpt-*, BERT, BLOOM, etc. The original Vaswani paper and GPT papers don't mention anything about an initial word embedding, nor do they mention anything about a trainable embedding matrix -- only trainable projections. So which one is it in those specific cases? Is the initial embedding trained or not?
Answers.UserDisplayName: Paul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-10T06:34:02.470Z
Answers.Text: When you set up a transformer you can choose if you want to train from scratch or with pre-trained embeddings, and for the latter, whether it should stay fixed or continue to be trained, just like the answer says. Embeddings and projections are the same concept, the first applied to tokens (non-numbers), the second to numbers, but sometimes you see the terms used synonymously.
Answers.UserDisplayName: 0dB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-06T14:09:41.783Z
Answers.LastActivityDate: 2021-02-06T15:14:23.083Z
Answers.LastEditDate: 2021-02-06T15:14:23.083Z
Answers.UserDisplayName: Bert Gayus
Body: <p>I am currently trying to understand transformers.</p>
  <p>To start, I read <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noreferrer">Attention Is All You Need</a> and also <a href="https://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="noreferrer">this</a> tutorial.</p>
  <p>What makes me wonder is the word embedding used in the model. Is word2vec or GloVe being used? Are the word embeddings trained from scratch?</p>
  <p>In the tutorial linked above, the transformer is implemented from scratch and nn.Embedding from pytorch is used for the embeddings. I looked up this function and didn't understand it well, but I tend to think that the embeddings are trained from scratch, right?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-10T07:15:22.583Z
Comments.Text: If you were to implement a Transformer or Word2Vec yourself with PyTorch, in both cases you will probably be using nn.Embedding. Agreed, the original paper is not clear on this. (The other parts of your question have already been answered well by you and in comments.)
Comments.UserDisplayName: 0dB
Tags: <natural-language-processing><transformer><attention><word-embedding>
Title: What kind of word embedding is used in the original transformer?
UserDisplayName: Bert Gayus
ThreadText: # What kind of word embedding is used in the original transformer?
  
  (Asked by: username_0 on 2021-02-05)
  
  I am currently trying to understand transformers.
  
  To start, I read [Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf) and also [this](https://nlp.seas.harvard.edu/2018/04/03/attention.html) tutorial.
  
  What makes me wonder is the word embedding used in the model. Is word2vec or GloVe being used? Are the word embeddings trained from scratch?
  
  In the tutorial linked above, the transformer is implemented from scratch and nn.Embedding from pytorch is used for the embeddings. I looked up this function and didn't understand it well, but I tend to think that the embeddings are trained from scratch, right?
  
  **Tags:** natural-language-processing, transformer, attention, word-embedding
  > **Comments:**
  > **username_1**: If you were to implement a Transformer or Word2Vec yourself with PyTorch, in both cases you will probably be using nn.Embedding. Agreed, the original paper is not clear on this. (The other parts of your question have already been answered well by you and in comments.)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2021-02-06. Score: 18 | Accepted answer
  
  I have found a good answer in this blog post [The Transformer: Attention Is All You Need](https://glassboxmedicine.com/2019/08/15/the-transformer-attention-is-all-you-need/):
  
  > we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight matrix W.
  
  > There are two options for dealing with the Pytorch nn.Embedding weight matrix. One option is to initialize it with pre-trained embeddings and keep it fixed, in which case it’s really just a lookup table. Another option is to initialize it randomly, or with pre-trained embeddings, but keep it trainable. In that case the word representations will get refined and modified throughout training because the weight matrix will get refined and modified throughout training.
  >
  > The Transformer uses a random initialization of the weight matrix and refines these weights during training – i.e. it learns its own word embeddings.
  
  > **Comments:**
  > **username_8**: This response is confusing because it basically says it could be either trained or not trained. But I assume there is a single overall approach taken by the initial transformers paper, gpt-*, BERT, BLOOM, etc. The original Vaswani paper and GPT papers don't mention anything about an initial word embedding, nor do they mention anything about a trainable embedding matrix -- only trainable projections. So which one is it in those specific cases? Is the initial embedding trained or not?  
  > **username_1**: When you set up a transformer you can choose if you want to train from scratch or with pre-trained embeddings, and for the latter, whether it should stay fixed or continue to be trained, just like the answer says. Embeddings and projections are the same concept, the first applied to tokens (non-numbers), the second to numbers, but sometimes you see the terms used synonymously.  
  
  ---
  
  ### Answer by username_2 on 2021-02-05. Score: 5
  
  No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.
  
  The embeddings are trained from scratch.
  
  > **Comments:**
  > **username_3**: Maybe you should provide more details about the embeddings used by the transformers (apart from saying that they are learned from scratch).  
  > **username_4**: Embeddings in Word2Vec are also trained from scratch  
  > **username_0**: And how are they trained from scratch then? In the tutorials I have read yet that part is never mentioned. Every time just the attention part is focussed and the embedding part is skipped.  
  > **username_6**: This answer is deceptive. After an input sequence has been tokenized, the transformer's first step will be to use a STATIC matrix to find that token's embedding. Bert Gayus' answer gives some insight into how that static matrix is trained.  
  > **username_1**: Re. static (Word2Vec) and dynamic/contextual (Transformer) embeddings, for the Transformer you would still use PyTorch nn.Embedding ("static") but the attention mechanism disambiguates homonyms (words with different meaning but spelled the same) using the context of the  words.  
  > **username_5**: @BertGayus yeah, so true. seem like they treat it as highest secret.  
  
  ---
  
  ### Answer by username_7 on 2021-05-01. Score: 0
  
  As "initial" word embeddings (those without any positional or context information for each word or sub word) are used from the very beginning It seems to me that someone has to provide a trained embedding for each word at the very beginning.
  
  > **Comments:**
  > **username_1**: They can and will be trained from random starting values.  
  
  ---
  
  