Answers.Body: <p>Transformers do still use pre-trained embeddings but these are then given context by the positional encoding and self-attention blocks of the architecture.</p>
  <p>I will explain in the context of a sequence-to-sequence transformer used for translating a sentence from one language to another (e.g. that used in <em>Attention is All You Need</em>):</p>
  <p>At inference time the input sequence process (i.e. the encoder) is the same as during training time: each word in the input sequence is mapped to the embedding vector for that word, and 'given context' by the positional encoding and encoder self-attention blocks.</p>
  <p>In the decoder, at inference time we are predicting the translation of each word in the input sequence, one at a time. So instead of feeding in the target translated sentence to the decoder, we feed in an empty sequence with a start-of-sentence token. In much the same way as in the encoder, each token in this sentence is given context by the positional encoding and self-attention blocks. The next word in the output sequence is then predicted by the rest of the decoder layers and this is appended to the output sentence. This output sentence (now consisting of the start-of-sentence token and the first word(s)) is then fed into the decoder to predict the next word, until the end-of-sentence token is predicted.</p>
  <p>So in answer to your question, the blocks of the transformer that give the word embeddings context are the same during training and inference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T13:06:52.947Z
Answers.Text: So to sum up, the word embeddings are the same as in training time, the positional encoding differs, that's right?
Answers.UserDisplayName: Julien
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T15:20:12.430Z
Answers.Text: Yes - the word embeddings are the same in training and inference. The word embedding stage is just a lookup from word to pre-trained embedding representation for that word. 
  
  As you say, these are then transformed by the positional encoding (depending on where the word is in the sentence) and by the attention layers (i.e. weights of the key, value and query matrices that were learnt during training).
Answers.UserDisplayName: Atticus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T18:09:14.310Z
Answers.Text: it might also help to point out that that self-attention isn't technically an embedding and isn't learned but computed on the fly for each input (whether at training or at inference).
Answers.UserDisplayName: waxalas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T12:00:17.553Z
Answers.LastActivityDate: 2022-11-03T12:00:17.553Z
Answers.UserDisplayName: Atticus
Answers.Body: <p>Embedding doesn't provide context like self-attention does. Embedding is what you do to represent your training data (words or pictures) into numerical vectors.</p>
  <p>Word embedding is learned and will be the same at training and inference. For example, &quot;Transformers are cool&quot; will be embedded as [122, 5, 43], while &quot;How are you&quot; will be embedded as [10, 5, 2].</p>
  <p>Positional embedding is a deterministic function. It is not only the same at training and inference, but it is also the same on every sample in your training data. Both &quot;Transformers are cool&quot; and &quot;How are you&quot; will have positional embedding [1, 2, 3]. (Positional embedding could also be a learned transformation, but in <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="nofollow noreferrer">the original paper</a>, they didn't find much improvement in using one).</p>
  <p>Now, self-attention is what gives inputs context (for example, &quot;are&quot; and &quot;you&quot; are related and conjugated accordingly in the sentence &quot;How are you&quot;). Self-attention is a matrix of weights which is indeed dynamic since it is computed on the fly during training. But you can also compute self-attention on your new predictor on the fly during inference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T18:58:55.150Z
Answers.LastActivityDate: 2022-11-03T18:58:55.150Z
Answers.UserDisplayName: waxalas
Body: <p>In Word2Vec, the embeddings don't depend on the context.</p>
  <p>But in Transformers, the embeddings depend on the context.</p>
  <p>So how are the words' embeddings set at inference time?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-17T14:39:25.497Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <transformer><word-embedding><inference>
Title: How do Transformers compute the words embeddings at inference time since the embeddings are dynamic?
UserDisplayName: Julien
ThreadText: # How do Transformers compute the words embeddings at inference time since the embeddings are dynamic?
  
  (Asked by: username_0 on 2022-10-17)
  
  In Word2Vec, the embeddings don't depend on the context.
  
  But in Transformers, the embeddings depend on the context.
  
  So how are the words' embeddings set at inference time?
  
  **Tags:** transformer, word-embedding, inference
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-11-03. Score: 2
  
  Transformers do still use pre-trained embeddings but these are then given context by the positional encoding and self-attention blocks of the architecture.
  
  I will explain in the context of a sequence-to-sequence transformer used for translating a sentence from one language to another (e.g. that used in *Attention is All You Need*):
  
  At inference time the input sequence process (i.e. the encoder) is the same as during training time: each word in the input sequence is mapped to the embedding vector for that word, and 'given context' by the positional encoding and encoder self-attention blocks.
  
  In the decoder, at inference time we are predicting the translation of each word in the input sequence, one at a time. So instead of feeding in the target translated sentence to the decoder, we feed in an empty sequence with a start-of-sentence token. In much the same way as in the encoder, each token in this sentence is given context by the positional encoding and self-attention blocks. The next word in the output sequence is then predicted by the rest of the decoder layers and this is appended to the output sentence. This output sentence (now consisting of the start-of-sentence token and the first word(s)) is then fed into the decoder to predict the next word, until the end-of-sentence token is predicted.
  
  So in answer to your question, the blocks of the transformer that give the word embeddings context are the same during training and inference.
  
  > **Comments:**
  > **username_0**: So to sum up, the word embeddings are the same as in training time, the positional encoding differs, that's right?  
  > **username_2**: Yes - the word embeddings are the same in training and inference. The word embedding stage is just a lookup from word to pre-trained embedding representation for that word.   As you say, these are then transformed by the positional encoding (depending on where the word is in the sentence) and by the attention layers (i.e. weights of the key, value and query matrices that were learnt during training).  
  > **username_3**: it might also help to point out that that self-attention isn't technically an embedding and isn't learned but computed on the fly for each input (whether at training or at inference).  
  
  ---
  
  ### Answer by username_3 on 2022-11-03. Score: 1
  
  Embedding doesn't provide context like self-attention does. Embedding is what you do to represent your training data (words or pictures) into numerical vectors.
  
  Word embedding is learned and will be the same at training and inference. For example, "Transformers are cool" will be embedded as [122, 5, 43], while "How are you" will be embedded as [10, 5, 2].
  
  Positional embedding is a deterministic function. It is not only the same at training and inference, but it is also the same on every sample in your training data. Both "Transformers are cool" and "How are you" will have positional embedding [1, 2, 3]. (Positional embedding could also be a learned transformation, but in [the original paper](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf), they didn't find much improvement in using one).
  
  Now, self-attention is what gives inputs context (for example, "are" and "you" are related and conjugated accordingly in the sentence "How are you"). Self-attention is a matrix of weights which is indeed dynamic since it is computed on the fly during training. But you can also compute self-attention on your new predictor on the fly during inference.
  
  
  ---
  
  