Answers.Body: <p>1 - yes, they are considered the conditioning of the model, so they are always known</p>
  <p>2&amp;3 - at inference, you start with a single <code>&lt;SOS&gt;</code> token, which will be forwarded through the decoder, and conditioned on the encoder output, to get a final softmax distribution at the end; at this point, you sample a token from such distribution, you concatenate it to the input to get <code>&lt;SOS&gt;sampled_token</code>, and you feed this new string to the decoder (which will be than forwarded, conditioned, and will produce a softmax from which you'll sample the next) and the end is just determined when the sampled token is the end of sentece <code>&lt;EOS&gt;</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T14:28:48.530Z
Answers.Text: Wouldn't the decoder produce two word predictions in the 2nd iteration, three in the 3rd and so on then? Would you then just take the one for the new word? So ignore the first prediction in the 2nd iteration, ignore the first two in the 3rd and so on?
Answers.UserDisplayName: Felix Crazzolara
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T17:00:38.007Z
Answers.Text: @FelixCrazzolara depends on how you treat the input, but no, if you just input a string,  you predict only the next, it's during training that you have that behavior because you want to train it in one forward pass
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T13:01:37.870Z
Answers.LastActivityDate: 2023-11-14T13:01:37.870Z
Answers.UserDisplayName: Alberto
Body: <p>Let's consider language translation and let <span class="math-container">$I_1,\ldots,I_{N_i}$</span> be the <span class="math-container">$N_i$</span> input tokens. My understanding is that the encoder produces <span class="math-container">$N_i$</span> embeddings, which I will refer to as <span class="math-container">$E_1,\ldots,E_{N_i}$</span>. In what follows, let's consider what happens at inference time and not during training. My understanding is that the decoder has multiple inputs and they possibly change when he produces the different output tokens. If considering a single decoder module, I guess it makes sense to distinguish between the external inputs that are fed into the multi-head attention module and the ones that go into the masked multi-head attention module as shown below.<br>
  <a href="https://i.sstatic.net/rdHA8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rdHA8.png" alt="enter image description here" /></a><br>
  Here are now my questions:</p>
  <ol>
  <li>Is it correct that what comes from the left into the multi-head attention modules are always the encoder embeddings <span class="math-container">$E_1,\ldots,E_{N_i}$</span>?</li>
  <li>I believe to understand that the bottom input to the masked multi-head attention module is not always the same and depends on the iteration we find ourselves in, i.e. which output token we predict and which decoder module is considered if there are multiple. More precisely, is it true that for the first prediction, i.e. when predicting the first output token, the sole input to the first decoder module is the embedding of the 'START' token plus positional encoding?</li>
  <li>What are the bottom inputs of the different decoder modules when predicting the individual output tokens?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-17T00:44:02.130Z
Comments.Text: Please, ask only one question per post, even if they are related. As a rule of thumb, if you can't put a single question in the title that summaries your problem/question, then it means you need to split the post into multiple ones.  "Understanding the transformer at inference time" looks like a too big problem for a single post. So, maybe you could change your title to ask such specific question. Thanks.
Comments.UserDisplayName: nbro
Tags: <transformer>
Title: Understanding the transformer at inference time
UserDisplayName: Felix Crazzolara
ThreadText: # Understanding the transformer at inference time
  
  (Asked by: username_0 on 2023-11-14)
  
  Let's consider language translation and let $I\_1,\ldots,I\_{N\_i}$ be the $N\_i$ input tokens. My understanding is that the encoder produces $N\_i$ embeddings, which I will refer to as $E\_1,\ldots,E\_{N\_i}$. In what follows, let's consider what happens at inference time and not during training. My understanding is that the decoder has multiple inputs and they possibly change when he produces the different output tokens. If considering a single decoder module, I guess it makes sense to distinguish between the external inputs that are fed into the multi-head attention module and the ones that go into the masked multi-head attention module as shown below.  
  [![enter image description here](https://i.sstatic.net/rdHA8.png)](https://i.sstatic.net/rdHA8.png)  
  Here are now my questions:
  
  1. Is it correct that what comes from the left into the multi-head attention modules are always the encoder embeddings $E\_1,\ldots,E\_{N\_i}$?
  2. I believe to understand that the bottom input to the masked multi-head attention module is not always the same and depends on the iteration we find ourselves in, i.e. which output token we predict and which decoder module is considered if there are multiple. More precisely, is it true that for the first prediction, i.e. when predicting the first output token, the sole input to the first decoder module is the embedding of the 'START' token plus positional encoding?
  3. What are the bottom inputs of the different decoder modules when predicting the individual output tokens?
  
  **Tags:** transformer
  
  > **Comments:**
  > **username_1**: Please, ask only one question per post, even if they are related. As a rule of thumb, if you can't put a single question in the title that summaries your problem/question, then it means you need to split the post into multiple ones.  "Understanding the transformer at inference time" looks like a too big problem for a single post. So, maybe you could change your title to ask such specific question. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-11-14. Score: 1
  
  1 - yes, they are considered the conditioning of the model, so they are always known
  
  2&3 - at inference, you start with a single `<SOS>` token, which will be forwarded through the decoder, and conditioned on the encoder output, to get a final softmax distribution at the end; at this point, you sample a token from such distribution, you concatenate it to the input to get `<SOS>sampled_token`, and you feed this new string to the decoder (which will be than forwarded, conditioned, and will produce a softmax from which you'll sample the next) and the end is just determined when the sampled token is the end of sentece `<EOS>`
  
  > **Comments:**
  > **username_0**: Wouldn't the decoder produce two word predictions in the 2nd iteration, three in the 3rd and so on then? Would you then just take the one for the new word? So ignore the first prediction in the 2nd iteration, ignore the first two in the 3rd and so on?  
  > **username_2**: @username_0 depends on how you treat the input, but no, if you just input a string,  you predict only the next, it's during training that you have that behavior because you want to train it in one forward pass  
  
  ---
  
  