Answers.Body: <p>The main issue during training is that you haven't right-shifted the input of the decoder, which is probably why you set the diagonals of <code>mask</code> to -inf (when it should be <span class="math-container">$0$</span>).</p>
  <p>Also, just an FYI, although you haven't focused on evaluation/prediction yet, I will explain the evaluation/prediction here as well for completeness, since it works so differently than training, and also since you will need it when generating the graphs for debugging.</p>
  <h2>Training</h2>
  <p>Both <code>tgt</code> and <code>tgt_mask</code> should be changed to simulate auto-regressive properties.</p>
  <p>You are feeding in <code>tgt</code> as the input to the decoder, where <code>tgt</code> is the ground truth target sequence to predict. <code>tgt</code> should have dimension <code>length(sequence)</code> x <code>batchSize</code> x <span class="math-container">$\|dict\|$</span>. Additionally, you are feeding in mask where the diagonals are <code>-inf</code>.</p>
  <p>Instead, you should do the following:</p>
  <ul>
  <li>Add 2 new special tokens to your vocabulary that signals the start and end of decoding, i.e. <code>&lt;START&gt;</code> and <code>&lt;END&gt;</code>. So your vocabulary will need to be extended to 16 events.</li>
  <li>Every input to the decoder (training, evaluation, prediciton) should have <code>&lt;START&gt;</code> as the first token. So during training, when you want to feed in <code>tgt</code> to the decoder, you should right-shift <code>tgt</code> by adding in the <code>&lt;START&gt;</code> token at the beginning.
  <ul>
  <li><code>tgt_shifted</code> will now be of dimension <code>length(sequence)+1</code> x <code>batchSize</code> x <span class="math-container">$\| dict \|$</span></li>
  <li><code>tgt_shifted_mask</code> will now be of dimension <code>length(sequence)+1</code> x <code>length(sequence)+1</code>. Diagonal should be all <span class="math-container">$0$</span></li>
  <li>the inputs to the decoder should be <code>tgt_shifted</code>, <code>tgt_shifted_mask</code>, and <code>memory</code></li>
  <li>the output of the decoder will have dimension <code>length(sequence)+1</code> x <code>batchSize</code> x <span class="math-container">$\| dict\|$</span>, and will not be right-shifted because it will not have <code>&lt;START&gt;</code> as the first word. But it should have <code>&lt;END&gt;</code> as the last word.</li>
  <li>Append <code>&lt;END&gt;</code> to the ground truth <code>tgt</code>, and turn it into one-hot encoding, so <code>tgt</code> should have dimension <code>length(sequence)+1</code> x <code>batchSize</code> x <span class="math-container">$\|dict\|$</span>. Your loss operator should be some sort of element-wise comparison between <code>tgt</code> and the output of the decoder.</li>
  </ul>
  </li>
  </ul>
  <p>Note that you only run the decoder once per training batch during training, i.e. the decoder simultaneously predicts the logits of all <code>length(sequence)</code> tokens at the same time. On the other hand, during evaluation/prediction, you must run the decoder <code>length(sequence)</code> times, since you can only use the decoder to predict one token at a time.</p>
  <h2>Evaluation/Prediction</h2>
  <p>During evaluation/prediction, the model should generate its own sentence entirely from scratch, one word at a time:</p>
  <ol>
  <li><code>out</code> should be initialized as a single token, <code>&lt;START&gt;</code>, and have dimensions <span class="math-container">$1$</span>  x <code>batchSize</code> x <span class="math-container">$\|dict\|$</span>.</li>
  <li>Do the following in a loop until termination or max output length is reached:
  <ol>
  <li>Generate a <code>out_mask</code> for the current version of <code>out</code>. <code>out_mask</code> should be a matrix with dimension <code>length(out)</code> x <code>length(out)</code>. Diagonal should be all <span class="math-container">$0$</span></li>
  <li>Pass <code>out</code>, <code>out_mask</code>, and <code>memory</code> into the decoder layer. The output of the decoder will be a <code>length(out)</code> x <code>batchSize</code> x <span class="math-container">$d_{model}$</span> tensor (note that in your case, <span class="math-container">$d_{model} = \|dict\|$</span>), which indicates <code>tokenIdx</code> x <code>batchIdx</code> x <code>vocabularyIdx</code></li>
  <li>Convert the last time step (e.g. <code>decoderOutput[-1,:,:]</code> if all sequences in your batch have the same length) into probabilities, and take all the events that surpasses some probability threshold and set them to <span class="math-container">$1$</span>. Set all other events to <span class="math-container">$0$</span>. This is the new time step that the decoder layer predicts. So add this time step to <code>out</code> to append it's length by <span class="math-container">$1$</span>.</li>
  <li>Repeat the previous 3 steps in a loop until you reach the <code>&lt;END&gt;</code> token, which signals the end of decoding, or until the max output length is reached.</li>
  </ol>
  </li>
  </ol>
  <p>During evaluation, you can calculate the validation loss by using the logits of only the final iteration of the decoder (which should have dimension <code>length(sequence)+1</code> x <code>batchSize</code> x <span class="math-container">$\|dict\|$</span>), and compare those logits with the ground truth with the <code>&lt;END&gt;</code> token added at the end (resulting in dimensions <code>length(sequence)+1</code> x <code>batchSize</code> x <span class="math-container">$\|dict\|$</span>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T16:16:33.683Z
Answers.Text: Sorry, I wasn't clear about this - I get this issue specifically during training, not inference time. Here's how I understand training should go: for an output token at timestamp `t` we  give a model the whole `src` sequence as well as `tgt[0 : t-1]`. It's not like generating the whole sentence in English given a sentence in French, but instead like predicting the next word user is going to write given previous sentence and previous words in this sentence user has already written. IIUC, for each `out[t]` mask should hide `tgt[t : ]` even though training loop sees the whole `tgt`.
Answers.UserDisplayName: ffriend
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T16:26:11.120Z
Answers.Text: But visualization in the question shows that `out[t]` is extremely similar to `tgt[t]`. I can think of 2 reasons for this - overfitting or wrong usage of the mask. I have already checked that the visualization is done on the test set, so the problem must be the mask.
Answers.UserDisplayName: ffriend
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T16:33:19.003Z
Answers.Text: Perhaps, the question can be reduced to: is `decoder(tgt, memory, subsequent_mask)` the correct way to predict all outputs in parallel, restricting each `out[t]` to see only true values `tgt[ : t - 1]`?
Answers.UserDisplayName: ffriend
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T22:50:33.987Z
Answers.Text: Thanks for the clarification. I have edited my answer which should help with all the things you mentioned. Let me know if there's anything further to clarify.
Answers.UserDisplayName: user3667125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T22:54:54.240Z
Answers.Text: Thanks! I appreciate your time and patience. Indeed, I used a shifted mask instead of shifted target for the same reason you described, but I could have made a mistake in it and your version looks more common anyway. So I'm going to update the code as you suggest and come back with the result!
Answers.UserDisplayName: ffriend
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-03T22:47:24.220Z
Answers.Text: After shifting the target the prediction during training looks much more realistic - see the edited post for visualization. I still have some doubts regarding my implementation, but anyway, your answer is a great tutorial for people coming here with similar issues. Thanks a lot for your help!
Answers.UserDisplayName: ffriend
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-04T02:07:04.750Z
Answers.Text: Glad it helped! One more thing I'd verify is why `out[t]` resembles `tgt[t-1]` though -- it does seem weird. Wouldn't be surprised if there's an off-by-one somewhere, since we shift things so many times by 1 token.
Answers.UserDisplayName: user3667125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-04T20:01:55.180Z
Answers.Text: It's pretty natural for my task to have `tgt[t]` resembling `tgt[t - 1]`, so the model is expected to learn something like this. But yes, having a similar mistake previously I look at it very carefully :)
Answers.UserDisplayName: ffriend
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-09T03:28:17.227Z
Answers.Text: @user3667125 so should we also using the mask during the testing time? I thought during the testing the masks should be set to None, am I wrong?
Answers.UserDisplayName: Alex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-22T16:09:15.127Z
Answers.Text: @Alex I think [this](https://datascience.stackexchange.com/a/81492/65120) answers your question.
Answers.UserDisplayName: tjysdsg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T02:44:14.230Z
Answers.LastActivityDate: 2021-01-04T02:03:31.007Z
Answers.LastEditDate: 2021-01-04T02:03:31.007Z
Answers.UserDisplayName: user3667125
Body: <p>I try to apply Transformers to an unusual use case - predict the next user session based on the previous one. A user session is described by a list of events per second, e.g. whether the user watches a particular video, clicks a specific button, etc. Typical sessions are around 20-30 seconds, I pad them to 45 seconds. Here's a visual example of 2 subsequent sessions:</p>
  <p><a href="https://i.sstatic.net/ndcOw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ndcOw.png" alt="enter image description here" /></a></p>
  <p><code>x</code> axis is time in seconds, <code>y</code> axis is the list of events (black line divides the 2 sessions). I extend the vocabulary with 2 additional tokens - start and end of a session (<code>&lt;sos&gt;</code> and <code>&lt;eos&gt;</code>), where <code>&lt;sos&gt;</code> is a one-hot vector at the very beginning and <code>&lt;eos&gt;</code> - similar vector at the end of the session (which makes this long red line).</p>
  <p>Now I use these extended vectors of events as <em>embeddings</em> and want to train a Transformer model to predict the next events in the current session based on previous events in this (target) session and all events in the previous (source) session. So pretty much like seq2seq autoregressive models, but in a bit unusual settings.</p>
  <p>Here's the problem. When I train a Transformer using the built-in PyTorch components and square subsequent mask for the target, my generated (<em>during training</em>) output is too good to be true:</p>
  <p><a href="https://i.sstatic.net/hdCym.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hdCym.png" alt="enter image description here" /></a></p>
  <p>Although there's some noise, many event vectors in the output are modeled exactly as in the target. After checking train-val-test split is correct, my best guess is that the model cheats by attending to the same day in the target, which the mask should have prevented. The mask is (5x5 version for brevity):</p>
  <pre><code>[[0., -inf, -inf, -inf, -inf],
   [0., -inf, -inf, -inf, -inf],
   [0., 0., -inf, -inf, -inf],
   [0., 0., 0., -inf, -inf],
   [0., 0., 0., 0., -inf]]
  </code></pre>
  <p>Note that since I use <code>&lt;sos&gt;</code> in both - source and target - <code>mask[i, i]</code> is set to <code>-inf</code> (except for <code>mask[0, 0]</code> for numerical reasons), so the output timestamp <code>i</code> should not attend to the target timestamp <code>i</code>.</p>
  <p>The code for the model's <code>forward</code> method:</p>
  <pre><code>def forward(self, src, tgt):
      memory = self.encoder(src)
      out = self.decoder(tgt, memory, self.tgt_mask.type_as(tgt))
      out = torch.sigmoid(out)
      return out
  </code></pre>
  <p>I also tried to avoid the target mask altogether and set it to all <code>-inf</code> (again, except for the first column for numerical stability), but the result is always the same.</p>
  <p>Am I using the mask the wrong way? If the mask looks fine, what other reasons could lead to such a &quot;perfect&quot; result?</p>
  <hr />
  <p>After shifting the target to the right as suggested in the accepted answer I get the following result:</p>
  <p><a href="https://i.sstatic.net/iXT00.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iXT00.jpg" alt="enter image description here" /></a></p>
  <p>Which is much more realistic. One suspicious thing is that <code>out[t]</code> now resembles <code>tgt[t - 1]</code>, but it can be explained by the fact that the user state tends to be &quot;sticky&quot;, e.g. if a user watches a video at <code>t - 1</code>, most likely he will watch it at <code>t</code> as well.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-31T18:11:42.273Z
Comments.Text: When you say "I use these extended vectors of events as embeddings", do you mean that you do not learn the initial $\|dict\|$ x $d_{model}$ weight matrix to embed the inputs, but instead feed the inputs raw into the first layer of the transformer? And do you perform any positional encoding? Also, what is the dimension of your model ($d_{model}$ in some papers), and what is the dimension of your mask?
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-31T18:23:34.713Z
Comments.Text: It would also help if we can read the code of your entire Transformer model. Something like the code under "Define the Model" section here (https://pytorch.org/tutorials/beginner/transformer_tutorial.html). Would help us clarify some details like what is self.encoder, self.decoder, etc.
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-31T18:42:21.433Z
Comments.Text: @user3667125 Here's the [code](https://pastebin.pl/view/8a60d862) for the whole model. It's a PyTorch Lightning module, but you can safely ignore all methods except for `__init__()` and `forward()`. The loss should also be irrelevant. Regarding "embeddings", yes, you got it right: my sequences are not tokens, but instead vectors of events, so I decided to use them directly instead of usual embeddings, without learnable matrix or positional encoding. $d_{model} = 14$, sequences are padded to 45 seconds and thus mask is 45x45.
Comments.UserDisplayName: ffriend
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-31T20:24:51.283Z
Comments.Text: Can you confirm if a single instance have dimension $45$ x $14$ (and therefore a batch has dimension $45$ x $batchSize$ x $14$)? And can you explain why the mask diagonal entries mask[i, i] = -inf? Normally they would be 0.
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-31T20:39:56.720Z
Comments.Text: Also, what is the reasoning behind type-casting the mask? self.tgt_mask.type_as(tgt) -- have you tried without using the .type_as typecasting?
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T00:11:58.897Z
Comments.Text: @user3667125 That's correct, single instances of both - source and target - have shape $45 \times 14$. As far as I understand, PyTorch uses an additive mask, so `-inf` is what is suggested in the tutorial you pointed to earlier as well as in the [Transformer module docs](https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html#torch.nn.Transformer.generate_square_subsequent_mask). I use `.type_as()` to convert mask to the same _device_ because for whatever reason Lightning doesn't move it to GPU automatically. But on CPU and without `.type_as()` the problem still persists.
Comments.UserDisplayName: ffriend
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T00:13:23.957Z
Comments.Text: By the way, thank you for your quick replies!
Comments.UserDisplayName: ffriend
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T03:36:12.540Z
Comments.Text: My understanding of the tutorial I linked is that generate_square_subsequent_mask() creates a mask matrix similar to yours, but the diagonals mask[i, i] are all 0. This is so that the current token/time step can attend to itself, as well as the past tokens/time steps. Can you try with the mask where all the diagonals are 0? Also, I'm not sure on the .type_as() usage, since the docs says .type_as() is meant to typecast the tensor (e.g. from an int to a float). Can you try moving the device via .to(device), and remove the .type_as() function entirely?
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T03:37:46.980Z
Comments.Text: Also, just to confirm, both src and tgt have dimensions $45$ x $batchSize$ x $14$ right? And not $batchSize$ x $45$ x $14$? The dimensionality is a bit counter-intuitive to me, but that seems to be what PyTorch is expecting for the Transformer functions.
Comments.UserDisplayName: user3667125
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T11:36:53.980Z
Comments.Text: @user3667125 Yep, source, target and generated output all have size $45 \times batchSize \times 14$. Removing `.type_as()` didn't change the result. Mask with all zeros on the main diagonal (and below it) didn't change the result too. Shouldn't we avoid zeros (i.e. unmasked positions) on the main diagonal? As far as I understand, in autoregressive settings when generating output at position `i` we should avoid attending to target at position `i`, i.e. set `mask[i, I] = -inf`.
Comments.UserDisplayName: ffriend
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T11:38:43.307Z
Comments.Text: Note that in the tutorial they use the mask for the source sequence, while I want to attend to the whole source (no mask) and only to the previous positions in the target sequence (square matrix with -inf on the main diagonal and below). But the results suggest I'm doing something wrong.
Comments.UserDisplayName: ffriend
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-02T02:50:36.810Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/117935/discussion-between-user3667125-and-ffriend).
Comments.UserDisplayName: user3667125
Tags: <pytorch><transformer>
Title: Transformers: How to use the target mask properly?
UserDisplayName: ffriend
ThreadText: # Transformers: How to use the target mask properly?
  
  (Asked by: username_0 on 2020-12-31)
  
  I try to apply Transformers to an unusual use case - predict the next user session based on the previous one. A user session is described by a list of events per second, e.g. whether the user watches a particular video, clicks a specific button, etc. Typical sessions are around 20-30 seconds, I pad them to 45 seconds. Here's a visual example of 2 subsequent sessions:
  
  [![enter image description here](https://i.sstatic.net/ndcOw.png)](https://i.sstatic.net/ndcOw.png)
  
  `x` axis is time in seconds, `y` axis is the list of events (black line divides the 2 sessions). I extend the vocabulary with 2 additional tokens - start and end of a session (`<sos>` and `<eos>`), where `<sos>` is a one-hot vector at the very beginning and `<eos>` - similar vector at the end of the session (which makes this long red line).
  
  Now I use these extended vectors of events as *embeddings* and want to train a Transformer model to predict the next events in the current session based on previous events in this (target) session and all events in the previous (source) session. So pretty much like seq2seq autoregressive models, but in a bit unusual settings.
  
  Here's the problem. When I train a Transformer using the built-in PyTorch components and square subsequent mask for the target, my generated (*during training*) output is too good to be true:
  
  [![enter image description here](https://i.sstatic.net/hdCym.png)](https://i.sstatic.net/hdCym.png)
  
  Although there's some noise, many event vectors in the output are modeled exactly as in the target. After checking train-val-test split is correct, my best guess is that the model cheats by attending to the same day in the target, which the mask should have prevented. The mask is (5x5 version for brevity):
  
  ```
  [[0., -inf, -inf, -inf, -inf],
   [0., -inf, -inf, -inf, -inf],
   [0., 0., -inf, -inf, -inf],
   [0., 0., 0., -inf, -inf],
   [0., 0., 0., 0., -inf]]
  
  ```
  
  Note that since I use `<sos>` in both - source and target - `mask[i, i]` is set to `-inf` (except for `mask[0, 0]` for numerical reasons), so the output timestamp `i` should not attend to the target timestamp `i`.
  
  The code for the model's `forward` method:
  
  ```
  def forward(self, src, tgt):
      memory = self.encoder(src)
      out = self.decoder(tgt, memory, self.tgt_mask.type_as(tgt))
      out = torch.sigmoid(out)
      return out
  
  ```
  
  I also tried to avoid the target mask altogether and set it to all `-inf` (again, except for the first column for numerical stability), but the result is always the same.
  
  Am I using the mask the wrong way? If the mask looks fine, what other reasons could lead to such a "perfect" result?
  
  ---
  
  After shifting the target to the right as suggested in the accepted answer I get the following result:
  
  [![enter image description here](https://i.sstatic.net/iXT00.jpg)](https://i.sstatic.net/iXT00.jpg)
  
  Which is much more realistic. One suspicious thing is that `out[t]` now resembles `tgt[t - 1]`, but it can be explained by the fact that the user state tends to be "sticky", e.g. if a user watches a video at `t - 1`, most likely he will watch it at `t` as well.
  
  **Tags:** pytorch, transformer
  
  > **Comments:**
  > **username_1**: When you say "I use these extended vectors of events as embeddings", do you mean that you do not learn the initial $\|dict\|$ x $d_{model}$ weight matrix to embed the inputs, but instead feed the inputs raw into the first layer of the transformer? And do you perform any positional encoding? Also, what is the dimension of your model ($d_{model}$ in some papers), and what is the dimension of your mask?  
  > **username_1**: It would also help if we can read the code of your entire Transformer model. Something like the code under "Define the Model" section here (https://pytorch.org/tutorials/beginner/transformer_tutorial.html). Would help us clarify some details like what is self.encoder, self.decoder, etc.  
  > **username_0**: @username_1 Here's the [code](https://pastebin.pl/view/8a60d862) for the whole model. It's a PyTorch Lightning module, but you can safely ignore all methods except for `__init__()` and `forward()`. The loss should also be irrelevant. Regarding "embeddings", yes, you got it right: my sequences are not tokens, but instead vectors of events, so I decided to use them directly instead of usual embeddings, without learnable matrix or positional encoding. $d_{model} = 14$, sequences are padded to 45 seconds and thus mask is 45x45.  
  > **username_1**: Can you confirm if a single instance have dimension $45$ x $14$ (and therefore a batch has dimension $45$ x $batchSize$ x $14$)? And can you explain why the mask diagonal entries mask[i, i] = -inf? Normally they would be 0.  
  > **username_1**: Also, what is the reasoning behind type-casting the mask? self.tgt_mask.type_as(tgt) -- have you tried without using the .type_as typecasting?  
  > **username_0**: @username_1 That's correct, single instances of both - source and target - have shape $45 \times 14$. As far as I understand, PyTorch uses an additive mask, so `-inf` is what is suggested in the tutorial you pointed to earlier as well as in the [Transformer module docs](https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html#torch.nn.Transformer.generate_square_subsequent_mask). I use `.type_as()` to convert mask to the same _device_ because for whatever reason Lightning doesn't move it to GPU automatically. But on CPU and without `.type_as()` the problem still persists.  
  > **username_0**: By the way, thank you for your quick replies!  
  > **username_1**: My understanding of the tutorial I linked is that generate_square_subsequent_mask() creates a mask matrix similar to yours, but the diagonals mask[i, i] are all 0. This is so that the current token/time step can attend to itself, as well as the past tokens/time steps. Can you try with the mask where all the diagonals are 0? Also, I'm not sure on the .type_as() usage, since the docs says .type_as() is meant to typecast the tensor (e.g. from an int to a float). Can you try moving the device via .to(device), and remove the .type_as() function entirely?  
  > **username_1**: Also, just to confirm, both src and tgt have dimensions $45$ x $batchSize$ x $14$ right? And not $batchSize$ x $45$ x $14$? The dimensionality is a bit counter-intuitive to me, but that seems to be what PyTorch is expecting for the Transformer functions.  
  > **username_0**: @username_1 Yep, source, target and generated output all have size $45 \times batchSize \times 14$. Removing `.type_as()` didn't change the result. Mask with all zeros on the main diagonal (and below it) didn't change the result too. Shouldn't we avoid zeros (i.e. unmasked positions) on the main diagonal? As far as I understand, in autoregressive settings when generating output at position `i` we should avoid attending to target at position `i`, i.e. set `mask[i, I] = -inf`.  
  > **username_0**: Note that in the tutorial they use the mask for the source sequence, while I want to attend to the whole source (no mask) and only to the previous positions in the target sequence (square matrix with -inf on the main diagonal and below). But the results suggest I'm doing something wrong.  
  > **username_1**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/117935/discussion-between-username_1-and-username_0).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-02. Score: 7 | Accepted answer
  
  The main issue during training is that you haven't right-shifted the input of the decoder, which is probably why you set the diagonals of `mask` to -inf (when it should be $0$).
  
  Also, just an FYI, although you haven't focused on evaluation/prediction yet, I will explain the evaluation/prediction here as well for completeness, since it works so differently than training, and also since you will need it when generating the graphs for debugging.
  
  ## Training
  
  Both `tgt` and `tgt_mask` should be changed to simulate auto-regressive properties.
  
  You are feeding in `tgt` as the input to the decoder, where `tgt` is the ground truth target sequence to predict. `tgt` should have dimension `length(sequence)` x `batchSize` x $\|dict\|$. Additionally, you are feeding in mask where the diagonals are `-inf`.
  
  Instead, you should do the following:
  
  * Add 2 new special tokens to your vocabulary that signals the start and end of decoding, i.e. `<START>` and `<END>`. So your vocabulary will need to be extended to 16 events.
  * Every input to the decoder (training, evaluation, prediciton) should have `<START>` as the first token. So during training, when you want to feed in `tgt` to the decoder, you should right-shift `tgt` by adding in the `<START>` token at the beginning.
    + `tgt_shifted` will now be of dimension `length(sequence)+1` x `batchSize` x $\| dict \|$
    + `tgt_shifted_mask` will now be of dimension `length(sequence)+1` x `length(sequence)+1`. Diagonal should be all $0$
    + the inputs to the decoder should be `tgt_shifted`, `tgt_shifted_mask`, and `memory`
    + the output of the decoder will have dimension `length(sequence)+1` x `batchSize` x $\| dict\|$, and will not be right-shifted because it will not have `<START>` as the first word. But it should have `<END>` as the last word.
    + Append `<END>` to the ground truth `tgt`, and turn it into one-hot encoding, so `tgt` should have dimension `length(sequence)+1` x `batchSize` x $\|dict\|$. Your loss operator should be some sort of element-wise comparison between `tgt` and the output of the decoder.
  
  Note that you only run the decoder once per training batch during training, i.e. the decoder simultaneously predicts the logits of all `length(sequence)` tokens at the same time. On the other hand, during evaluation/prediction, you must run the decoder `length(sequence)` times, since you can only use the decoder to predict one token at a time.
  
  ## Evaluation/Prediction
  
  During evaluation/prediction, the model should generate its own sentence entirely from scratch, one word at a time:
  
  1. `out` should be initialized as a single token, `<START>`, and have dimensions $1$ x `batchSize` x $\|dict\|$.
  2. Do the following in a loop until termination or max output length is reached:
     1. Generate a `out_mask` for the current version of `out`. `out_mask` should be a matrix with dimension `length(out)` x `length(out)`. Diagonal should be all $0$
     2. Pass `out`, `out_mask`, and `memory` into the decoder layer. The output of the decoder will be a `length(out)` x `batchSize` x $d\_{model}$ tensor (note that in your case, $d\_{model} = \|dict\|$), which indicates `tokenIdx` x `batchIdx` x `vocabularyIdx`
     3. Convert the last time step (e.g. `decoderOutput[-1,:,:]` if all sequences in your batch have the same length) into probabilities, and take all the events that surpasses some probability threshold and set them to $1$. Set all other events to $0$. This is the new time step that the decoder layer predicts. So add this time step to `out` to append it's length by $1$.
     4. Repeat the previous 3 steps in a loop until you reach the `<END>` token, which signals the end of decoding, or until the max output length is reached.
  
  During evaluation, you can calculate the validation loss by using the logits of only the final iteration of the decoder (which should have dimension `length(sequence)+1` x `batchSize` x $\|dict\|$), and compare those logits with the ground truth with the `<END>` token added at the end (resulting in dimensions `length(sequence)+1` x `batchSize` x $\|dict\|$).
  
  > **Comments:**
  > **username_0**: Sorry, I wasn't clear about this - I get this issue specifically during training, not inference time. Here's how I understand training should go: for an output token at timestamp `t` we  give a model the whole `src` sequence as well as `tgt[0 : t-1]`. It's not like generating the whole sentence in English given a sentence in French, but instead like predicting the next word user is going to write given previous sentence and previous words in this sentence user has already written. IIUC, for each `out[t]` mask should hide `tgt[t : ]` even though training loop sees the whole `tgt`.  
  > **username_0**: But visualization in the question shows that `out[t]` is extremely similar to `tgt[t]`. I can think of 2 reasons for this - overfitting or wrong usage of the mask. I have already checked that the visualization is done on the test set, so the problem must be the mask.  
  > **username_0**: Perhaps, the question can be reduced to: is `decoder(tgt, memory, subsequent_mask)` the correct way to predict all outputs in parallel, restricting each `out[t]` to see only true values `tgt[ : t - 1]`?  
  > **username_1**: Thanks for the clarification. I have edited my answer which should help with all the things you mentioned. Let me know if there's anything further to clarify.  
  > **username_0**: Thanks! I appreciate your time and patience. Indeed, I used a shifted mask instead of shifted target for the same reason you described, but I could have made a mistake in it and your version looks more common anyway. So I'm going to update the code as you suggest and come back with the result!  
  > **username_0**: After shifting the target the prediction during training looks much more realistic - see the edited post for visualization. I still have some doubts regarding my implementation, but anyway, your answer is a great tutorial for people coming here with similar issues. Thanks a lot for your help!  
  > **username_1**: Glad it helped! One more thing I'd verify is why `out[t]` resembles `tgt[t-1]` though -- it does seem weird. Wouldn't be surprised if there's an off-by-one somewhere, since we shift things so many times by 1 token.  
  > **username_0**: It's pretty natural for my task to have `tgt[t]` resembling `tgt[t - 1]`, so the model is expected to learn something like this. But yes, having a similar mistake previously I look at it very carefully :)  
  > **username_2**: @username_1 so should we also using the mask during the testing time? I thought during the testing the masks should be set to None, am I wrong?  
  > **username_3**: @username_2 I think [this](https://datascience.stackexchange.com/a/81492/65120) answers your question.  
  
  ---
  
  