Answers.Body: <p>I know it's too late to answer your query after 1.5 years</p>
  <p>The inference algorithm is explained at <a href="https://charon.me/posts/pytorch/pytorch_seq2seq_5/#inference" rel="nofollow noreferrer">https://charon.me/posts/pytorch/pytorch_seq2seq_5/#inference</a></p>
  <p>Implementation of the algorithm is available at: <a href="https://github.com/pytorch/fairseq" rel="nofollow noreferrer">https://github.com/pytorch/fairseq</a>.</p>
  <p>One could get a complete understanding from their source code.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-15T11:38:48.717Z
Answers.LastActivityDate: 2021-03-15T11:38:48.717Z
Answers.UserDisplayName: vanquishers
Body: <p>I am struggling to understand the use of the Convolutional Sequence to Sequence (Conv-Seq2Seq) model.  The image below is take directly from the paper and is the nearly canonical diagram of the parallel training procedure.  After puzzling over it for quite some time, it has come to seem straight forward to me:</p>
  
  <ul>
  <li>An input sentence of N tokens can be encoded in one step because the input sentence exists prior to the start of training, and therefore the token-wise convolution can be trivially parallelized. (Compare to RNN encoders, which require N steps)</li>
  <li><strong>During training</strong>, an output sentence can similarly be parallelized in the decoder because <strong>during training</strong>, the entire output sentence is known.</li>
  <li>Therefore, <strong>during training</strong>, the attention function can be fully parallelized in the two dimensional array of dot products shown below</li>
  <li>Finally, <strong>during training</strong>, the attention is used to weight the input embeddings and encodings, combined with the output training encodings (as such) and the final output assembled. </li>
  </ul>
  
  <p>This is clearly not the case after the network is trained and evaluation input sequences are translated without reference outputs.  I understand from various resources (including but not limited to Gehring's conference presentation) that post-training, output sequences are generated token by token in a fashion vaguely similar to earlier architectures, but I cannot find a clear description of that process.</p>
  
  <p>(I speculate that this is because the parallel training routine was so revolutionary at the time, that the focus of the publications was rightly on the training routines.)</p>
  
  <p>Can someone please help me understand the post-training generation algorithm, <em>if possible in terms of the training diagram?</em></p>
  
  <p>My current non-confident understanding is that the sentence below would be handled something like the following:</p>
  
  <ul>
  <li>Prime the decoder with a default token string of <code>&lt;p&gt; &lt;p&gt; &lt;s&gt;</code>, this results (due to convolution) in a single decoder encoding (as such) input into the attention function, and would hopefully generate the single token <code>&lt;'Sie'&gt;</code> as output</li>
  <li>Restart the decoder with the token string <code>&lt;p&gt; &lt;p&gt; &lt;s&gt; &lt;'Sie'&gt;</code> which would generate two inputs into the attention function and hopefully output <code>&lt;'Sie'&gt; &lt;'stimmen'&gt;</code></li>
  <li>Proceed with lengthening input sentences until the final token generated output ends with a <code>&lt;/s&gt;</code> token signifying the end of the sentence. </li>
  </ul>
  
  <p>If that is a correct understanding, can someone confirm it?  If close, can someone correct me?</p>
  
  <p><a href="https://i.sstatic.net/eIVS1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eIVS1.png" alt="Fig 1, Gehring, et al"></a> </p>
  
  <p><a href="https://arxiv.org/pdf/1705.03122.pdf" rel="nofollow noreferrer">Convolutional Sequence to Sequence Learning</a>, Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017</p>
  
Tags: <convolutional-neural-networks><sequence-modeling>
Title: Convolutional Sequence to Sequence Learning: Training vs Generation
UserDisplayName: Novak
ThreadText: # Convolutional Sequence to Sequence Learning: Training vs Generation
  
  (Asked by: username_0 on 2019-04-03)
  
  I am struggling to understand the use of the Convolutional Sequence to Sequence (Conv-Seq2Seq) model. The image below is take directly from the paper and is the nearly canonical diagram of the parallel training procedure. After puzzling over it for quite some time, it has come to seem straight forward to me:
  
  * An input sentence of N tokens can be encoded in one step because the input sentence exists prior to the start of training, and therefore the token-wise convolution can be trivially parallelized. (Compare to RNN encoders, which require N steps)
  * **During training**, an output sentence can similarly be parallelized in the decoder because **during training**, the entire output sentence is known.
  * Therefore, **during training**, the attention function can be fully parallelized in the two dimensional array of dot products shown below
  * Finally, **during training**, the attention is used to weight the input embeddings and encodings, combined with the output training encodings (as such) and the final output assembled.
  
  This is clearly not the case after the network is trained and evaluation input sequences are translated without reference outputs. I understand from various resources (including but not limited to Gehring's conference presentation) that post-training, output sequences are generated token by token in a fashion vaguely similar to earlier architectures, but I cannot find a clear description of that process.
  
  (I speculate that this is because the parallel training routine was so revolutionary at the time, that the focus of the publications was rightly on the training routines.)
  
  Can someone please help me understand the post-training generation algorithm, *if possible in terms of the training diagram?*
  
  My current non-confident understanding is that the sentence below would be handled something like the following:
  
  * Prime the decoder with a default token string of `<p> <p> <s>`, this results (due to convolution) in a single decoder encoding (as such) input into the attention function, and would hopefully generate the single token `<'Sie'>` as output
  * Restart the decoder with the token string `<p> <p> <s> <'Sie'>` which would generate two inputs into the attention function and hopefully output `<'Sie'> <'stimmen'>`
  * Proceed with lengthening input sentences until the final token generated output ends with a `</s>` token signifying the end of the sentence.
  
  If that is a correct understanding, can someone confirm it? If close, can someone correct me?
  
  [![Fig 1, Gehring, et al](https://i.sstatic.net/eIVS1.png)](https://i.sstatic.net/eIVS1.png)
  
  [Convolutional Sequence to Sequence Learning](https://arxiv.org/pdf/1705.03122.pdf), Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017
  
  **Tags:** convolutional-neural-networks, sequence-modeling
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-15. Score: 1
  
  I know it's too late to answer your query after 1.5 years
  
  The inference algorithm is explained at <https://charon.me/posts/pytorch/pytorch_seq2seq_5/#inference>
  
  Implementation of the algorithm is available at: <https://github.com/pytorch/fairseq>.
  
  One could get a complete understanding from their source code.
  
  
  ---
  
  