Answers.Body: <p>(Old question, I know...)</p>
  
  <p>It is not that we <em>need</em> both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.</p>
  
  <p><strong>Example for Sequence-to-sequence without two RNNs</strong></p>
  
  <p>To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-to-sequence models, and virtually all models have the bipartite structure of encoder and decoder.</p>
  
  <p>Approaches like <a href="https://arxiv.org/abs/1810.13409" rel="nofollow noreferrer">Eager Translation</a> break this implied convention. They learn translation models that do not encode and decode with separate RNNs, but at every time step 1) read a source token and 2) produce a target token - with a single RNN.</p>
  
  <p><strong>Why encoder-decoder works better very often</strong></p>
  
  <p>Sequence-to-sequence modeling with encoder-decoder structure almost always implies attention in-between encoder and decoder. Attention relays information between the encoder and decoder, in the sense that every time the decoder has to generate the next item in the target sequence, an attention network computes a dynamic, useful "summary" of all encoder states.</p>
  
  <p>This attention summary is different and recomputed for every decoding step. On the other hand, encoding the source sequence is done only once and then all encoder states are kept in memory.</p>
  
  <p>The ability to have a direct view of the source sequence (using as a proxy the entire sequence of encoder states) via attention is what makes the encoder-decoder approach superioir to a single RNN.</p>
  
  <p>In comparison, a single RNN only has a direct view on one element of the input sequence. Some interesting scenarios for a single RNN:</p>
  
  <ul>
  <li><strong>At every time step, read one source token, then write one target token</strong>: Previous elements in the source sequence are represented only in lossy recurrent states, while future elements cannot be accessed at all.</li>
  <li><strong>First read all source tokens, then write all target tokens.</strong>: the meaning of the entire source sentence has to be compressed into a fixed-size recurrent state vector.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-28T17:30:43.397Z
Answers.LastActivityDate: 2020-01-28T17:30:43.397Z
Answers.UserDisplayName: Mathias Müller
Body: <p>Why do we need both encoder and decoder in sequence to sequence prediction?</p>
  
  <p>We could just have a single RNN that, given input <span class="math-container">$x$</span>, outputs some value <span class="math-container">$y(t)$</span> and hidden state <span class="math-container">$h(t)$</span>. Next, given <span class="math-container">$h(t)$</span> and <span class="math-container">$y(t)$</span>, the next output <span class="math-container">$y(t+1)$</span> and hidden state <span class="math-container">$h(t+1)$</span> should be produced, and so on. The architecture shall consists of only one network instead of two separate ones. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-03T16:55:35.653Z
Comments.Text: Could you maybe draw a diagram of your suggested RNN architecture? It is not 100% clear to me what you mean.The encoder/decoder architecture used in seq2seq language translation already uses a feedback loop from output to input in the decoder. I think you are asking why you cannot use the same network/parameters for both stages? But I'd like to see a picture or something more concrete in order to understand what you are proposing
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-03T20:14:07.927Z
Comments.Text: Precisely, why not just use one network? I updated the original post by describing what I mean
Comments.UserDisplayName: greensquare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-03T22:10:55.260Z
Comments.Text: Sorry I still don't get your proposed architecture. You don't give indication of time steps for x, which is also a sequence. Are you suggesting that you run the network just like a seq2seq network, where you feed in a sequence of $x$ (ignoring the predictions), then after some point - perhaps after an end token - start reading $y$ and feeding that in to the same inputs as $x$? What if $x$ and $y$ are sequences of different things? Are you proposing to input "blank" $y$ whilst feeding in $x$, then blank $x$ when feeding in $y$?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-04T07:41:55.740Z
Comments.Text: I am assuming batch size of 1. So, you feed in x1 (the first training sample), ignore the output and just keep the last hidden state of the RNN. Then, you run a loop that goes up to some max_len value and starts producing the output values based on the last hidden state and prev output value. Then you repeat the process for the next training sample x2. x is sequence of numbers and y(t) (at single time step) is just a number. Hope this clears things up
Comments.UserDisplayName: greensquare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-04T10:16:17.747Z
Comments.Text: In your case do $x$ and $y$ represent the same kind of measurement? Typically you would use a seq2seq model if that was *not* the case (e.g. English and French words are not the same data type). However, you might use your proposed architecture if $x$ and $y$ were from the same series and your goal is to predict a continuation of the series - e.g. a financial series or NLP language model of a single language.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-05T08:10:29.417Z
Comments.Text: Thanks, I convert the output in the representation space of the input. The output consists of two values that have separate output vocabulary. My input space is word embeddings. Output space is only 10 words (so I have output space ranging from 0 to 10) Later I convert the output number to a word in the input space since there is one-to-one mapping. Is this ok?
Comments.UserDisplayName: greensquare
Tags: <machine-learning><ai-design><sequence-modeling><encoder-decoder>
Title: Why do we need both encoder and decoder in sequence to sequence prediction?
UserDisplayName: greensquare
ThreadText: # Why do we need both encoder and decoder in sequence to sequence prediction?
  
  (Asked by: username_0 on 2018-12-03)
  
  Why do we need both encoder and decoder in sequence to sequence prediction?
  
  We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$ and $y(t)$, the next output $y(t+1)$ and hidden state $h(t+1)$ should be produced, and so on. The architecture shall consists of only one network instead of two separate ones.
  
  **Tags:** machine-learning, ai-design, sequence-modeling, encoder-decoder
  > **Comments:**
  > **username_1**: Could you maybe draw a diagram of your suggested RNN architecture? It is not 100% clear to me what you mean.The encoder/decoder architecture used in seq2seq language translation already uses a feedback loop from output to input in the decoder. I think you are asking why you cannot use the same network/parameters for both stages? But I'd like to see a picture or something more concrete in order to understand what you are proposing  
  > **username_0**: Precisely, why not just use one network? I updated the original post by describing what I mean  
  > **username_1**: Sorry I still don't get your proposed architecture. You don't give indication of time steps for x, which is also a sequence. Are you suggesting that you run the network just like a seq2seq network, where you feed in a sequence of $x$ (ignoring the predictions), then after some point - perhaps after an end token - start reading $y$ and feeding that in to the same inputs as $x$? What if $x$ and $y$ are sequences of different things? Are you proposing to input "blank" $y$ whilst feeding in $x$, then blank $x$ when feeding in $y$?  
  > **username_0**: I am assuming batch size of 1. So, you feed in x1 (the first training sample), ignore the output and just keep the last hidden state of the RNN. Then, you run a loop that goes up to some max_len value and starts producing the output values based on the last hidden state and prev output value. Then you repeat the process for the next training sample x2. x is sequence of numbers and y(t) (at single time step) is just a number. Hope this clears things up  
  > **username_1**: In your case do $x$ and $y$ represent the same kind of measurement? Typically you would use a seq2seq model if that was *not* the case (e.g. English and French words are not the same data type). However, you might use your proposed architecture if $x$ and $y$ were from the same series and your goal is to predict a continuation of the series - e.g. a financial series or NLP language model of a single language.  
  > **username_0**: Thanks, I convert the output in the representation space of the input. The output consists of two values that have separate output vocabulary. My input space is word embeddings. Output space is only 10 words (so I have output space ranging from 0 to 10) Later I convert the output number to a word in the input space since there is one-to-one mapping. Is this ok?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-01-28. Score: 3
  
  (Old question, I know...)
  
  It is not that we *need* both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.
  
  **Example for Sequence-to-sequence without two RNNs**
  
  To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-to-sequence models, and virtually all models have the bipartite structure of encoder and decoder.
  
  Approaches like [Eager Translation](https://arxiv.org/abs/1810.13409) break this implied convention. They learn translation models that do not encode and decode with separate RNNs, but at every time step 1) read a source token and 2) produce a target token - with a single RNN.
  
  **Why encoder-decoder works better very often**
  
  Sequence-to-sequence modeling with encoder-decoder structure almost always implies attention in-between encoder and decoder. Attention relays information between the encoder and decoder, in the sense that every time the decoder has to generate the next item in the target sequence, an attention network computes a dynamic, useful "summary" of all encoder states.
  
  This attention summary is different and recomputed for every decoding step. On the other hand, encoding the source sequence is done only once and then all encoder states are kept in memory.
  
  The ability to have a direct view of the source sequence (using as a proxy the entire sequence of encoder states) via attention is what makes the encoder-decoder approach superioir to a single RNN.
  
  In comparison, a single RNN only has a direct view on one element of the input sequence. Some interesting scenarios for a single RNN:
  
  * **At every time step, read one source token, then write one target token**: Previous elements in the source sequence are represented only in lossy recurrent states, while future elements cannot be accessed at all.
  * **First read all source tokens, then write all target tokens.**: the meaning of the entire source sentence has to be compressed into a fixed-size recurrent state vector.
  
  
  ---
  
  