Answers.Body: <p>You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector <span class="math-container">$x \in \mathbb{R}^d$</span> and could not take as input for instance a vector <span class="math-container">$y \in \mathbb{R}^b$</span> where <span class="math-container">$b \neq d$</span>. The size you refer to in the context of the RNN is the <em>length</em> of the input sequence.</p>
  <p>What you are getting confused with is that RNN's can make predictions relating to sequences, that is imagine now rather than one <span class="math-container">$x$</span> we have a sequence of <em>related</em>, i.e. not i.i.d. data such as time series data, data <span class="math-container">$\{x_i\}_{i=1}^n$</span>. Assuming we are given some initial <span class="math-container">$h_0$</span> (a hidden state) then an RNN will take as input <span class="math-container">$x_1$</span> and <span class="math-container">$h_0$</span> and output a prediction <span class="math-container">$y_1$</span> and a new hidden state <span class="math-container">$h_1$</span>. In general an RNN will take as input <span class="math-container">$x_n$</span> and <span class="math-container">$h_{n-1}$</span> and output <span class="math-container">$y_n$</span> and <span class="math-container">$h_n$</span> where the hidden state is passed as input to the RNN at the next time step. However, the dimensionality of all the <span class="math-container">$x$</span>'s and <span class="math-container">$h$</span>'s will all be the same, i.e. <span class="math-container">$x_i \in \mathbb{R}^d$</span> and <span class="math-container">$h_i \in \mathbb{R}^c$</span> for all <span class="math-container">$i$</span>, where <span class="math-container">$d$</span> does not necessarily have to be equal to <span class="math-container">$c$</span> (and in my experience rarely is).</p>
  <p>Note that RNN's can also perform sequence to sequence prediction (such as language translation) where the predicted sequence can be a different length to the input sequence, as is the case when doing translation (the input sentence is not necessarily the same length in the translated language). They do this by having an encoder and a decoder which are two separate RNNs. The encoder is fed the input sequence and we maintain all the hidden states outputted by the encoder <span class="math-container">$\{e_i\}_{i=1}^n$</span>. The decoder is then given a token that represents the start of a sequence and the last hidden state of the encoder <span class="math-container">$e_n$</span> as input to which it will make a prediction on what the word should be (I believe but am not 100% certain that the decoder outputs a probability distribution over the dictionary of words it can predict from) and then the word chosen is passed as input to the decoder at the next step, along with the hidden state from the decoder at the previous step and another word is predicted. This continues until the system predicts an End of Sequence token (or until it is forced to by some time limit). More details can be found in <a href="https://arxiv.org/abs/1409.3215" rel="nofollow noreferrer">this</a> paper but I believe in NLP it is much more common to use attention models now rather than the methods introduced in this paper.</p>
  <p>Now, with this data <span class="math-container">$\{x_i\}_{i=1}^n$</span> you could in theory pass each one individually to a FFNN but a) it would not capture the sequential nature of the data as the assumption is that each data point is independent of the others and you can see this from the architecture of a FFNN -- they are Directed Acyclic Graphs, the Acyclic-ness is what causes the issue as there is no recursion which stops any sequential information from being passed from one time step to another and b) training using SGD would likely cause issues as we have violated the i.i.d. assumption needed for SGD to converge to local optima. More info on the i.i.d. requirement can be found <a href="https://ai.stackexchange.com/questions/10839/why-exactly-do-neural-networks-require-i-i-d-data?noredirect=1&amp;lq=1">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-04T22:47:12.443Z
Answers.Text: So, how can RNNs transform variable-length sentences (e.g. in English) to other variable-length sentences (e.g. in French), i.e. for tasks like machine translation (where sentences can have a different number of words or words could have a different number of letters)? :D So, you may want to give the example of machine translation to be more concrete and how could the variable-length word/sentence solved in this task.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-05T00:17:43.233Z
Answers.Text: NLP is not my cup of tea but I can give a brief explanation of eg seq2seq and point them to a relevant paper!
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-04T16:52:08.207Z
Answers.LastActivityDate: 2021-04-07T09:44:04.160Z
Answers.LastEditDate: 2021-04-07T09:44:04.160Z
Answers.UserDisplayName: David
Body: <p>Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this with an example?</p>
  
Tags: <neural-networks><recurrent-neural-networks><feedforward-neural-networks><multilayer-perceptrons>
Title: Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?
UserDisplayName: Daniel
ThreadText: # Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?
  
  (Asked by: username_0 on 2021-02-04)
  
  Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this with an example?
  
  **Tags:** neural-networks, recurrent-neural-networks, feedforward-neural-networks, multilayer-perceptrons
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-04. Score: 7 | Accepted answer
  
  You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the *length* of the input sequence.
  
  What you are getting confused with is that RNN's can make predictions relating to sequences, that is imagine now rather than one $x$ we have a sequence of *related*, i.e. not i.i.d. data such as time series data, data $\{x\_i\}\_{i=1}^n$. Assuming we are given some initial $h\_0$ (a hidden state) then an RNN will take as input $x\_1$ and $h\_0$ and output a prediction $y\_1$ and a new hidden state $h\_1$. In general an RNN will take as input $x\_n$ and $h\_{n-1}$ and output $y\_n$ and $h\_n$ where the hidden state is passed as input to the RNN at the next time step. However, the dimensionality of all the $x$'s and $h$'s will all be the same, i.e. $x\_i \in \mathbb{R}^d$ and $h\_i \in \mathbb{R}^c$ for all $i$, where $d$ does not necessarily have to be equal to $c$ (and in my experience rarely is).
  
  Note that RNN's can also perform sequence to sequence prediction (such as language translation) where the predicted sequence can be a different length to the input sequence, as is the case when doing translation (the input sentence is not necessarily the same length in the translated language). They do this by having an encoder and a decoder which are two separate RNNs. The encoder is fed the input sequence and we maintain all the hidden states outputted by the encoder $\{e\_i\}\_{i=1}^n$. The decoder is then given a token that represents the start of a sequence and the last hidden state of the encoder $e\_n$ as input to which it will make a prediction on what the word should be (I believe but am not 100% certain that the decoder outputs a probability distribution over the dictionary of words it can predict from) and then the word chosen is passed as input to the decoder at the next step, along with the hidden state from the decoder at the previous step and another word is predicted. This continues until the system predicts an End of Sequence token (or until it is forced to by some time limit). More details can be found in [this](https://arxiv.org/abs/1409.3215) paper but I believe in NLP it is much more common to use attention models now rather than the methods introduced in this paper.
  
  Now, with this data $\{x\_i\}\_{i=1}^n$ you could in theory pass each one individually to a FFNN but a) it would not capture the sequential nature of the data as the assumption is that each data point is independent of the others and you can see this from the architecture of a FFNN -- they are Directed Acyclic Graphs, the Acyclic-ness is what causes the issue as there is no recursion which stops any sequential information from being passed from one time step to another and b) training using SGD would likely cause issues as we have violated the i.i.d. assumption needed for SGD to converge to local optima. More info on the i.i.d. requirement can be found [here](https://ai.stackexchange.com/questions/10839/why-exactly-do-neural-networks-require-i-i-d-data?noredirect=1&lq=1).
  
  > **Comments:**
  > **username_2**: So, how can RNNs transform variable-length sentences (e.g. in English) to other variable-length sentences (e.g. in French), i.e. for tasks like machine translation (where sentences can have a different number of words or words could have a different number of letters)? :D So, you may want to give the example of machine translation to be more concrete and how could the variable-length word/sentence solved in this task.  
  > **username_1**: NLP is not my cup of tea but I can give a brief explanation of eg seq2seq and point them to a relevant paper!  
  
  ---
  
  