Answers.Body: <p>The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step.</p>
  <p>A simple RNN then might have an input <span class="math-container">$x_t$</span>, a hidden layer <span class="math-container">$h_t$</span>, and an output <span class="math-container">$y_t$</span> at each time step <span class="math-container">$t$</span>. The values of the hidden layer <span class="math-container">$h_t$</span> are often computed as:</p>
  <p><span class="math-container">$h_t = f(W_{xh}x_t + W_{hh}h_{t-1})$</span></p>
  <p>Where <span class="math-container">$f$</span> is some non-linear function, <span class="math-container">$W_{xh}$</span> is a weight matrix of size <span class="math-container">$h\times x$</span>, and <span class="math-container">$W_{hh}$</span> is a weight matrix of size <span class="math-container">$h\times h$</span>. I've left out the bias terms for simplicity.</p>
  <p>Thus, the values of the hidden layer <span class="math-container">$h_t$</span> depend on the input <span class="math-container">$x_t$</span> as well as on the previous hidden state <span class="math-container">$h_{t-1}$</span> (literally, the previous values of the hidden layer).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-04T13:47:41.640Z
Answers.Text: Clearest explanation, bravo
Answers.UserDisplayName: Wuaner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T19:26:04.690Z
Answers.LastActivityDate: 2021-12-14T19:26:04.690Z
Answers.UserDisplayName: Burrito
Answers.Body: <p>As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn't an optimisation process decode it?</p>
  
  <p>The hidden state is essentially just an encoding of the information you gave it keeping the time-dependencies in check. Most encoder-decoder networks are trained end to end meaning, when the encoding is learned a corresponding decoding is learned simultaneously to decode the encoded latent in your desired format. </p>
  
  <p>I'd recommend you read <a href="http://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">this blog</a> on how transformer models are used to convert French to English, as it would give you better intuition and understanding on what happens with encoder-decoder sequence models</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-29T06:34:12.777Z
Answers.LastActivityDate: 2019-10-29T06:34:12.777Z
Answers.UserDisplayName: ashenoy
Answers.Body: <p>I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can produce some target output. Each layer within a neural network can only really &quot;see&quot; an input according to the specifics of its nodes, so each layer produces unique &quot;snapshots&quot; of whatever it is processing. Hidden states are sort of intermediate snapshots of the original input data, transformed in whatever way the given layer's nodes and neural weighting require.</p>
  <p>The snapshots are just vectors so they can theoretically be processed by any other layer - by either an encoding layer or a decoding layer in your example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-11T05:14:31.690Z
Answers.LastActivityDate: 2021-12-11T05:14:31.690Z
Answers.UserDisplayName: ticiarai
Answers.Body: <p>This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.</p>
  <p>Let's consider the following two input and output sequences</p>
  <p><span class="math-container">\begin{align}
  X &amp;= [a, b, c, d, \dots,y , z]\\
  Y &amp;= [b, c, d, e, \dots,z , a]
  \end{align}</span></p>
  <p>We will first try to train a multi-layer perceptron (MLP) with one input and one output from <span class="math-container">$X$</span> and <span class="math-container">$Y$</span>. Here, the details of the hidden layers don't matter.</p>
  <p>We can write this relationship in maths as</p>
  <p><span class="math-container">$$f(x)\rightarrow y$$</span></p>
  <p>where <span class="math-container">$x$</span> is an element of <span class="math-container">$X$</span> and <span class="math-container">$y$</span> is an element of <span class="math-container">$Y$</span> and <span class="math-container">$f(\cdot)$</span> is our MLP.</p>
  <p>After training, if given the input <span class="math-container">$a = x$</span>, our neural network will give an output <span class="math-container">$b = y$</span> because <span class="math-container">$f(\cdot)$</span> learned the mapping between the sequence <span class="math-container">$X$</span> and <span class="math-container">$Y$</span>.</p>
  <p>Now, instead of the above sequences, try to teach the following sequences to the same MLP.</p>
  <p><span class="math-container">\begin{align}
  X &amp;= [a,a,b,b,c,c,\cdots, y,z,z]\\
  Y &amp;= [a,b,c,\cdots, z,a,b,c, \cdots, y,z]
  \end{align}</span></p>
  <p>More than likely, this MLP will not be able to learn the relationship between <span class="math-container">$X$</span> and <span class="math-container">$Y$</span>. This is because a simple MLP <strong>can't learn and understand the relationship</strong> between the previous and current characters.</p>
  <p>Now, we use the same sequences to train an RNN. In an RNN, we take <strong>two inputs</strong>, one for our input and the previous hidden values, and <strong>two outputs</strong>, one for the output and the next hidden values.</p>
  <p><span class="math-container">$$f(x, h_t)\rightarrow (y, h_{t+1})$$</span></p>
  <p><strong>Important</strong>: here <span class="math-container">$h_{t+1}$</span> represents the next hidden value.</p>
  <p>We will execute some sequences of this RNN model. We initialize the hidden value to zero.</p>
  <pre><code>x = a and h = 0
  (a,next_hidden) &lt;- f(x,h)
  prev_hidden = next_hidden
  
  x = a and h = prev_hidden
  (b,next_hidden) &lt;- f(x,h)
  prev_hidden = next_hidden
  
  x = b and h = prev_hidden
  (c,next_hidden) &lt;- f(x,h)
  prev_hidden = next_hidden
  
  and so on 
  </code></pre>
  <p>If we look at the above process we can see that we are taking the previous hidden state values to compute the next hidden state. What happens is while we iterate through this process <code>prev_hidden = next_hidden</code> it also <strong>encodes some information</strong> about our sequence which will help in predicting our next character.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-29T12:19:02.417Z
Answers.Text: I have a question regarding the output and hidden state here.
  here i already know what i output will look like at each time step then then should the hidden state be unknown?
  a normal neural net is farely simple i give it some inputs and corresponding outputs and it calculates the weigths and biases in order to predict a data set that you give it.. can i be using lstm to do the same thing?
Answers.UserDisplayName: sneha verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-28T10:35:12.507Z
Answers.LastActivityDate: 2024-02-18T08:25:22.057Z
Answers.LastEditDate: 2024-02-18T08:25:22.057Z
Answers.UserDisplayName: Eka
Body: <p>I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its <strong>hidden states</strong>. In my specific case, the hidden state of the encoder is passed to the decoder, and this would allow the model to learn better latent representations.</p>
  <p>Does this make sense?</p>
  <p>I am a bit confused about this because I really don't know what the hidden state is. Moreover, we're using separate LSTMs for the encoder and decoder, so I can't see how the hidden state from the encoder LSTM can be useful to the decoder LSTM because only the encoder LSTM really understands it.</p>
  
Tags: <recurrent-neural-networks><long-short-term-memory><hidden-layers><seq2seq><encoder-decoder>
Title: What exactly is a hidden state in an LSTM and RNN?
UserDisplayName: user8714896
ThreadText: # What exactly is a hidden state in an LSTM and RNN?
  
  (Asked by: username_0 on 2019-10-29)
  
  I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its **hidden states**. In my specific case, the hidden state of the encoder is passed to the decoder, and this would allow the model to learn better latent representations.
  
  Does this make sense?
  
  I am a bit confused about this because I really don't know what the hidden state is. Moreover, we're using separate LSTMs for the encoder and decoder, so I can't see how the hidden state from the encoder LSTM can be useful to the decoder LSTM because only the encoder LSTM really understands it.
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, hidden-layers, seq2seq, encoder-decoder
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2019-11-28. Score: 14 | Accepted answer
  
  This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.
  Let's consider the following two input and output sequences
  \begin{align}
  X &= [a, b, c, d, \dots,y , z]\\
  Y &= [b, c, d, e, \dots,z , a]
  \end{align}
  We will first try to train a multi-layer perceptron (MLP) with one input and one output from $X$ and $Y$. Here, the details of the hidden layers don't matter.
  We can write this relationship in maths as
  $$f(x)\rightarrow y$$
  where $x$ is an element of $X$ and $y$ is an element of $Y$ and $f(\cdot)$ is our MLP.
  After training, if given the input $a = x$, our neural network will give an output $b = y$ because $f(\cdot)$ learned the mapping between the sequence $X$ and $Y$.
  Now, instead of the above sequences, try to teach the following sequences to the same MLP.
  \begin{align}
  X &= [a,a,b,b,c,c,\cdots, y,z,z]\\
  Y &= [a,b,c,\cdots, z,a,b,c, \cdots, y,z]
  \end{align}
  More than likely, this MLP will not be able to learn the relationship between $X$ and $Y$. This is because a simple MLP can't learn and understand the relationship between the previous and current characters.
  Now, we use the same sequences to train an RNN. In an RNN, we take two inputs, one for our input and the previous hidden values, and two outputs, one for the output and the next hidden values.
  $$f(x, h_t)\rightarrow (y, h_{t+1})$$
  Important: here $h_{t+1}$ represents the next hidden value.
  We will execute some sequences of this RNN model. We initialize the hidden value to zero.
  x = a and h = 0
  (a,next_hidden) <- f(x,h)
  prev_hidden = next_hidden
  
  x = a and h = prev_hidden
  (b,next_hidden) <- f(x,h)
  prev_hidden = next_hidden
  
  x = b and h = prev_hidden
  (c,next_hidden) <- f(x,h)
  prev_hidden = next_hidden
  
  and so on 
  
  If we look at the above process we can see that we are taking the previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.
  
  > **Comments:**
  > **username_6**: I have a question regarding the output and hidden state here. here i already know what i output will look like at each time step then then should the hidden state be unknown? a normal neural net is farely simple i give it some inputs and corresponding outputs and it calculates the weigths and biases in order to predict a data set that you give it.. can i be using lstm to do the same thing?  
  
  ---
  
  ### Answer by username_1 on 2021-12-14. Score: 5
  
  The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step.
  
  A simple RNN then might have an input $x\_t$, a hidden layer $h\_t$, and an output $y\_t$ at each time step $t$. The values of the hidden layer $h\_t$ are often computed as:
  
  $h\_t = f(W\_{xh}x\_t + W\_{hh}h\_{t-1})$
  
  Where $f$ is some non-linear function, $W\_{xh}$ is a weight matrix of size $h\times x$, and $W\_{hh}$ is a weight matrix of size $h\times h$. I've left out the bias terms for simplicity.
  
  Thus, the values of the hidden layer $h\_t$ depend on the input $x\_t$ as well as on the previous hidden state $h\_{t-1}$ (literally, the previous values of the hidden layer).
  
  > **Comments:**
  > **username_2**: Clearest explanation, bravo  
  
  ---
  
  ### Answer by username_3 on 2019-10-29. Score: 3
  
  As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn't an optimisation process decode it?
  
  The hidden state is essentially just an encoding of the information you gave it keeping the time-dependencies in check. Most encoder-decoder networks are trained end to end meaning, when the encoding is learned a corresponding decoding is learned simultaneously to decode the encoded latent in your desired format.
  
  I'd recommend you read [this blog](http://jalammar.github.io/illustrated-transformer/) on how transformer models are used to convert French to English, as it would give you better intuition and understanding on what happens with encoder-decoder sequence models
  
  
  ---
  
  ### Answer by username_4 on 2021-12-11. Score: 3
  
  I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can produce some target output. Each layer within a neural network can only really "see" an input according to the specifics of its nodes, so each layer produces unique "snapshots" of whatever it is processing. Hidden states are sort of intermediate snapshots of the original input data, transformed in whatever way the given layer's nodes and neural weighting require.
  
  The snapshots are just vectors so they can theoretically be processed by any other layer - by either an encoding layer or a decoding layer in your example.
  
  
  ---
  
  