Answers.Body: <p>So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model. The repeating module in long short term memory networks looks like this:
  <a href="https://i.sstatic.net/6TrLT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6TrLT.png" alt="LSTM repeating module"></a>
  As you can see there are many different parts to this module. There are three main parts. The first is the forget gate layer. This layer tells the cell state or the line running across the stop what to keep. The cell state is kept by this line:<a href="https://i.sstatic.net/CthhV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CthhV.png" alt="enter image description here"></a>
  The entire network is based off of manipulating this cell state in order to get accurate results. The equation that you mentioned was related to backprop which is used to train the neural network. This is related to the cell state because it is used to calculate it during the backwards pass. @BlueMoon93 mentioned that this equation has t+1 one in it, but this is because as the recurrent neural network propogates backwards through each module the time goes from high to low. So to conclude, the cell state in an LSTM is one of the vectors that the neural network modifies based on input.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T03:22:33.650Z
Answers.LastActivityDate: 2017-06-02T03:22:33.650Z
Answers.UserDisplayName: Aiden Grossman
Body: <p>I am Reading "<a href="http://www.cs.toronto.edu/~graves/phd.pdf" rel="nofollow noreferrer">Supervised Sequence Labelling with Recurrent Neural Networks</a>" written by Alex Graves to try to understand LSTM networks and I am a bit confused about the equations.</p>
  
  <p>Specifically, what I am confused about is the term "state". When used in an equation (section 4.5.2), it says:</p>
  
  <p><a href="https://i.sstatic.net/3ANyI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3ANyI.png" alt="enter image description here"></a></p>
  
  <p>I know that some system can be in a state, for example, due to the setup of values of different nodes in a graph. But how can a state be described in the case of a neural network and how can the equation above be explained other than that it is the state (or states of several timesteps as in recurrent neural networks) of a neural network?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T09:21:07.027Z
Comments.Text: I'm actually quite baffled by these equations. The state at time t requires the state at time t+1...
Comments.UserDisplayName: BlueMoon93
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T09:24:29.923Z
Comments.Text: @BlueMoon93 When talking about the dependencies of different values in different timesteps in recurrent neural networks, it has been claimed (if I understood correctly) that BPTT should be the same as the chain rule in calculus... however I don't think the term "state" is as intuitive as e.g. "output" which is the actual value that has been received after passing something through an activation function.
Comments.UserDisplayName: Stephen Johnson
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T14:13:20.273Z
Comments.Text: I have found this blog post to be very helpful in understanding LSTMs:http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Comments.UserDisplayName: Aiden Grossman
Tags: <neural-networks><recurrent-neural-networks><terminology><long-short-term-memory><definitions>
Title: What is a state in a recurrent neural network?
UserDisplayName: Stephen Johnson
ThreadText: # What is a state in a recurrent neural network?
  
  (Asked by: username_0 on 2017-06-01)
  
  I am Reading "[Supervised Sequence Labelling with Recurrent Neural Networks](http://www.cs.toronto.edu/~graves/phd.pdf)" written by Alex Graves to try to understand LSTM networks and I am a bit confused about the equations.
  
  Specifically, what I am confused about is the term "state". When used in an equation (section 4.5.2), it says:
  
  [![enter image description here](https://i.sstatic.net/3ANyI.png)](https://i.sstatic.net/3ANyI.png)
  
  I know that some system can be in a state, for example, due to the setup of values of different nodes in a graph. But how can a state be described in the case of a neural network and how can the equation above be explained other than that it is the state (or states of several timesteps as in recurrent neural networks) of a neural network?
  
  **Tags:** neural-networks, recurrent-neural-networks, terminology, long-short-term-memory, definitions
  > **Comments:**
  > **username_1**: I'm actually quite baffled by these equations. The state at time t requires the state at time t+1...  
  > **username_0**: @BlueMoon93 When talking about the dependencies of different values in different timesteps in recurrent neural networks, it has been claimed (if I understood correctly) that BPTT should be the same as the chain rule in calculus... however I don't think the term "state" is as intuitive as e.g. "output" which is the actual value that has been received after passing something through an activation function.  
  > **username_2**: I have found this blog post to be very helpful in understanding LSTMs:http://colah.github.io/posts/2015-08-Understanding-LSTMs/  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-06-02. Score: 3
  
  So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model. The repeating module in long short term memory networks looks like this:
  [![LSTM repeating module](https://i.sstatic.net/6TrLT.png)](https://i.sstatic.net/6TrLT.png)
  As you can see there are many different parts to this module. There are three main parts. The first is the forget gate layer. This layer tells the cell state or the line running across the stop what to keep. The cell state is kept by this line:[![enter image description here](https://i.sstatic.net/CthhV.png)](https://i.sstatic.net/CthhV.png)
  The entire network is based off of manipulating this cell state in order to get accurate results. The equation that you mentioned was related to backprop which is used to train the neural network. This is related to the cell state because it is used to calculate it during the backwards pass. @BlueMoon93 mentioned that this equation has t+1 one in it, but this is because as the recurrent neural network propogates backwards through each module the time goes from high to low. So to conclude, the cell state in an LSTM is one of the vectors that the neural network modifies based on input.
  
  
  ---
  
  