Answers.Body: <ol>
  <li><p>There are different variants of LSTM, in most ML packages now of days you'll probably see what's shown in the bottom picture. For more details, intuition and motivations please see <a href="https://arxiv.org/pdf/1503.04069.pdf" rel="nofollow noreferrer">this paper</a>. </p></li>
  <li><p>It is not reshaped and there is no softmax layer. That is all done <strong>outside</strong> the LSTM.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-08T05:05:00.863Z
Answers.LastActivityDate: 2019-09-08T05:05:00.863Z
Answers.UserDisplayName: mshlis
Body: <p>I've found multiple depictions of how an LSTM cell operates. See 2 below:</p>
  
  <p><a href="https://i.sstatic.net/YGgZO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YGgZO.png" alt="enter image description here"></a></p>
  
  <p>and</p>
  
  <p><a href="https://i.sstatic.net/3e630.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3e630.png" alt="enter image description here"></a></p>
  
  <p>Each of these images suggest the hidden state is utilised differently. On the top diagram, it is shown that the hidden state is added along with the previous output and current input to both the forget gate and the input gate. The bottom image suggests the input and forget gates are calculated only using the previous output and current input. Which is it?</p>
  
  <p>Also, when the previous output is fed in for the current layer, is this before or after it has been reshaped to the final output size and been put through a softmax?</p>
  
Tags: <neural-networks><backpropagation><long-short-term-memory>
Title: Structure discrepancy of an LSTM?
UserDisplayName: Recessive
ThreadText: # Structure discrepancy of an LSTM?
  
  (Asked by: username_0 on 2019-09-08)
  
  I've found multiple depictions of how an LSTM cell operates. See 2 below:
  
  [![enter image description here](https://i.sstatic.net/YGgZO.png)](https://i.sstatic.net/YGgZO.png)
  
  and
  
  [![enter image description here](https://i.sstatic.net/3e630.png)](https://i.sstatic.net/3e630.png)
  
  Each of these images suggest the hidden state is utilised differently. On the top diagram, it is shown that the hidden state is added along with the previous output and current input to both the forget gate and the input gate. The bottom image suggests the input and forget gates are calculated only using the previous output and current input. Which is it?
  
  Also, when the previous output is fed in for the current layer, is this before or after it has been reshaped to the final output size and been put through a softmax?
  
  **Tags:** neural-networks, backpropagation, long-short-term-memory
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-08. Score: 1 | Accepted answer
  
  1. There are different variants of LSTM, in most ML packages now of days you'll probably see what's shown in the bottom picture. For more details, intuition and motivations please see [this paper](https://arxiv.org/pdf/1503.04069.pdf).
  2. It is not reshaped and there is no softmax layer. That is all done **outside** the LSTM.
  
  
  ---
  
  