Answers.Body: <p>After some more research, I have found the answer.</p>
  
  <p>An LSTM is comprised of 1 LSTM cell that is continuously updated by passing in new inputs, the hidden state and the previous output. All nodes inside the LSTM cell are of size hidden_units. That means the output of the activation gate, input gate, forget gate and output gate are all of size hidden_units. As an extension of this, the size of the hidden state is also equal to hidden_units.</p>
  
  <p>The problem arises when the desired output is not equal to that of hidden_units. This is fixed simple and easily, by slapping a basic feed forward net onto the end of the output of the LSTM, so even if it outputs 256, you can convert it to n nodes. These final output nodes are typically put through a softmax with cross-entropy loss applied.</p>
  
  <p>I came to the conclusion by looking at the code included here: <a href="https://towardsdatascience.com/lstm-by-example-using-tensorflow-feb0c1968537" rel="nofollow noreferrer">https://towardsdatascience.com/lstm-by-example-using-tensorflow-feb0c1968537</a></p>
  
  <p>In which you can see the final output has its own separate weights and bias'.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-03T03:53:32.623Z
Answers.LastActivityDate: 2019-09-03T03:53:32.623Z
Answers.UserDisplayName: Recessive
Body: <p>Take the below LSTM:</p>
  
  <pre><code>input: 5x1 matrix
  hidden units: 256
  output size (aka classes, 1 hot vector): 10x1 matrix
  </code></pre>
  
  <p>It is my understanding that an LSTM of this size will do the following:</p>
  
  <p><span class="math-container">$w_x$</span> = weight matrix at <span class="math-container">$x$</span></p>
  
  <p><span class="math-container">$b_x$</span> = bias matrix at <span class="math-container">$x$</span></p>
  
  <p>activation_gate = tanh(<span class="math-container">$w_1$</span> <span class="math-container">$\cdot$</span> input + <span class="math-container">$w_2$</span> <span class="math-container">$\cdot$</span> prev_output + <span class="math-container">$b_1$</span>)</p>
  
  <p>input_gate = sigmoid(<span class="math-container">$w_3$</span> <span class="math-container">$\cdot$</span> input + <span class="math-container">$w_4$</span> <span class="math-container">$\cdot$</span> prev_output + <span class="math-container">$b_2$</span>)</p>
  
  <p>forget_gate = sigmoid(<span class="math-container">$w_5$</span> <span class="math-container">$\cdot$</span> input + <span class="math-container">$w_6$</span> <span class="math-container">$\cdot$</span> prev_output + <span class="math-container">$b_3$</span>)</p>
  
  <p>output_gate = sigmoid(<span class="math-container">$w_7$</span> <span class="math-container">$\cdot$</span> input + <span class="math-container">$w_8$</span> <span class="math-container">$\cdot$</span> prev_output + <span class="math-container">$b_4$</span>)</p>
  
  <p>The size of the output of each gate should be equal to the number of hidden units, ie, 256. The problem arrises when trying to convert to the correct final output size, of 10. If the forget gate outputs 256, then it is summed with the element wise product of the activation and input gate to find the new state, this will result in a hidden state of size 256. (Also in all my research I have not found anywhere whether this addition is actually addition, or simply appending the two matrices).</p>
  
  <p>So if I have a hidden state of 256, and the output gate outputs 256, doing an element wise product of these two results in, surprise surprise, 256, not 10. If I instead ensure the output gate outputs a size of 10, this no longer works with the hidden state in an element wise product.</p>
  
  <p>How is this handled? I can come up with many ways of doing it myself, but I want an identical replica of the basic LSTM unit, as I have some theories I want to test, and if it is even the slightest bit different it would make the research invalid.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-31T05:55:10.260Z
Comments.Text: I found this Quora answer to be helpful in understanding the dimensions at each point in the LSTM cell. [Dimensions of LSTM](https://www.quora.com/In-LSTM-how-do-you-figure-out-what-size-the-weights-are-supposed-to-be)
Comments.UserDisplayName: Kartik Podugu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-01T02:02:28.593Z
Comments.Text: Thankyou, this is a good source. So from this it leads me to believe that the output of an LSTM must be equal to the number of hidden units multiplied by the batch size? This doesn't sound right
Comments.UserDisplayName: Recessive
Tags: <long-short-term-memory>
Title: How does an LSTM output the correct dimensions for classes?
UserDisplayName: Recessive
ThreadText: # How does an LSTM output the correct dimensions for classes?
  
  (Asked by: username_0 on 2019-08-31)
  
  Take the below LSTM:
  
  ```
  input: 5x1 matrix
  hidden units: 256
  output size (aka classes, 1 hot vector): 10x1 matrix
  
  ```
  
  It is my understanding that an LSTM of this size will do the following:
  
  $w\_x$ = weight matrix at $x$
  
  $b\_x$ = bias matrix at $x$
  
  activation\_gate = tanh($w\_1$ $\cdot$ input + $w\_2$ $\cdot$ prev\_output + $b\_1$)
  
  input\_gate = sigmoid($w\_3$ $\cdot$ input + $w\_4$ $\cdot$ prev\_output + $b\_2$)
  
  forget\_gate = sigmoid($w\_5$ $\cdot$ input + $w\_6$ $\cdot$ prev\_output + $b\_3$)
  
  output\_gate = sigmoid($w\_7$ $\cdot$ input + $w\_8$ $\cdot$ prev\_output + $b\_4$)
  
  The size of the output of each gate should be equal to the number of hidden units, ie, 256. The problem arrises when trying to convert to the correct final output size, of 10. If the forget gate outputs 256, then it is summed with the element wise product of the activation and input gate to find the new state, this will result in a hidden state of size 256. (Also in all my research I have not found anywhere whether this addition is actually addition, or simply appending the two matrices).
  
  So if I have a hidden state of 256, and the output gate outputs 256, doing an element wise product of these two results in, surprise surprise, 256, not 10. If I instead ensure the output gate outputs a size of 10, this no longer works with the hidden state in an element wise product.
  
  How is this handled? I can come up with many ways of doing it myself, but I want an identical replica of the basic LSTM unit, as I have some theories I want to test, and if it is even the slightest bit different it would make the research invalid.
  
  **Tags:** long-short-term-memory
  
  > **Comments:**
  > **username_1**: I found this Quora answer to be helpful in understanding the dimensions at each point in the LSTM cell. [Dimensions of LSTM](https://www.quora.com/In-LSTM-how-do-you-figure-out-what-size-the-weights-are-supposed-to-be)  
  > **username_0**: Thankyou, this is a good source. So from this it leads me to believe that the output of an LSTM must be equal to the number of hidden units multiplied by the batch size? This doesn't sound right  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-09-03. Score: 0 | Accepted answer
  
  After some more research, I have found the answer.
  
  An LSTM is comprised of 1 LSTM cell that is continuously updated by passing in new inputs, the hidden state and the previous output. All nodes inside the LSTM cell are of size hidden\_units. That means the output of the activation gate, input gate, forget gate and output gate are all of size hidden\_units. As an extension of this, the size of the hidden state is also equal to hidden\_units.
  
  The problem arises when the desired output is not equal to that of hidden\_units. This is fixed simple and easily, by slapping a basic feed forward net onto the end of the output of the LSTM, so even if it outputs 256, you can convert it to n nodes. These final output nodes are typically put through a softmax with cross-entropy loss applied.
  
  I came to the conclusion by looking at the code included here: <https://towardsdatascience.com/lstm-by-example-using-tensorflow-feb0c1968537>
  
  In which you can see the final output has its own separate weights and bias'.
  
  
  ---
  
  