Answers.Body: <p>For question 1) I dont understand what you are getting at... LSTM cells will work on a contiguous block of inputs, where it sequentially uses the states from a previous time step and the new input to generate the next ones.</p>
  
  <p>question 2) Please look into the LSTM atchitecture <a href="https://i.sstatic.net/BA9gg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BA9gg.png" alt="LSTM pic"></a>
  As you see, biases are already there, is there somewhere specific you want it, that it isnt?</p>
  
  <p>question 3) Generally yes, but the normalization step can be expensive (such as softmax), so if you want to get clever, you can use negative sampling or hierarchial softmax-- but generally, you you predict a probability over all possible words given the previous text</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T13:43:57.163Z
Answers.Text: Okay, I'll try to clarify: 1. A LTSM network (I assume) has more than a single cell. Say I had a 2 inputs a single hidden layer with 3 cells and 2 outputs. Would all those cells be of the LSTM variety? Would their vertical outputs connect at random to the vertical inputs? 2. I see from the diagram you included there are two horizontal line, the memory blocks and the outputs. Ommiting the vertical output and input, would the "horizontal output" feed into it's own "output of previous block"? And would the cells own "memory concurrent block" feed into it's own "memory from previous block" essenti
Answers.UserDisplayName: user11376058
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T16:21:29.047Z
Answers.Text: There is only one cell that gets reused every timestep per lstm.... and yes, the output is one of the hidden states of the next layer..... and the memory get fed forward, never backwards
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T16:51:18.997Z
Answers.Text: I see, so each cell holds its memory independent from one another, only the vertical output gets fed forward. And a network can have multiple lstm cell layers feeding information to an output layer.
Answers.UserDisplayName: user11376058
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-06T00:09:34.223Z
Answers.Text: so there is no weights or dot mult in there? how do you backpropagate that?
Answers.UserDisplayName: user8426627
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-06T12:15:33.700Z
Answers.Text: @user8426627 where ever you see a non-linearity there is a linear operation preceding it... and side note (basic ops like additions and activation fns still have gradients, so youd be able to backprop it regardless, it just wouldnt have learnable params)
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-06T12:23:42.840Z
Answers.Text: i still dont get it . pls, write calculation what are done with backprop. Andi guess it will be BPTT, so i must stove values of all items from previous  times ??
Answers.UserDisplayName: user8426627
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T11:44:05.227Z
Answers.LastActivityDate: 2019-06-05T11:44:05.227Z
Answers.UserDisplayName: mshlis
Body: <p>I have recently begun researching LSTM networks, as I have finished my GA and am looking to progress to something more difficult. I believe I am using the classic LSTM (if that makes any sense) and have a few questions.</p>
  
  <p>Do I need LSTM units everywhere in the network? For example, can I only use LSTM units for the first and last layer and use feedforward units everywhere else?</p>
  
  <p>How do I go about implementing bias values into an LSTM?</p>
  
  <p>Assuming I create a network that predicts the next few words of a sentence, does that mean my outputs should be every possible word that the network could conceivably use?</p>
  
Tags: <long-short-term-memory>
Title: Do I need LSTM units everywhere in the network?
UserDisplayName: user11376058
ThreadText: # Do I need LSTM units everywhere in the network?
  
  (Asked by: username_0 on 2019-06-05)
  
  I have recently begun researching LSTM networks, as I have finished my GA and am looking to progress to something more difficult. I believe I am using the classic LSTM (if that makes any sense) and have a few questions.
  
  Do I need LSTM units everywhere in the network? For example, can I only use LSTM units for the first and last layer and use feedforward units everywhere else?
  
  How do I go about implementing bias values into an LSTM?
  
  Assuming I create a network that predicts the next few words of a sentence, does that mean my outputs should be every possible word that the network could conceivably use?
  
  **Tags:** long-short-term-memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-05. Score: 1
  
  For question 1) I dont understand what you are getting at... LSTM cells will work on a contiguous block of inputs, where it sequentially uses the states from a previous time step and the new input to generate the next ones.
  
  question 2) Please look into the LSTM atchitecture [![LSTM pic](https://i.sstatic.net/BA9gg.png)](https://i.sstatic.net/BA9gg.png)
  As you see, biases are already there, is there somewhere specific you want it, that it isnt?
  
  question 3) Generally yes, but the normalization step can be expensive (such as softmax), so if you want to get clever, you can use negative sampling or hierarchial softmax-- but generally, you you predict a probability over all possible words given the previous text
  
  > **Comments:**
  > **username_0**: Okay, I'll try to clarify: 1. A LTSM network (I assume) has more than a single cell. Say I had a 2 inputs a single hidden layer with 3 cells and 2 outputs. Would all those cells be of the LSTM variety? Would their vertical outputs connect at random to the vertical inputs? 2. I see from the diagram you included there are two horizontal line, the memory blocks and the outputs. Ommiting the vertical output and input, would the "horizontal output" feed into it's own "output of previous block"? And would the cells own "memory concurrent block" feed into it's own "memory from previous block" essenti  
  > **username_1**: There is only one cell that gets reused every timestep per lstm.... and yes, the output is one of the hidden states of the next layer..... and the memory get fed forward, never backwards  
  > **username_0**: I see, so each cell holds its memory independent from one another, only the vertical output gets fed forward. And a network can have multiple lstm cell layers feeding information to an output layer.  
  > **username_2**: so there is no weights or dot mult in there? how do you backpropagate that?  
  > **username_1**: @username_2 where ever you see a non-linearity there is a linear operation preceding it... and side note (basic ops like additions and activation fns still have gradients, so youd be able to backprop it regardless, it just wouldnt have learnable params)  
  > **username_2**: i still dont get it . pls, write calculation what are done with backprop. Andi guess it will be BPTT, so i must stove values of all items from previous  times ??  
  
  ---
  
  