Body: <p>I have gone through some theoretical introductions of RNN and LSTM, which do not contain any code, and they describe in fair detail what the cells do, how they apply operations like forget, sigmoid, etc.</p>
  <p>Then I am trying to implement them with tensorflow, and even after reading the documentation, I am unable to connect the layers' API with my theoretical understanding of the operations. For example, take the following simple code:</p>
  <pre><code>import tensorflow as tf # tensorflow 2.5.0
  inputs=tf.random.normal(shape=(32, 10, 8))
  lstm = tf.keras.layers.LSTM(units=4, return_sequences=True, return_state=True)
  outputs=lstm(inputs) # Call the layer, gives a list of three tensors
  lstm.trainable_weights # Gives a list of three tensors 
  </code></pre>
  <p>So what exactly is the layer doing here based on the input it receives and the weights that were initialised randomly?</p>
  <p>If I am to implement the layer's operation myself, how do I do that?</p>
  <p>The Google and Keras documentation contain a lot of example code, but not really explanations of the internal mathematical operations. So any help in this area, or any reference that explains the mathematical operations (not in general, but what's happening in the Tensorflow layer) would be greatly appreciated.</p>
  <p>I have the exact same question regarding RNN and GRU layers too.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-10T06:46:55.543Z
Comments.Text: I think what you're asking for here is too much to expect in an answer. I'd also be curious to know the details of the mathematical implementation in Tensorflow; and this is the type of stuff I'd expect documentation for. Tensorflow is open-source software, so you can find all of the details here https://github.com/tensorflow/tensorflow ; are you sure there isn't documentation here for what you're looking for? At worst, since it is open-source, you can look through the code yourself (which is likely correct and of high quality, since it is a popular and well-regarded open-source project).
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-10T09:35:24.877Z
Comments.Text: I have to agree with @ThePointer .  In fact, to find out the details of what these layers "do", you have two accessible options:  The documentation and the source code.  The code that implements the LSTM class is completely open as part of the tensorflow package.  Perhaps the right course is to look at those two things and then bring specific additional questions back here.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-12T10:15:20.067Z
Comments.Text: Thanks. The documentation as I see it (https://keras.io/api/layers/recurrent_layers/simple_rnn/) does not give much hint about it. So the last option is to go through the tensorflow source code. I was not expecting a complete answer here, but was wondering whether there are any other resources on the web apart from going through the source code myself.
Comments.UserDisplayName: Della
Tags: <tensorflow><keras><recurrent-neural-networks><long-short-term-memory><gated-recurrent-unit>
Title: What do RNN, LSTM, and GRU layers do in Tensorflow?
UserDisplayName: Della
ThreadText: # What do RNN, LSTM, and GRU layers do in Tensorflow?
  
  (Asked by: username_0 on 2021-06-09)
  
  I have gone through some theoretical introductions of RNN and LSTM, which do not contain any code, and they describe in fair detail what the cells do, how they apply operations like forget, sigmoid, etc.
  Then I am trying to implement them with tensorflow, and even after reading the documentation, I am unable to connect the layers' API with my theoretical understanding of the operations. For example, take the following simple code:
  import tensorflow as tf # tensorflow 2.5.0
  inputs=tf.random.normal(shape=(32, 10, 8))
  lstm = tf.keras.layers.LSTM(units=4, return_sequences=True, return_state=True)
  outputs=lstm(inputs) # Call the layer, gives a list of three tensors
  lstm.trainable_weights # Gives a list of three tensors 
  
  So what exactly is the layer doing here based on the input it receives and the weights that were initialised randomly?
  If I am to implement the layer's operation myself, how do I do that?
  The Google and Keras documentation contain a lot of example code, but not really explanations of the internal mathematical operations. So any help in this area, or any reference that explains the mathematical operations (not in general, but what's happening in the Tensorflow layer) would be greatly appreciated.
  I have the exact same question regarding RNN and GRU layers too.
  
  **Tags:** tensorflow, keras, recurrent-neural-networks, long-short-term-memory, gated-recurrent-unit
  > **Comments:**
  > **username_1**: I think what you're asking for here is too much to expect in an answer. I'd also be curious to know the details of the mathematical implementation in Tensorflow; and this is the type of stuff I'd expect documentation for. Tensorflow is open-source software, so you can find all of the details here https://github.com/tensorflow/tensorflow ; are you sure there isn't documentation here for what you're looking for? At worst, since it is open-source, you can look through the code yourself (which is likely correct and of high quality, since it is a popular and well-regarded open-source project).  
  > **username_2**: I have to agree with @ThePointer .  In fact, to find out the details of what these layers "do", you have two accessible options:  The documentation and the source code.  The code that implements the LSTM class is completely open as part of the tensorflow package.  Perhaps the right course is to look at those two things and then bring specific additional questions back here.  
  > **username_0**: Thanks. The documentation as I see it (https://keras.io/api/layers/recurrent_layers/simple_rnn/) does not give much hint about it. So the last option is to go through the tensorflow source code. I was not expecting a complete answer here, but was wondering whether there are any other resources on the web apart from going through the source code myself.  
  
  ---
  
  ## 0 answers
  
  