Answers.Body: <p><a href="https://i.sstatic.net/ZNnrA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZNnrA.png" alt="enter image description here" /></a></p>
  <p>This is an image to better understand lstm... At <span class="math-container">$f_t$</span>, we are taking the sigmoid of a weight matrix * the input at the current timestep + another weight matrix * <span class="math-container">$h_{t-1}$</span></p>
  <p>Code Sample for <span class="math-container">$f_t$</span>:</p>
  <pre><code>import numpy as np
  import math
  
  def sigmoid(values):
      sigmoid_applied = []
      for value in values:
          result = 1 / (1 + math.pow(math.e, -value))
          sigmoid_applied.append(result)
      return np.array(sigmoid_applied)
  
  w1 = np.random.uniform(0, 1, size=[hidden_vector_len, input_len])
  w2 = np.random.uniform(0, 1, size=[hidden_vector_len, hidden_vector_len])
  
  f_t = sigmoid(np.dot(w1, input) + np.dot(w2, prev_hidden_state)) # Its matrix multiplication and not just simple multiplication
  </code></pre>
  <p><strong>Note - There is also a bias term which I haven't included here for simplicity</strong></p>
  <p>If you understood <span class="math-container">$f_t$</span>, you can do the same for other states also.</p>
  <p>If you feel I am wrong anywhere in this post, then please do consider adding a comment</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-10T09:07:32.053Z
Answers.Text: Shouldn't the size of w2 be [hidden_vector_len, hidden_vector_len] ?
Answers.UserDisplayName: razvanc92
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-09T11:52:08.657Z
Answers.LastActivityDate: 2020-12-10T11:30:02.633Z
Answers.LastEditDate: 2020-12-10T11:30:02.633Z
Answers.UserDisplayName: NITIN AGARWAL
Body: <p>I want to code up one time step in a LSTM. My focus is on understanding the functioning of the <strong>forget gate layer</strong>, <strong>input gate layer</strong>, <strong>candidate values</strong>, <strong>present</strong> and <strong>future</strong> <strong>cell states</strong>.</p>
  <p>Lets assume that my hidden state at t-1 and xt are the following. For simplicity, lets assume that the weight matrices are identity matrices, and  all biases are zero.</p>
  <pre><code>htminus1 = np.array( [0, 0.5, 0.1, 0.2, 0.6] )
  xt = np.array( [-0.1, 0.3, 0.1, -0.25, 0.1] )
  </code></pre>
  <p>I understand that forget state is sigmoid of <code>htminus1</code> and <code>xt</code></p>
  <p>So, is it?</p>
  <pre><code>ft = 1 / ( 1 + np.exp( -( htminus1 + xt ) ) )
  
  &gt;&gt; ft = array([0.47502081, 0.68997448, 0.549834  , 0.4875026 , 0.66818777])
  </code></pre>
  <p>I am referring to <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">this link</a> to implement of one iteration of one block LSTM. The link says that <code>ft</code> should be 0 or 1. Am I missing something here?</p>
  <p><strong>How do I get the forget gate layer as per schema given in the below mentioned picture? An example will be illustrative for me.</strong></p>
  <p><a href="https://i.sstatic.net/ZN8k0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZN8k0.png" alt="enter image description here" /></a></p>
  <p>Along the same lines, <strong>how do I get the input gate layer, <code>it</code> and vector of new candidate values, <code>\tilde{C}_t</code> as per the following picture?</strong></p>
  <p><a href="https://i.sstatic.net/NoG1h.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NoG1h.png" alt="enter image description here" /></a></p>
  <p>Finally, <strong>how do I get the new hidden state <code>ht</code> as per the scheme given in the following picture?</strong></p>
  <p>A simple, example will be helpful for me in understanding. Thanks in advance.</p>
  <p><a href="https://i.sstatic.net/meCCp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/meCCp.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><python><recurrent-neural-networks><long-short-term-memory>
Title: Understanding LSTM through example
UserDisplayName: Ling Guo
ThreadText: # Understanding LSTM through example
  
  (Asked by: username_0 on 2020-12-09)
  
  I want to code up one time step in a LSTM. My focus is on understanding the functioning of the forget gate layer, input gate layer, candidate values, present and future cell states.
  Lets assume that my hidden state at t-1 and xt are the following. For simplicity, lets assume that the weight matrices are identity matrices, and  all biases are zero.
  htminus1 = np.array( [0, 0.5, 0.1, 0.2, 0.6] )
  xt = np.array( [-0.1, 0.3, 0.1, -0.25, 0.1] )
  
  I understand that forget state is sigmoid of htminus1 and xt
  So, is it?
  ft = 1 / ( 1 + np.exp( -( htminus1 + xt ) ) )
  
  >> ft = array([0.47502081, 0.68997448, 0.549834  , 0.4875026 , 0.66818777])
  
  I am referring to this link to implement of one iteration of one block LSTM. The link says that ft should be 0 or 1. Am I missing something here?
  How do I get the forget gate layer as per schema given in the below mentioned picture? An example will be illustrative for me.
  
  Along the same lines, how do I get the input gate layer, it and vector of new candidate values, \tilde{C}_t as per the following picture?
  
  Finally, how do I get the new hidden state ht as per the scheme given in the following picture?
  A simple, example will be helpful for me in understanding. Thanks in advance.
  
  **Tags:** neural-networks, python, recurrent-neural-networks, long-short-term-memory
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-09. Score: 1
  
  This is an image to better understand lstm... At $f_t$, we are taking the sigmoid of a weight matrix * the input at the current timestep + another weight matrix * $h_{t-1}$
  Code Sample for $f_t$:
  import numpy as np
  import math
  
  def sigmoid(values):
      sigmoid_applied = []
      for value in values:
          result = 1 / (1 + math.pow(math.e, -value))
          sigmoid_applied.append(result)
      return np.array(sigmoid_applied)
  
  w1 = np.random.uniform(0, 1, size=[hidden_vector_len, input_len])
  w2 = np.random.uniform(0, 1, size=[hidden_vector_len, hidden_vector_len])
  
  f_t = sigmoid(np.dot(w1, input) + np.dot(w2, prev_hidden_state)) # Its matrix multiplication and not just simple multiplication
  
  Note - There is also a bias term which I haven't included here for simplicity
  If you understood $f_t$, you can do the same for other states also.
  If you feel I am wrong anywhere in this post, then please do consider adding a comment
  
  > **Comments:**
  > **username_2**: Shouldn't the size of w2 be [hidden_vector_len, hidden_vector_len] ?  
  
  ---
  
  