Answers.Body: <p><strong>TL;DR</strong> Here is a beautiful explanation with diagrams: <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">source</a>.</p>
  <p>To address:</p>
  <blockquote>
  <p>the cell state is essentially long term memory embedding (correct me if I'm wrong)</p>
  </blockquote>
  <p>The embedding can be long or short term and it is a vector.</p>
  <p>To answer:</p>
  <blockquote>
  <p>Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?</p>
  </blockquote>
  <p>Excerpt from source:</p>
  <blockquote>
  <p>The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”</p>
  </blockquote>
  <p>Formally the forget vector is:</p>
  <p><span class="math-container">$$f_t = \sigma(W_f\cdot[h_{t-1},x_t],+b_f)$$</span></p>
  <p>So we see that it is actually a linear operation followed by a non-linear operation, which restricts the values to be between 0 and 1. This is followed by an element-wise operation on the previous cell state. That is we &quot;gate&quot;/&quot;filter&quot; the previous cell state doing:</p>
  <p><span class="math-container">$$C_{t-1}\odot f_t,$$</span></p>
  <p>where <span class="math-container">$\odot$</span> is element-wise multiplication.</p>
  <p>Thus we see that the forget gate <em>really is acting like a gate</em>: It either lets values pass through or it pushes them toward zero. So, it's purpose is to assign what values to forget/remember.</p>
  <p>To answer:</p>
  <blockquote>
  <p>How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?</p>
  </blockquote>
  <p>This concatenation takes into account the &quot;rolling&quot; hidden state <em>and</em> the current input. That is, the linear operation results in a new vector which can be seen as a &quot;consideration embedding&quot; of the current input and a &quot;summary&quot; of past inputs and cell states. The sigmoid then converts this &quot;consideration embedding&quot; into the forget vector.</p>
  <p>In summary, the forget gate has the primary purpose of forming a vector of values between zero and one. This vector results by considering the current input and the previous hidden state. The vector is used to forget/remember parts of the previous cell state via element-wise multiplication. So, no, it is not a black box, it is a highly considerate non-linear operation through time.</p>
  <p><strong>Additional Intuition - More Mathematical</strong></p>
  <p>Recall that multiplying a matrix on the right by a column vector produces a vector, which is a linear combination of the columns of the matrix. So each column of <span class="math-container">$W_f$</span> can be seen as a column of &quot;toggles&quot; that push up to 1 or pull down to 0 (once we apply sigmoid). That is, the columns of <span class="math-container">$W_f$</span> form a sort of &quot;basis&quot; for a &quot;toggle space.&quot;</p>
  <p>When we concatenate to form <span class="math-container">$[h_{t-1},x_t]$</span> and then multiply we see that the term <span class="math-container">$W_f\cdot[h_{t-1},x_t]$</span> gives consideration to the hidden state and the input. That is, both <span class="math-container">$h_{t-1}$</span> and <span class="math-container">$x_t$</span> have there hand in toggling the gate.</p>
  <p>The LSTM has learned from the data how to make this toggling &quot;meaningful.&quot; Below is a simple pictographic example.</p>
  <p><a href="https://i.sstatic.net/HhJE3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HhJE3.png" alt="Pictographic parallel of matrix form" /></a></p>
  <p>That is, <span class="math-container">$W_f\cdot[h_{t-1},x_t]$</span> will be a vector in the column space of <span class="math-container">$W_f$</span> (ie the &quot;toggle space&quot;). Thus, the concatenated vector will &quot;point&quot; to the best &quot;toggle&quot; in the column (&quot;toggle&quot;) space of <span class="math-container">$W_f$</span>. This &quot;toggle&quot; is then transformed into a &quot;gate&quot; in the &quot;gate space.&quot;</p>
  <p>So intuitively, the range of the forget component is a sort of &quot;gate space&quot; where the domain is all vectors of form <span class="math-container">$[h_{t-1},x_t]$</span>. Formally:</p>
  <p><span class="math-container">$$f_t:[h_{t-1},x_t]\mapsto g_t$$</span></p>
  <p>That is to say that the forget gate learns the best mapping from the domain of concatenated hidden/input's to the range of possible gates.</p>
  <p><strong>Important Note</strong></p>
  <p>The LSTM is only <em>one</em> of many types of forget-gate mappings. The main takeaway is that the LSTM works <em>empirically</em> for many applications.</p>
  <p>That is a forget gate is an instance of a general &quot;gate mapping&quot;:</p>
  <p><span class="math-container">$$f_t:v_t \mapsto g_t,$$</span></p>
  <p>where <span class="math-container">$v_t$</span> could be any number of vectors resulting from any number of concatenations.</p>
  <p><strong>Conclusion</strong></p>
  <p>Some data from the past is irrelevant to the current time step. We need a way to correctly &quot;forget&quot; irrelevant information. One way to &quot;forget&quot; is to use a forget gate mapping (like the one in LSTMs). Then we need to optimize the parameters of that forget gate. <em>One</em> implementation is used in the common LSTM architecture. Finally, the concatenation of the previous hidden state provides contextual information to the current input which is very helpful with forgetting.</p>
  <p><strong>More Mathematical Intuition</strong></p>
  <p>Here is another view of what is happening in the LSTM from a dynamic point of view: <a href="https://youtu.be/jvj4WefJ0S0?list=PLiJYYsK6pD-nVcROKw2YDOVaNMjST5YyL&amp;t=1711" rel="nofollow noreferrer">video</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-15T00:14:32.827Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118474/discussion-on-answer-by-respectful-how-does-the-forget-layer-of-an-lstm-work).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-25T19:28:20.693Z
Answers.LastActivityDate: 2022-02-06T21:21:16.917Z
Answers.LastEditDate: 2022-02-06T21:21:16.917Z
Answers.UserDisplayName: respectful
Body: <p>Can someone explain the mathematical intuition behind the forget layer of an LSTM? </p>
  
  <p>So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), but I'm also assuming it's a matrix. Then the forget vector is calculated by concatenating the previous hidden state and the current input and adding the bias to it, then putting that through a sigmoid function that outputs a vector then that gets multiplied by the cell state matrix.</p>
  
  <p>How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?</p>
  
  <p>Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?</p>
  
  <p>I'd really like to understand the theory behind calculating the cell states and hidden states. Most people just tell me to treat it like a black box, but I think that, in order to have a successful application of LSTMs to a problem, I need to know what's going on under the hood. If anyone has any resources that are good for learning the theory behind why cell state and hidden state calculation extract key features in short and long term memory I'd love to read it.</p>
  
Tags: <neural-networks><long-short-term-memory><math>
Title: How does the forget layer of an LSTM work?
UserDisplayName: user8714896
ThreadText: # How does the forget layer of an LSTM work?
  
  (Asked by: username_0 on 2019-12-23)
  
  Can someone explain the mathematical intuition behind the forget layer of an LSTM?
  
  So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), but I'm also assuming it's a matrix. Then the forget vector is calculated by concatenating the previous hidden state and the current input and adding the bias to it, then putting that through a sigmoid function that outputs a vector then that gets multiplied by the cell state matrix.
  
  How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?
  
  Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?
  
  I'd really like to understand the theory behind calculating the cell states and hidden states. Most people just tell me to treat it like a black box, but I think that, in order to have a successful application of LSTMs to a problem, I need to know what's going on under the hood. If anyone has any resources that are good for learning the theory behind why cell state and hidden state calculation extract key features in short and long term memory I'd love to read it.
  
  **Tags:** neural-networks, long-short-term-memory, math
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-25. Score: 5 | Accepted answer
  
  **TL;DR** Here is a beautiful explanation with diagrams: [source](https://colah.github.io/posts/2015-08-Understanding-LSTMs/).
  
  To address:
  
  > the cell state is essentially long term memory embedding (correct me if I'm wrong)
  
  The embedding can be long or short term and it is a vector.
  
  To answer:
  
  > Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?
  
  Excerpt from source:
  
  > The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”
  
  Formally the forget vector is:
  
  $$f\_t = \sigma(W\_f\cdot[h\_{t-1},x\_t],+b\_f)$$
  
  So we see that it is actually a linear operation followed by a non-linear operation, which restricts the values to be between 0 and 1. This is followed by an element-wise operation on the previous cell state. That is we "gate"/"filter" the previous cell state doing:
  
  $$C\_{t-1}\odot f\_t,$$
  
  where $\odot$ is element-wise multiplication.
  
  Thus we see that the forget gate *really is acting like a gate*: It either lets values pass through or it pushes them toward zero. So, it's purpose is to assign what values to forget/remember.
  
  To answer:
  
  > How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?
  
  This concatenation takes into account the "rolling" hidden state *and* the current input. That is, the linear operation results in a new vector which can be seen as a "consideration embedding" of the current input and a "summary" of past inputs and cell states. The sigmoid then converts this "consideration embedding" into the forget vector.
  
  In summary, the forget gate has the primary purpose of forming a vector of values between zero and one. This vector results by considering the current input and the previous hidden state. The vector is used to forget/remember parts of the previous cell state via element-wise multiplication. So, no, it is not a black box, it is a highly considerate non-linear operation through time.
  
  **Additional Intuition - More Mathematical**
  
  Recall that multiplying a matrix on the right by a column vector produces a vector, which is a linear combination of the columns of the matrix. So each column of $W\_f$ can be seen as a column of "toggles" that push up to 1 or pull down to 0 (once we apply sigmoid). That is, the columns of $W\_f$ form a sort of "basis" for a "toggle space."
  
  When we concatenate to form $[h\_{t-1},x\_t]$ and then multiply we see that the term $W\_f\cdot[h\_{t-1},x\_t]$ gives consideration to the hidden state and the input. That is, both $h\_{t-1}$ and $x\_t$ have there hand in toggling the gate.
  
  The LSTM has learned from the data how to make this toggling "meaningful." Below is a simple pictographic example.
  
  [![Pictographic parallel of matrix form](https://i.sstatic.net/HhJE3.png)](https://i.sstatic.net/HhJE3.png)
  
  That is, $W\_f\cdot[h\_{t-1},x\_t]$ will be a vector in the column space of $W\_f$ (ie the "toggle space"). Thus, the concatenated vector will "point" to the best "toggle" in the column ("toggle") space of $W\_f$. This "toggle" is then transformed into a "gate" in the "gate space."
  
  So intuitively, the range of the forget component is a sort of "gate space" where the domain is all vectors of form $[h\_{t-1},x\_t]$. Formally:
  
  $$f\_t:[h\_{t-1},x\_t]\mapsto g\_t$$
  
  That is to say that the forget gate learns the best mapping from the domain of concatenated hidden/input's to the range of possible gates.
  
  **Important Note**
  
  The LSTM is only *one* of many types of forget-gate mappings. The main takeaway is that the LSTM works *empirically* for many applications.
  
  That is a forget gate is an instance of a general "gate mapping":
  
  $$f\_t:v\_t \mapsto g\_t,$$
  
  where $v\_t$ could be any number of vectors resulting from any number of concatenations.
  
  **Conclusion**
  
  Some data from the past is irrelevant to the current time step. We need a way to correctly "forget" irrelevant information. One way to "forget" is to use a forget gate mapping (like the one in LSTMs). Then we need to optimize the parameters of that forget gate. *One* implementation is used in the common LSTM architecture. Finally, the concatenation of the previous hidden state provides contextual information to the current input which is very helpful with forgetting.
  
  **More Mathematical Intuition**
  
  Here is another view of what is happening in the LSTM from a dynamic point of view: [video](https://youtu.be/jvj4WefJ0S0?list=PLiJYYsK6pD-nVcROKw2YDOVaNMjST5YyL&t=1711).
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118474/discussion-on-answer-by-respectful-how-does-the-forget-layer-of-an-lstm-work).  
  
  ---
  
  