Answers.Body: <p>It is not the sigmoid in particular.
  LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a &quot;canvas&quot; in which the model can decide what to write (and thus keep in memory) and what to erase (and thus what to forget).</p>
  <p>Observe the top horizontal line in the image below. The line represents the &quot;internal state&quot; <span class="math-container">$C$</span>. Such state is not used as output for the outside world; it is only passed from the cell to itself at different time steps. This is where the old information is &quot;stored&quot;.</p>
  <p>The sigmoid is a convenient means to write information in <span class="math-container">$C$</span>, as the output of a sigmoid ranges between <span class="math-container">$[0,1]$</span>. For example: think of computing point-wise multiplication between <span class="math-container">$C$</span> and some other generic vector <span class="math-container">$v$</span>. If all values of <span class="math-container">$v$</span> are in range <span class="math-container">$[0,1]$</span>, we are essentially <em>masking</em> <span class="math-container">$C$</span>, i.e. selectively erasing information from it.</p>
  <p>LSTMs try to <strong>learn</strong> the values of such vector <span class="math-container">$v$</span>. To make sure that is it in the desired range, sigmoid activations are used.</p>
  <p><a href="https://i.sstatic.net/IRMKM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IRMKM.png" alt="LSTM cell" /></a></p>
  <p>For more information, I recommend consulting Goodfellow's book Deep Learning, that has <a href="https://www.deeplearningbook.org/contents/rnn.html" rel="nofollow noreferrer">a section dedicated to recurrent nets and sequence modeling</a> in general.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-05T14:08:41.540Z
Answers.Text: Thanks for the answer. Pictured is the Gated Recurrent Unit, correct? So it is this unit in particular that is responsible for the keeping and forgetting of past information from different time scales, rather than just the sigmoid activation itself?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-05T14:35:34.857Z
Answers.Text: The picture is **not** a Gated Recurrent Unit. It is a Long Short Term Memory ([LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory)) unit, which is similar to a GRU, but not exactly the same. Apart from that, you are correct: it is not the sigmoid that is responsible for keeping and forgetting information, but rather the cell architecture as a whole.
Answers.UserDisplayName: m-pana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-04T21:18:24.093Z
Answers.LastActivityDate: 2021-06-04T21:18:24.093Z
Answers.UserDisplayName: m-pana
Body: <p>My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long Short-Term Memory, model the keeping and forgetting mechanisms explicitly with sigmoid activations, namely gates. What is it about <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noreferrer">sigmoid</a> <a href="https://en.wikipedia.org/wiki/Activation_function" rel="nofollow noreferrer">activations</a> in particular that allows for the keeping and forgetting of past information from different time scales?</p>
  
Tags: <recurrent-neural-networks><long-short-term-memory><activation-functions><sigmoid><gated-recurrent-unit>
Title: What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?
UserDisplayName: The Pointer
ThreadText: # What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?
  
  (Asked by: username_0 on 2021-06-04)
  
  My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long Short-Term Memory, model the keeping and forgetting mechanisms explicitly with sigmoid activations, namely gates. What is it about [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) [activations](https://en.wikipedia.org/wiki/Activation_function) in particular that allows for the keeping and forgetting of past information from different time scales?
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, activation-functions, sigmoid, gated-recurrent-unit
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-04. Score: 1 | Accepted answer
  
  It is not the sigmoid in particular.
  LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a "canvas" in which the model can decide what to write (and thus keep in memory) and what to erase (and thus what to forget).
  
  Observe the top horizontal line in the image below. The line represents the "internal state" $C$. Such state is not used as output for the outside world; it is only passed from the cell to itself at different time steps. This is where the old information is "stored".
  
  The sigmoid is a convenient means to write information in $C$, as the output of a sigmoid ranges between $[0,1]$. For example: think of computing point-wise multiplication between $C$ and some other generic vector $v$. If all values of $v$ are in range $[0,1]$, we are essentially *masking* $C$, i.e. selectively erasing information from it.
  
  LSTMs try to **learn** the values of such vector $v$. To make sure that is it in the desired range, sigmoid activations are used.
  
  [![LSTM cell](https://i.sstatic.net/IRMKM.png)](https://i.sstatic.net/IRMKM.png)
  
  For more information, I recommend consulting Goodfellow's book Deep Learning, that has [a section dedicated to recurrent nets and sequence modeling](https://www.deeplearningbook.org/contents/rnn.html) in general.
  
  > **Comments:**
  > **username_0**: Thanks for the answer. Pictured is the Gated Recurrent Unit, correct? So it is this unit in particular that is responsible for the keeping and forgetting of past information from different time scales, rather than just the sigmoid activation itself?  
  > **username_1**: The picture is **not** a Gated Recurrent Unit. It is a Long Short Term Memory ([LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory)) unit, which is similar to a GRU, but not exactly the same. Apart from that, you are correct: it is not the sigmoid that is responsible for keeping and forgetting information, but rather the cell architecture as a whole.  
  
  ---
  
  