Answers.Body: <p>LSTMs use gates (forget, input, and output gates) and memory cells to control the flow of information through the network over time. As you've mentioned, these gates and memory cells allow the newtork to keep or discard information from previous time steps. In more detail, the forget gate uses the current input and previous hidden state to compute a value between 0 and 1 that represents the proportion of information from the previous cell state to forget. This gate, together with the input gate and the cell state update, enables the LSTM to selectively remember or forget information over time. As you pointed out, it's simply a sigmoid function that outputs values between 0 (forget completely) and 1 (remember everything).</p>
  <p>Although the forget gate directly influences the <em>forgetting</em> process in an LSTM, it uses information from the current input and previous hidden state to decide how much of the previous cell state to retain or discard. So no, the the forget gate doesn't &quot;decide forgetting&quot; based on embeddings or so.</p>
  <p>However, keep in mind that each gate has its own set of learnable parameters, which are trained during the learning process through backpropagation, aiming to minimise the cost function. Based on these weights, the forget gate can determine how much of the past information to forget, and these weights are adjusted during training to improve the model's performance on the given task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T15:43:21.297Z
Answers.LastActivityDate: 2024-02-19T15:43:21.297Z
Answers.UserDisplayName: tanasr
Body: <p>Currently, I am learning about LSTM, and I understand the intuition behind it, such as how forget gate works (sigmoid function yields a value between 0 and 1; if it is 0 it &quot;completely&quot; forgets, if it is 1 &quot;it remembers everything&quot;). What I don't understand is how this process unfolds. I mean, is it because of embeddings, as in, we store and keep them, thus deciding what to forget based on them?</p>
  <p>A clear explanation would be helpful.</p>
  
Tags: <neural-networks><training><recurrent-neural-networks><long-short-term-memory>
Title: How LSTM really decide what to forget and not?
UserDisplayName: Ashraf
ThreadText: # How LSTM really decide what to forget and not?
  
  (Asked by: username_0 on 2024-02-19)
  
  Currently, I am learning about LSTM, and I understand the intuition behind it, such as how forget gate works (sigmoid function yields a value between 0 and 1; if it is 0 it "completely" forgets, if it is 1 "it remembers everything"). What I don't understand is how this process unfolds. I mean, is it because of embeddings, as in, we store and keep them, thus deciding what to forget based on them?
  
  A clear explanation would be helpful.
  
  **Tags:** neural-networks, training, recurrent-neural-networks, long-short-term-memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-19. Score: 2 | Accepted answer
  
  LSTMs use gates (forget, input, and output gates) and memory cells to control the flow of information through the network over time. As you've mentioned, these gates and memory cells allow the newtork to keep or discard information from previous time steps. In more detail, the forget gate uses the current input and previous hidden state to compute a value between 0 and 1 that represents the proportion of information from the previous cell state to forget. This gate, together with the input gate and the cell state update, enables the LSTM to selectively remember or forget information over time. As you pointed out, it's simply a sigmoid function that outputs values between 0 (forget completely) and 1 (remember everything).
  
  Although the forget gate directly influences the *forgetting* process in an LSTM, it uses information from the current input and previous hidden state to decide how much of the previous cell state to retain or discard. So no, the the forget gate doesn't "decide forgetting" based on embeddings or so.
  
  However, keep in mind that each gate has its own set of learnable parameters, which are trained during the learning process through backpropagation, aiming to minimise the cost function. Based on these weights, the forget gate can determine how much of the past information to forget, and these weights are adjusted during training to improve the model's performance on the given task.
  
  
  ---
  
  