Answers.Body: <p>It is actually very simple, it is not about forgetting, but not learning at all means no memorization at all.</p>
  <p>For long term dependencies, since the gradient is very small, what is learned is very little, so nothing was memorized either.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-16T07:11:55.733Z
Answers.LastActivityDate: 2022-11-16T07:11:55.733Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>I am totally aware of the problem of the vanishing gradient. It usually occurs with vanilla RNN, where with a long sequence of data, the gradient will vanish or explode for far input sequence, and hence, the network will not learn from far inputs. This is fine, however, the side effect of this problem is making the RNN lose the ability to memorize long sequences (memorizing far inputs). I don't understand what is the relationship between not learning the far inputs and not memorizing the far inputs.</p>
  <p>In other words, how does not learning far inputs make the RNN forget far inputs?</p>
  
Tags: <recurrent-neural-networks><generalization><vanishing-gradient-problem>
Title: How does not learning far inputs make the RNN forget far inputs?
UserDisplayName: John adams
ThreadText: # How does not learning far inputs make the RNN forget far inputs?
  
  (Asked by: username_0 on 2022-11-15)
  
  I am totally aware of the problem of the vanishing gradient. It usually occurs with vanilla RNN, where with a long sequence of data, the gradient will vanish or explode for far input sequence, and hence, the network will not learn from far inputs. This is fine, however, the side effect of this problem is making the RNN lose the ability to memorize long sequences (memorizing far inputs). I don't understand what is the relationship between not learning the far inputs and not memorizing the far inputs.
  
  In other words, how does not learning far inputs make the RNN forget far inputs?
  
  **Tags:** recurrent-neural-networks, generalization, vanishing-gradient-problem
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-16. Score: 1 | Accepted answer
  
  It is actually very simple, it is not about forgetting, but not learning at all means no memorization at all.
  
  For long term dependencies, since the gradient is very small, what is learned is very little, so nothing was memorized either.
  
  
  ---
  
  