Answers.Body: <p>An identity recurrent neural network (IRNN) is a <em>vanilla</em> recurrent neural network (as opposed to e.g. <a href="https://www.bioinf.jku.at/publications/older/2604.pdf" rel="nofollow noreferrer">LSTMs</a>) whose recurrent weight matrices are initialized with the <em>identity matrix</em>, the biases are initialized to zero, and the hidden units (or neurons) use the rectified linear unit (ReLU).</p>
  
  <p>An IRNN can be trained more easily using gradient descent (as opposed to a vanilla RNN that is not an IRNN), given that it behaves similarly to an LSTM-based RNN, that is, an IRNN does not suffer (much) from the <em>vanishing gradient problem</em>. </p>
  
  <p>The IRNN achieves a performance similar to LSTM-based RNNs in certain tasks, including the <em>adding problem</em> (a standard problem that is used to examine the power of recurrent models in learning long-term dependencies). In terms of architecture, the vanilla RNNs are much simpler than LSTM-based RNNs, so this is an advantage.</p>
  
  <p>For more details, see the paper <a href="https://arxiv.org/pdf/1504.00941.pdf" rel="nofollow noreferrer">A Simple Way to Initialize Recurrent Networks of Rectified Linear Units</a> (2015), by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. See also <a href="https://github.com/keras-team/keras/blob/master/examples/mnist_irnn.py" rel="nofollow noreferrer">this Keras implementation of the MNIST experiment</a> described in the linked paper.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-04T20:08:42.683Z
Answers.LastActivityDate: 2019-08-04T20:08:42.683Z
Answers.UserDisplayName: nbro
Body: <p>What is an identity recurrent neural network (IRNN)? What is the difference between an IRNN and RNN?</p>
  
Tags: <recurrent-neural-networks><comparison><definitions><papers>
Title: What is an identity recurrent neural network?
UserDisplayName: nbro
ThreadText: # What is an identity recurrent neural network?
  
  (Asked by: username_0 on 2019-08-04)
  
  What is an identity recurrent neural network (IRNN)? What is the difference between an IRNN and RNN?
  
  **Tags:** recurrent-neural-networks, comparison, definitions, papers
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-08-04. Score: 1
  
  An identity recurrent neural network (IRNN) is a *vanilla* recurrent neural network (as opposed to e.g. [LSTMs](https://www.bioinf.jku.at/publications/older/2604.pdf)) whose recurrent weight matrices are initialized with the *identity matrix*, the biases are initialized to zero, and the hidden units (or neurons) use the rectified linear unit (ReLU).
  
  An IRNN can be trained more easily using gradient descent (as opposed to a vanilla RNN that is not an IRNN), given that it behaves similarly to an LSTM-based RNN, that is, an IRNN does not suffer (much) from the *vanishing gradient problem*.
  
  The IRNN achieves a performance similar to LSTM-based RNNs in certain tasks, including the *adding problem* (a standard problem that is used to examine the power of recurrent models in learning long-term dependencies). In terms of architecture, the vanilla RNNs are much simpler than LSTM-based RNNs, so this is an advantage.
  
  For more details, see the paper [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://arxiv.org/pdf/1504.00941.pdf) (2015), by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton. See also [this Keras implementation of the MNIST experiment](https://github.com/keras-team/keras/blob/master/examples/mnist_irnn.py) described in the linked paper.
  
  
  ---
  
  