Reversible Recurrent Neural Networks
Recurrent neural networks (RNNs) provide state-of-the-art performance in processing sequential data but are memory intensive to train, limiting the flexibility of RNN models which can be trained. Reversible RNNs-RNNs for which the hidden-to-hidden transition can be reversed-offer a path to reduce the memory requirements of training, as hidden states need not be stored and instead can be recomputed during backpropagation. We first show that perfectly reversible RNNs, which require no storage of the hidden activations, are fundamentally limited because they cannot forget information from their hidden state. We then provide a scheme for storing a small number of bits in order to allow perfect reversal with forgetting. Our method achieves comparable performance to traditional models while reducing the activation memory cost by a factor of 10-15. We extend our technique to attention-based sequence-to-sequence models, where it maintains performance while reducing activation memory cost by a factor of 5-10 in the encoder, and a factor of 10-15 in the decoder.
Summary: Authors investigate reversible architectures for RNN to reduce the memory requirement of their  training. They build upon the reversible residual network architecture. By allowing reversibility,  RNN can recompute the hidden activation during the backpropagation through time, trading computation for memory.   Authors argue that a naïve reversible RNNs are not able to forget, hence have trouble to solve some simple learning tasks. To tackle this issue, they propose a variant, that store the lost information due to the forgetting. Authors then perform an extensive empirical evaluation on Penn TreeBank, Wikitext-2, Multi30K and IWLST, where they show that the reversible GRU/LSTM are on part and sometime outperform their non-reversible counterparts.   Clarity,  The paper is clearly written and pleasant to read overall.   Section 4  clarity could be improved. It would be informative to report more details about the experiments (model architectures (size of the hidden states), dataset splits..) and what are the final performances of the different model. In addition, the statement ‘the remember task for an ordinary model with forgetting reduces to the repeat task for a perfectly reversible RNN’ is confusing to me?   Quality, Authors perform an extended empirical evaluation to validate their claims on several language modeling and machine translation tasks. My main concern is that the authors do not report previous works scores on those different tasks. Therefore, it is not clear if the baselines studied in this paper are competitive with respect to previous work.   Few remarks: - While section 4 results indicates the memorization capability of NF-RevGRU/NF-RevLSTM, it does not really show their ‘Impossibility of Zero Forgetting’ ?  - Given a fix hidden state size, a reversible architecture have more parameters than its non-reversible counterpart. Is this factor taken into account in the comparison? - It would be informative to reports that additional computational cost in addition to the memory saving.  Originality: To my knowledge, this is the first work exploring reversible architecture for recurrent neural networks.  Significance: The memory cost associated with training RNN is an important limitation. Tackling this issue could allow the exploration of larger RNN models.  Other approaches have try to reduce the memory cost associated with training a RNN, by investigating real-time recurrent learning or recurrent backpropagation (see ‘Unbiased Online Recurrent Optimization (Tallec et al., 2017) or ‘Reviving and Improving Recurrent Back-Propagation’ (Liao, 2018)).  How does reversible architecture compare in term of computation/memory cost to those approaches?   Overall, I think the paper tackles an important problem and I appreciate the extended empirical evaluation. For those reason I lean toward acceptance. Some aspect of the paper could be improved as reporting previous work performances on the different tasks and a comparison with other approach that also tackle the problem of RNN training  memory cost.  After Rebuttal: Thanks for the feedback, as it addresses some of my concern, I will revise my score accordingly. Also it would be nice to refer to state-of-art model for the different task in the main text for completeness.