Capacity and Trainability in Recurrent Neural Networks
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.
The authors investigate a variety of existing and two new RNN architectures to obtain more insight about the effectiveness at which these models can store task information in their parameters and activations. The experimental setups look sound. To generalize comparisons between different architectures it’s necessary to consider multiple tasks and control for the effect of the hyperparameters. This work uses multiple tasks of varying complexities, principled hyperparameter tuning methodology and a number of tuning iterations that can currently only be achieved by the computational resources of some of the larger industrial research groups. The descriptions of the models and the objective where very clear to me. The descriptions of the experiments and presentation of the results were not always clear to me at times, even with the additional details in the appendix available. Most of these issues can easily be resolved by editing the text. For example, in the memory task the scaling of the inputs (and hence also outputs) is not provided so it’s hard to interpret the squared error scores in Figure 2c. It’s not clear to me what the term ‘unrollings’ refers to in Figure 2b. Is this a time lag with additional hidden state updates between the presentation of the input sequence and the generation of the output? Since the perceptron capacity task is somewhat central to the paper, I think a slightly more precise description of how and when the predictions are computed would be helpful. Due to the large number of graphs, it can be somewhat hard to find the most relevant results. Perhaps some of the more obvious findings (like Figure 1(b-d) given Figure 1a) could move to the appendix to make space for more detailed task descriptions. Novelty is not really the aim of this paper since it mostly investigates existing architectures. To use the mutual information to obtain bits per parameter scores in highly non-linear parameterized functions is new to me. The paper also proposed to new architectures that seem to have practical value. The paper adds to the currently still somewhat neglected research effort to employ the larger computational resources we currently have towards a better understanding of architectures which were designed when such resources were not yet available. I’d argue that the paper is original enough for that reason alone. The paper provides some interesting new insights into the properties of RNNs. While observed before, it is interesting to see the importance of gated units for maintaining trainability of networks with many layers. It is also interesting to see a potential new use for vanilla RNNs for simpler tasks where a high capacity per parameter may be required due to hardware constraints. The proposed +RNN may turn out to have practical value as well and the hyperparameter robustness results shed some light on the popularity of certain architectures when limited time for HP tuning is available. The large body of results and hyperparameter analysis should be useful to many researchers who want to use RNNs in the future. All in all, I think this paper would make a valuable addition to the ICLR conference but would benefit from some improvements to the text. Pros: * Thorough analysis. * Seemingly proper experiments. * The way of quantifying capacity in neural networks adds to the novelty of the paper. * The results have some practical value and suggest similar analysis of other architectures. * The results provide useful insights into the relative merits of different RNN architectures. Cons: * It’s hard to isolate the most important findings (some plots seem redundant). * Some relevant experimental details are missing.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Capacity and Trainability in Recurrent Neural Networks
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors investigate a variety of existing and two new RNN architectures to obtain more insight about the effectiveness at which these models can store task information in their parameters and activations. The experimental setups look sound. To generalize comparisons between different architectures it’s necessary to consider multiple tasks and control for the effect of the hyperparameters. This work uses multiple tasks of varying complexities, principled hyperparameter tuning methodology and a number of tuning iterations that can currently only be achieved by the computational resources of some of the larger industrial research groups. The descriptions of the models and the objective where very clear to me. The descriptions of the experiments and presentation of the results were not always clear to me at times, even with the additional details in the appendix available. Most of these issues can easily be resolved by editing the text. For example, in the memory task the scaling of the inputs (and hence also outputs) is not provided so it’s hard to interpret the squared error scores in Figure 2c. It’s not clear to me what the term ‘unrollings’ refers to in Figure 2b. Is this a time lag with additional hidden state updates between the presentation of the input sequence and the generation of the output? Since the perceptron capacity task is somewhat central to the paper, I think a slightly more precise description of how and when the predictions are computed would be helpful. Due to the large number of graphs, it can be somewhat hard to find the most relevant results. Perhaps some of the more obvious findings (like Figure 1(b-d) given Figure 1a) could move to the appendix to make space for more detailed task descriptions. Novelty is not really the aim of this paper since it mostly investigates existing architectures. To use the mutual information to obtain bits per parameter scores in highly non-linear parameterized functions is new to me. The paper also proposed to new architectures that seem to have practical value. The paper adds to the currently still somewhat neglected research effort to employ the larger computational resources we currently have towards a better understanding of architectures which were designed when such resources were not yet available. I’d argue that the paper is original enough for that reason alone. The paper provides some interesting new insights into the properties of RNNs. While observed before, it is interesting to see the importance of gated units for maintaining trainability of networks with many layers. It is also interesting to see a potential new use for vanilla RNNs for simpler tasks where a high capacity per parameter may be required due to hardware constraints. The proposed +RNN may turn out to have practical value as well and the hyperparameter robustness results shed some light on the popularity of certain architectures when limited time for HP tuning is available. The large body of results and hyperparameter analysis should be useful to many researchers who want to use RNNs in the future. All in all, I think this paper would make a valuable addition to the ICLR conference but would benefit from some improvements to the text. Pros: * Thorough analysis. * Seemingly proper experiments. * The way of quantifying capacity in neural networks adds to the novelty of the paper. * The results have some practical value and suggest similar analysis of other architectures. * The results provide useful insights into the relative merits of different RNN architectures. Cons: * It’s hard to isolate the most important findings (some plots seem redundant). * Some relevant experimental details are missing.
assistant
The authors investigate a variety of existing and two new RNN architectures to obtain more insight about the effectiveness at which these models can store task information in their parameters and activations. The experimental setups look sound. To generalize comparisons between different architectures it’s necessary to consider multiple tasks and control for the effect of the hyperparameters. This work uses multiple tasks of varying complexities, principled hyperparameter tuning methodology and a number of tuning iterations that can currently only be achieved by the computational resources of some of the larger industrial research groups. The descriptions of the models and the objective where very clear to me. The descriptions of the experiments and presentation of the results were not always clear to me at times, even with the additional details in the appendix available. Most of these issues can easily be resolved by editing the text. For example, in the memory task the scaling of the inputs (and hence also outputs) is not provided so it’s hard to interpret the squared error scores in Figure 2c. It’s not clear to me what the term ‘unrollings’ refers to in Figure 2b. Is this a time lag with additional hidden state updates between the presentation of the input sequence and the generation of the output? Since the perceptron capacity task is somewhat central to the paper, I think a slightly more precise description of how and when the predictions are computed would be helpful. Due to the large number of graphs, it can be somewhat hard to find the most relevant results. Perhaps some of the more obvious findings (like Figure 1(b-d) given Figure 1a) could move to the appendix to make space for more detailed task descriptions. Novelty is not really the aim of this paper since it mostly investigates existing architectures. To use the mutual information to obtain bits per parameter scores in highly non-linear parameterized functions is new to me. The paper also proposed to new architectures that seem to have practical value. The paper adds to the currently still somewhat neglected research effort to employ the larger computational resources we currently have towards a better understanding of architectures which were designed when such resources were not yet available. I’d argue that the paper is original enough for that reason alone. The paper provides some interesting new insights into the properties of RNNs. While observed before, it is interesting to see the importance of gated units for maintaining trainability of networks with many layers. It is also interesting to see a potential new use for vanilla RNNs for simpler tasks where a high capacity per parameter may be required due to hardware constraints. The proposed +RNN may turn out to have practical value as well and the hyperparameter robustness results shed some light on the popularity of certain architectures when limited time for HP tuning is available. The large body of results and hyperparameter analysis should be useful to many researchers who want to use RNNs in the future. All in all, I think this paper would make a valuable addition to the ICLR conference but would benefit from some improvements to the text. Pros: * Thorough analysis. * Seemingly proper experiments. * The way of quantifying capacity in neural networks adds to the novelty of the paper. * The results have some practical value and suggest similar analysis of other architectures. * The results provide useful insights into the relative merits of different RNN architectures. Cons: * It’s hard to isolate the most important findings (some plots seem redundant). * Some relevant experimental details are missing.