Deep recurrent neural networks perform well on sequence data and are the model of choice. However, it is a daunting task to decide the structure of the networks, i.e. the number of layers, especially considering different computational needs of a sequence. We propose a layer flexible recurrent neural network with adaptive computation time, and expand it to a sequence to sequence model. Different from the adaptive computation time model, our model has a dynamic number of transmission states which vary by step and sequence. We evaluate the model on a financial data set and Wikipedia language modeling. Experimental results show the performance improvement of 7% to 12% and indicate the model's ability to dynamically change the number of layers along with the computational steps.
Summary:
The authors propose Layer Flexible Adaptive Computation Time, an RNN-esque sequence model with varying depth at each time step. The idea is that the model can adaptively choose how much computational effort to spend on each example. The authors evaluate the model empirically on a financial dataset and Wikipedia language modeling, and find that it outperforms a vanilla RNN and the original adaptive computation time (ACT) model.
Unfortunately, the presentation of the idea is unclear, the idea itself is not very novel, and the experimental evaluation is lacking. These weaknesses lead me to vote for a weak reject.
I address specific clarity points below.
In regards to the novelty claim, there have been several developments of depth-based (as opposed to time-based) adaptive computation time in the literature, for example:
[1] McGill et al 2017 "Deciding How to Decide: Dynamic Routing in Artificial Neural Networks"
[2] Bolukbasi et al 2017 "Adaptive Neural Networks for Efficient Inference"
[3] Figurnov et al 2017 "Spatially Adaptive Computation Time for Residual Networks" (this paper is cited by the authors)
These papers do not present sequence models, but the ideas in them readily apply to sequence models. Thus, the main novelty in the authors' paper is handling the different number of hidden states at each timestep via their 'attention mechanism'. While this is definitely a contribution, the unclear presentation and lacking experimental evaluation combine to decrease the value of the paper.
Experimentally, the authors evaluate on a financial time-series dataset and Wikipedia language modeling. They compare to Adaptive computation time and a standard RNN. While the experiments demonstrate a modest improvement over ACT and an RNN, they do not compare on larger, more standard datasets such as the WMT datasets, etc... Additionally, they do not compare with other, more powerful models. Both are required to thoroughly demonstrate their model's effectiveness.
To change my mind the authors would have to (in order of importance):
1) Add experiments on WMT or other bigger datasets and compare with current SOTA models.
2) Thoroughly edit their paper for clarity (specific points below).
The authors may want also add explorations of how much computation time can be saved using their model versus others, as this is very common in the ACT-esque literature.
Specific points:
* While the original ACT paper does use 'mean-field' to denote the convex combination of states at a current timestep, that term has a specific technical meaning different from how it is used here. I would suggest using a different word.
* The introduction is too long and repeats itself in several places.
* There are repeated citations in the literature review.
* In figure 1 sometimes nodes denote functions and sometimes nodes denote outputs. Sometimes the nodes are round and sometimes they are rectangular. Sometimes arrows denote inputs to a function, and sometimes they denote multiplication. These inconsistencies make the figure very hard to decipher.
* The text description of your model is confusing. Specifically, distinguishing between the functions of u_t, \hat{u}_t and \overline{u}_t was difficult.
* In your experiments section the plots are very difficult to interpret because they are phrase as 'improvement over x'. The standard presentation is a table of absolute results. Furthermore, bar charts can be misleading because the scale can make improvements seem bigger than they actually are. The figures should stand alone without having to read the text.