Recurrent Neural Networks (RNNs) are the state-of-the-art approach to sequential learning. However, standard RNNs use the same amount of computation at each timestep, regardless of the input data. As a result, even for high-dimensional hidden states, all dimensions are updated at each timestep no matter the choice of recurrent memory cell. Reducing this rigid assumption could allow for models with large hidden states to perform inference more quickly. Intuitively, not all hidden state dimensions need to be recomputed from scratch at each timestep. Thus, recent methods have begun studying this problem by imposing mainly a priori-determined patterns for updating the states at each step. In contrast, we now design a fullylearned approach, SA-RNN, that augments any RNN by predicting discrete update patterns at the fine granularity of independent hidden state dimensions. This is achieved through the parameterization of a distribution of update-likelihoods driven by the input data. Notably, through this approach we impose no assumptions on the structure of the update pattern. Better yet, our method adapts the update patterns online, allowing different dimensions to be updated conditional to the input. To learn which dimensions to update, the model solves a multi-objective optimization problem, maximizing task performance while minimizing the number of updates based on a unified control. Using publicly-available datasets we demonstrate that our method consistently achieves higher accuracy with fewer updates compared to state-of-the-art alternatives. Additionally, our method can be directly applied to a wide variety of models containing RNN architectures.
Summary: This paper proposes selective activation RNN (SA-RNN), by using an update coordinator to determine which subset of the RNN’s hidden state dimensions should be updated at a given timestep. The proposed loss term is then a sum of the original objective (e.g. classification) and a weighted sum of the probability that each dimension will be updated for each timestep. The method is evaluated on 3 time series datasets: Seizures, TwitterBuzz, Yahoo. 
Decision: Weak Reject. Although the authors tackle a challenging problem, their empirical results are lacking to provably demonstrate that their approach outperforms existing baselines.
Supporting Arguments/Feedback: The authors compare SA-RNN to 5 baselines: random updates, clockwork RNN, phased LSTM, Skip RNN, and VC-GRU. Although I appreciated the authors’ comparison across the suite of methods with respect to various metrics (e.g. # FLOPS, proportion of neurons that weren’t updated, etc.), the experiments were conducted on datasets that were relatively simple. For example, in prior work, the empirical evaluations were on much larger-scale datasets such as Wikipedia [Shen et. al 2019], real clinical data sources [Liu et. al 2018], and Charades videos [Campos et. al 2018], among others. I would be very interested to see how this training procedure fairs when evaluated on much more complex tasks, and would make the results about computational speedups at train/test time much more convincing.
Questions:
- I’m curious if you tried different types of gradient estimators to get around the non-differentiability rather than the straight-through estimator. Also how was the slope-annealing conducted (e.g. annealing schedule)?