Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs). Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy. Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences. In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks. This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT.
Claim: Backpropagation of gradients from a higher to lower level in a HRNN can be removed and replaced with auxiliary losses predicting input tokens at the lower level without affecting performance. 
Significance: The significance of the claim hinges on whether HRNNs are more effective than other methods designed to help RNNs capture long-term dependencies (e.g. stacking RNNs or using different architectures). I think the authors could make a more substantive argument why this would be the case in the introduction, but they do a nice job of situating their work in the context of the present literature.
Novelty: The proposed method is not very original, since augmenting RNNs with auxiliary losses in order to better capture long-term dependencies has been used in many previous papers. The authors mention some of these papers in the related work section.
Clarity: The paper's description of the proposed method is well-written. Some parts of the experiment section could be made clearer. 
--  I encourage the authors to invent a new acronym to refer to "our model" (perhaps aux-HRNN?). In the description of the mr-HRNN (pg. 5), I find the sentence "trained using only as much memory as our model requires for training" confusing.  I initially thought our model referred to the mr-HRNN in the setence.
-- Training settings (e.g. the number of ticks of the upper RNN) should be described at the beginning of each section. 
-- A seeming contradiction is made when discussing the results in 4.3. First, it said that because short term dependencies dominate long term dependencies it is expected that the proposed method will suffer greatly (pg. 6, bottom). In the next paragraph, it is claimed that all three models perform similarly due to the same reason. Which is it?
Supporting evidence: The claim is empirical and the supporting evidence is experimental. As such, I find the comprehensiveness of the experiments wanting. There are several ways the experiments could be improved. 
-- Results for each \beta value should be included, to see how placing increasing significance on the auxiliary loss impacts the results.
-- Include all relevant details necessary to reproduce the results, such as the length of training or stopping criterion used. 
-- Additional results when varying the number of ticks.
-- More results with deeper hierarchies, since the ability to capture salient information at different levels of coarseness is a key selling point of HRNNs. 
-- Results on larger scale tasks besides character level language modelling on Penn TreeBank.
Other comments:
-- In the intro, I think some mention of parallel architectures such as transformers or convolutional architectures is warranted here, since parallelizability of training is a significant reason why these architectures are becoming preferred over RNNs.
-- Citations are mishandled throughout the paper. Citations should be enclosed in parentheses unless used as a subject in the sentence (e.g. "Sordoni et al. make the case that..."). There is no need to refer to a citation twice in a sentence, like you do in "More recently, Koutnik et al. introduced the Clockwork RNN Koutnik et al. (2014)..."
-- I don't understand why the permuted accuracy of the gr-HRNN is so much higher than the non-permuted accuracy. One possible explanation is that the important pixels ended up at the end in each of the three trials, hence the gr-HRNN did not have to remember much information from the past. This should be addressed in the paper. 
-- I would welcome some theoretical analysis as to why replacing the gradient path with this particular auxiliary loss does not impact results. I also think some discussion of what this means HRNNs are actually doing might be nice as well.