This paper proposes METAGROSS (Meta Gated Recursive Controller), a new neural sequence modeling unit. Our proposed unit is characterized by recursive parameterization of its gating functions, i.e., gating mechanisms of METAGROSS are controlled by instances of itself, which are repeatedly called in a recursive fashion. This can be interpreted as a form of meta-gating and recursively parameterizing a recurrent model. We postulate that our proposed inductive bias provides modeling benefits pertaining to learning with inherently hierarchically-structured sequence data (e.g., language, logical or music tasks). To this end, we conduct extensive experiments on recursive logic tasks (sorting, tree traversal, logical inference), sequential pixel-by-pixel classification, semantic parsing, code generation, machine translation and polyphonic music modeling, demonstrating the widespread utility of the proposed approach, i.e., achieving state-of-the-art (or close) performance on all tasks.
This paper proposes a neural sequence modelling unit called METAGROSS.  In principle, the aim of this unit is to introduce recursive parametrization of  gating functions, building on the gated RNN paradigm.  The authors motivate this work by arguing that while gated-RNNs tackle vanishing gradient problems and facilitate learning long-range dependencies in sequences, improvements can be made with respect to learning on hierarchically-structured data.  The authors propose a method to do so by also learning the depth of the parametrization, and claim that the inductive bias that emerges from this configuration is beneficial to learning such tasks.
I think the idea behind this work is sensible: introducing a meta-controller with recursive parametrization of gating functions for hierarchical tasks is sensible.  Another also strong point of this paper is that several experiments under different settings are presented, along with ablation studies and some model exploration.  The authors also show that integrating a non-autoregressive variant of the proposed meta-controller into the Transformer architecture can also be beneficial.  Results in general do show improvement over compared architectures.
On the negative side and besides empirical/experimental evidence, the paper would be much more convinving with some more insight into the model itself and some more qualitative evidence.  Figure 1 shows the architecture of the proposed method with a max depth of 3 indicating soft recursion with grayscale levels.  However, this figure is not referenced in the text and not explained further.  The non-autoregressive version (sec 2.3) simply does away with the dependence on hidden states and applies the proposed architecture directly on the input.  I think it would be very beneficial to see a simple toy example where the benefits of utilizing this meta-architecture can be qualitatively explained.  Finally, one can argue that this work is incremental, in the sense that it is a (relatively straightfwd) combination of meta-controllers with recursive architectures.  Differences and variations with respect to other methods in literature should be more clearly explained.
some more questions
- although most experiments show some resilience with respect to the varying max depth parameter, have the authors noticed any problems and limitations arising from setting the max-depth to be very high?  for example, in Table 1 it seems that accuracy may drop when increasing max length.  I think that more discussions and ellaboration on this could be useful, as the authors also propose a way of learning the depth parameters and also state that this is task dependent.  
- Figures 7-8-9 show variations of dynamic recursion on three databases.  Although these figures do show variation in learning (ranging from high activation fluctuations to static), it would be interesting to examine why this fluctuations occur in CIFAR  
- which brings me to the second question:  although the results do not show this, could this task-dependent nature of the controller lead to more chances of overfitting on a given training set that may be noisy?
- Although the authors perform ablation tests with multiple units and evaluate for max depth, we don't see many experiments with depth of more than 2 or 3 (besides table 1 - in many experiments the max depth is not mentioned).  Are the conclusions the same with all experiments wrt depth?
- It would be useful to have a comment on model complexity