Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur. Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language. In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output. This mechanism affords the modelling of a richer space of interactions between inputs and their context. Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent. Experiments demonstrate markedly improved generalization on language modelling in the range of 3-4 perplexity points on Penn Treebank and Wikitext-2, and 0.01-0.05 bpc on four character-based datasets. We establish a new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.
I have read the authors' response. Their points regarding baseline comparisons are sensible in that there isn't a reason to expect the observations to *not* generalization to other datasets. It is odd that mLSTM is outperformed by LSTM in Table 3, but as the authors note in section 4.2 this may be due to instability of mLSTM during training. The results in the paper demonstrate significant improvement over LSTM, and while there are not as many baseline comparison to similar models as I would have liked to see, the quality of this work is sufficiently high that this is not a fatal flaw. In light of the author response and other reviews, I am revising my rating to 6: Weak Accept.
=====
This paper proposes a modification of LSTM networks in the context of language modeling called Mogrifier LSTM. Ordinary LSTMs are defined as recurrent operations on the current input, previous hidden state, and previous cell state. The proposed Mogrifier LSTM utilizes the same recurrent unit as the LSTM, but the input and previous hidden state are updated with several rounds of mutual gating. In each round, the input  is multiplied elementwise by a gate computed as a function of the hidden state (or vice versa). The authors experiment on word-level and character-level modeling and compare their Mogrifier LSTMs to several state-of-the-art approaches. They also conduct an ablation study to show the effect of various design choices and hyperparameters and experiments on a reverse copy task.
Specific contributions include:
* Proposal of a novel approach for modulating inputs to a recurrent unit by mutual gating.
* Experiments demonstrating strong performance on a number of language modeling tasks.
  
The paper in its current state is borderline, leaning towards weak reject. Points in favor of acceptance include the high clarity of writing, good experiments of the proposed model, and a discussion of possible reasons for why the mogrification operation works well. The main shortcoming of the paper is experimental comparison to baselines.
The authors were able to train baseline LSTMs to high levels of performance (presumably due to tuning of hyperparameters) and then demonstrate that Mogrifier LSTMs improve upon LSTMs significantly. This is perhaps not entirely surprising, because the hyperparameter range of the Mogrifier LSTM includes zero rounds of updates, which would render it identical to the baseline LSTM. Therefore, if the hyperparameters are tuned sufficiently well, the performance of the Mogrifier LSTM should be at least as good as the LSTM. What the experiments do not show is that the proposed mogrification outperforms other forms of multiplicative interaction and/or gating. The closest that the authors come to this is the single validation perplexity of the Multiplicative LSTM in Table 3. If thorough hyperparameter tuning is applied to the Multiplicative LSTM or the approaches of Wu et al. (2016) and/or Sutskever et al. (2011), does the Mogrifier LSTM still outperform them?
Other than this critical issue of baseline comparison, the experiments are quite informative. The ablation study showing the effect of different design decisions and the hyperparameter visualiztion in Appendix B are particularly useful. The mogrification operation is described precisely enough for other researchers to implement and the arguments made in 4.4 are compelling.
Question for the authors:
* Some qualitative analysis of the learned mogrification operation would be helpful for understanding the nature of the modulation. For example, how do the predictions change depending on the modulation? If x is modulated by different hidden states, is there a noticeable effect on the output?
* Did you experiment with other forms of modulation before arriving upon the mogrification formulation? There are some naive approaches such as concatenating the hidden state to the input and applying a nonlinear layer, or predicting affine parameters for the input as a function of the hidden state in the style of FiLM [1]. Are there obvious shortcomings in these naive approaches that mogrification handles gracefully?
[1] Perez, E., Strub, F., De Vries, H., Dumoulin, V. and Courville, A., 2018, April. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence.