On Multiplicative Integration with Recurrent Neural Networks
We introduce a general and simple structural design called "Multiplicative Integration" (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.
In this paper the authors advocate for the use of multiplicative interactions in RNNs. The sum between the projected hidden state and the input embedding is replaced by a component wise product. They show results on a large variety of tasks with small but consistent improvements. They also claim faster convergence.
My biggest concern about this work is the lack of novelty. Despite the claimed differences, the proposed method is a special case of what proposed in [10]. I doubt that the slight different parameterization (remove one factor-hidden matrix and introduce more bias terms) makes much difference. I strongly suspect that the improved performance is due to better optimization (HF has proven to be very brittle).  I also found weak the argument for which gating makes gradients flow better because there is no guarantee this is going to happen. The gradient has an additional term it's multiplied by, There is no guarantee that the Jacobian between two consecutive hidden states has unitary eigenvalues...  The experimental section is solid, although improvements are rather marginal yet consistent.  -------------------------------------------------- I have read the rebuttal. I sincerely apologize with the authors. There is no fatal flaw in their paper, I just clicked that by mistake. Apologies.  I still think that the paper lacks novelty, however.  [10] proposes multiplicative interactions for a plain RNN but there is no reason why the same would not apply to other types of RNNs like LSTMs. Multiplicative interactions apply to anything really (true for [10] and this paper). In [10] they had: h_t = non_linearity( V ((W_x x) .* (W_h h_{t-1})) ) here the authors propose: h_t = non_linearity( \alpha .* ((W_x x) .* (W_h h_{t-1})) + bias_terms) I do understand that this is a simplification of [10], but is it so different? The difference is removing matrix V above and introducing bias terms. If they authors acknowledged more and analyzed the effect of each of these modifications, it would have been more informative in my opinion.  That said, this paper has pretty solid empirical validation which compensates the lack of novelty. Based on this, I'd certainly recommend acceptance, but I would be quite hesitant about promoting this to an oral.