We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others. Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though this often not emphasized and thus under-appreciated. We begin by showing that such layers strictly enrich the representable function classes of neural networks. We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required. We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation. Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support of multiplicative interactions playing a more prominent role when designing new neural network architectures.
The paper explores different types of multiplicative interactions. This allows understanding of both efficacy of multiplicative interaction (i.e., MI vs MLP), and the common between multiplicative-type models (e.g., hypernetworks, FiLM, gating,  attention etc). The authors also find MI models able to achieve a state-of-the-art performance on language modeling (WikiText-103) and reinforcement learning problems(DMLAB-30), along with several toy examples. 
Strengths: 
*The discussed MI subject is important research area, the paper presents the vast related work, proven by the fact MI techniques appears in many recent models.
* The authors  did comprehensive evaluation to show importance and usefulness of multiplicative interactions. A  toy regression experiment to show superiority of MI vs MLP . A Reinforcement learning task, i.e DMLAB-30 on par with the state-of-the-art model, but with simple model (i.e., less parameters). A sequence prediction with an alternative embedding technique that improves both accuracy and number of needed parameters. 
* The paper is of high quality: well organized, clear, with good supporting figures.  
Weaknesses:
* I suggest a better explanation how the suggested models compare to state-of-the-art models.  Currently, it is hard to assess the impact. For instance, proposed model alternate existing baselines, such as PopArt, or is completely novel?  
* Although mentioned, it's not the focus of this work, the paper should have discussed attention models more.  Specifically recent years multimodal attention relied on multiplicative interactions. Therefore, at least in this domain multiplicative interactions are not "under-appreciated". Relevant papers: [1, 2, 3, 4] - advancement of multiplicative interactions over the years, [5] - introduce co-attention, [6] - introduce higher-order interactions (between three vectors), [7] - introduce bilinear attention .
To conclude, multiplicative interactions are extremely important, and I find the paper exploration useful. In addition,  the paper is of high quality, and with satisfied experiments to prove their claims. I do suggest a better discussion about multimodal attention networks, which are relevant examples.
[1] - Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering; Xu et al.
[2] - Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding; Fukui et al.
[3] - Hadamard Product for Low-rank Bilinear Pooling; Kim et al.
[4] - Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering; Yu et al.
[5] - Hierarchical Question-Image Co-Attention for Visual Question Answering; Lu et al.
[6] - High-Order Attention Models for Visual Question Answering; Schwartz et al.
[7]  - Bilinear Attention Networks; Kim et al.