Published as a conference paper at ICLR 2018 COMPOSITIONAL ATTENTION NETWORKS FOR MACHINE REASONING
We present Compositional Attention Networks, a novel fully differentiable neural network architecture, designed to facilitate explicit and expressive reasoning. While many types of neural networks are effective at learning and generalizing from massive quantities of data, this model moves away from monolithic blackbox architectures towards a design that provides a strong prior for iterative reasoning, allowing it to support explainable and structured learning, as well as generalization from a modest amount of data. The model builds on the great success of existing recurrent cells such as LSTMs: It sequences a single recurrent Memory, Attention, and Control (MAC) cell, and by careful design imposes structural constraints on the operation of each cell and the interactions between them, incorporating explicit control and soft attention mechanisms into their interfaces. We demonstrate the model's strength and robustness on the challenging CLEVR dataset for visual reasoning, achieving a new state-of-the-art 98.9% accuracy, halving the error rate of the previous best model. More importantly, we show that the new model is more computationally efficient and data-efficient, requiring an order of magnitude less time and/or data to achieve good results.
This paper describes a new model architecture for machine reasoning. In contrast
to previous approaches that explicitly predict a question-specific module
network layout, the current paper introduces a monolithic feedforward network
with iterated rounds of attention and memory. On a few variants of the CLEVR
dataset, it outperforms both discrete modular approaches, existing iterated
attention models, and the conditional-normalization-based FiLM model. 
So many models are close to perfect accuracy on the standard CLEVR dataset that
I'm not sure how interesting these results are. In this respect I think the
current paper's results on CLEVR-Humans and smaller fractions of synthetic CLEVR
are much more exciting.
On the whole I think this is a strong paper. I have two main concerns. The
largest is that this paper offers very little in the way of analysis. The model
is structurally quite similar to a stacked attention network or a particular
fixed arrangement of attentive N2NMN modules, and it's not at all clear based on
the limited set of experimental results where the improvements are actually
coming from. It's also possible that many of the proposed changes are
complementary to NMN- or CBN-type models, and it would be nice to know if this
is the case.
Secondarily, the paper asserts that "our architecture can handle
datasets more diverse than CLEVR", but runs no experiments to validate this. It
seems like once all the pieces are in place it should be very easy to get
numbers on VQA or even a more interesting synthetic dataset like NLVR.
Based on a sibling comment, it seems that there may also be some problems with
the comparison to FiLM, and I would like to see this addressed.
On the whole, the results are probably strong enough on their own to justify
admitting this paper. But I will become much more enthusiastic about if if the
authors can provide results on other datasets (even if they're not
state-of-the-art!) as well as evidence for the following:
1. Does the control mechanism attend to reasonable parts of the sentence?
Here it's probably enough to generate a bunch of examples showing sentence
attentions evolving over time.
2. Do these induce reasonable attentions over regions of the image?
Again, examples are fine.
3. Do the self-attention and gating mechanisms recover the right structure?
In addition to examples, here I think there are some useful qualitative
measures. It should be possible to extract reasonable discretized "reasoning
maps" by running MST or just thesholding on the "edge weights" induced by
attention and gating. Having extracted these from a bunch of examples, you can
compare them to the structural properties of the ground-truth CLEVR network
layouts by plotting a comparison of sizes, branching factors, etc.
4. More on the left side of the dataset size / accuracy curve. What happens if
   you only give the model 7000 examples? 700? 70?
Fussy typographical notes:
- This paper makes use of a lot of multi-letter names in mathmode. These are
  currently written like $KB$, which looks bad, and should instead be
  $\mathit{KB}$.
- Variables with both superscripts and subscripts have the superscripts pushed
  off to the right; I think you're writing these like $b_5 ^d$ but they should
  just be $b_5^d$ (no space).
- Number equations and then don't bother carrying subscripts like $W_3$, $W_4$
  around across different parts of the model---this isn't helpful.
- The superscripts indicating the dimensions of parameter matrices and vectors
  are quite helpful, but don't seem to be explained anywhere in the text. I
  think the notation $W^{(d \times d)}$ is more standard than $W^{d, d}$.
- Put the cell diagrams right next to the body text that describes them (maybe even
  inline, rather than in figures). It's annoying to flip back and forth.