Published as a conference paper at ICLR 2018 LEARNING TO COUNT OBJECTS IN NATURAL IMAGES FOR VISUAL QUESTION ANSWERING
Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-theart accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.
Summary: 
- This paper proposes a hand-designed network architecture on a graph of object proposals to perform soft non-maximum suppression to get object count.
Contribution:
- This paper proposes a new object counting module which operates on a graph of object proposals.
Clarity:
- The paper is well written and clarity is good. Figure 2 & 3 helps the readers understand the core algorithm.
Pros:
- De-duplication modules of inter and intra object edges are interesting.
- The proposed method improves the baseline by 5% on counting questions.
Cons:
- The proposed model is pretty hand-crafted. I would recommend the authors to use something more general, like graph convolutional neural networks (Kipf & Welling, 2017) or graph gated neural networks (Li et al., 2016).
- One major bottleneck of the model is that the proposals are not jointly finetuned. So if the proposals are missing a single object, this cannot really be counted. In short, if the proposals don’t have 100% recall, then the model is then trained with a biased loss function which asks it to count all the objects even if some are already missing from the proposals. The paper didn’t study what is the recall of the proposals and how sensitive the threshold is.
- The paper doesn’t study a simple baseline that just does NMS on the proposal domain.
- The paper doesn’t compare experiment numbers with (Chattopadhyay et al., 2017).
- The proposed algorithm doesn’t handle symmetry breaking when two edges are equally confident (in 4.2.2 it basically scales down both edges). This is similar to a density map approach and the problem is that the model doesn’t develop a notion of instance.
- Compared to (Zhou et al., 2017), the proposed model does not improve much on the counting questions.
- Since the authors have mentioned in the related work, it would also be more convincing if they show experimental results on CL
Conclusion:
- I feel that the motivation is good, but the proposed model is too hand-crafted. Also, key experiments are missing: 1) NMS baseline 2) Comparison with VQA counting work  (Chattopadhyay et al., 2017). Therefore I recommend reject.
References:
- Kipf, T.N., Welling, M., Semi-Supervised Classification with Graph Convolutional Networks. ICLR 2017.
- Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R. Gated Graph Sequence Neural Networks. ICLR 2016.
Update:
Thank you for the rebuttal. The paper is revised and I saw NMS baseline is added. I understood the reason not to compare with certain related work. The rebuttal is convincing and I decided to increase my rating, because adding the proposed counting module achieve 5% increase in counting accuracy. However, I am a little worried that the proposed model may be hard to reproduce due to its complexity and therefore choose to give a 6.