Learning what and where to attend
Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs). Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels. Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition. We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps. Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization. As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers.
 The paper presents a new take on attention in which a large attention dataset is collected (crowdsourced) and used to train a NN (with a new module) in a supervised manner to exploit self-reported human attention. The empirical results demonstrate the advantages of this approach. *Pro*: - Well-written and relatively easily accessible paper (even for a non-expert in attention like myself) - Well-designed crowdsourcing experiment leading to a novel dataset (which is linked to state-of-the-art benchmark) - An empirical study demonstrates a clear advantage of using human (attention) supervision in a relevant comparison *Cons* - Some notational confusion/uncertainty in sec 3.1 and Fig 3 (perhaps also Sec 4.1): E.g. $.mathbf{M} and {L_clickmaps} are undefined in Sec 3.1. *Significance:* I believe this work would be of general interest to the image community at ICLR as it provides a new high-quality dataset and an attention module for grounding investigations into attention mechanisms for DNNs (and beyond). *Further comments/questions:* - The transition between sec 2 and sec 3 seems abrupt; consider providing a smoother transition. - Figure 3: reconsider the logical flow in the figure; it took me a while to figure out what going on (especially the feedback path to U’). - It would be beneficial to provide some more insight into the statistical tests casually reported (i.e., where did the p values come from) - The dataset appears to be available online but will the code for the GALA module also be published?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning what and where to attend
Most recent gains in visual recognition have originated from the inclusion of attention mechanisms in deep convolutional networks (DCNs). Because these networks are optimized for object recognition, they learn where to attend using only a weak form of supervision derived from image class labels. Here, we demonstrate the benefit of using stronger supervisory signals by teaching DCNs to attend to image regions that humans deem important for object recognition. We first describe a large-scale online experiment (ClickMe) used to supplement ImageNet with nearly half a million human-derived "top-down" attention maps. Using human psychophysics, we confirm that the identified top-down features from ClickMe are more diagnostic than "bottom-up" saliency features for rapid image categorization. As a proof of concept, we extend a state-of-the-art attention network and demonstrate that adding ClickMe supervision significantly improves its accuracy and yields visual features that are more interpretable and more similar to those used by human observers.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper presents a new take on attention in which a large attention dataset is collected (crowdsourced) and used to train a NN (with a new module) in a supervised manner to exploit self-reported human attention. The empirical results demonstrate the advantages of this approach. *Pro*: - Well-written and relatively easily accessible paper (even for a non-expert in attention like myself) - Well-designed crowdsourcing experiment leading to a novel dataset (which is linked to state-of-the-art benchmark) - An empirical study demonstrates a clear advantage of using human (attention) supervision in a relevant comparison *Cons* - Some notational confusion/uncertainty in sec 3.1 and Fig 3 (perhaps also Sec 4.1): E.g. $.mathbf{M} and {L_clickmaps} are undefined in Sec 3.1. *Significance:* I believe this work would be of general interest to the image community at ICLR as it provides a new high-quality dataset and an attention module for grounding investigations into attention mechanisms for DNNs (and beyond). *Further comments/questions:* - The transition between sec 2 and sec 3 seems abrupt; consider providing a smoother transition. - Figure 3: reconsider the logical flow in the figure; it took me a while to figure out what going on (especially the feedback path to U’). - It would be beneficial to provide some more insight into the statistical tests casually reported (i.e., where did the p values come from) - The dataset appears to be available online but will the code for the GALA module also be published?
assistant
 The paper presents a new take on attention in which a large attention dataset is collected (crowdsourced) and used to train a NN (with a new module) in a supervised manner to exploit self-reported human attention. The empirical results demonstrate the advantages of this approach. *Pro*: - Well-written and relatively easily accessible paper (even for a non-expert in attention like myself) - Well-designed crowdsourcing experiment leading to a novel dataset (which is linked to state-of-the-art benchmark) - An empirical study demonstrates a clear advantage of using human (attention) supervision in a relevant comparison *Cons* - Some notational confusion/uncertainty in sec 3.1 and Fig 3 (perhaps also Sec 4.1): E.g. $.mathbf{M} and {L_clickmaps} are undefined in Sec 3.1. *Significance:* I believe this work would be of general interest to the image community at ICLR as it provides a new high-quality dataset and an attention module for grounding investigations into attention mechanisms for DNNs (and beyond). *Further comments/questions:* - The transition between sec 2 and sec 3 seems abrupt; consider providing a smoother transition. - Figure 3: reconsider the logical flow in the figure; it took me a while to figure out what going on (especially the feedback path to U’). - It would be beneficial to provide some more insight into the statistical tests casually reported (i.e., where did the p values come from) - The dataset appears to be available online but will the code for the GALA module also be published?