Learned Region Sparsity and Diversity Also Predict Visual Attention
Learned region sparsity has achieved state-of-the-art performance in classification tasks by exploiting and integrating a sparse set of local information into global decisions. The underlying mechanism resembles how people sample information from an image with their eye movements when making similar decisions. In this paper we incorporate the biologically plausible mechanism of Inhibition of Return into the learned region sparsity model, thereby imposing diversity on the selected regions. We investigate how these mechanisms of sparsity and diversity relate to visual attention by testing our model on three different types of visual search tasks. We report state-of-the-art results in predicting the locations of human gaze fixations, even though our model is trained only on image-level labels without object location annotations. Notably, the classification performance of the extended model remains the same as the original. This work suggests a new computational perspective on visual attention mechanisms, and shows how the inclusion of attention-based mechanisms can improve computer vision techniques.
The paper proposes a Sparse Diverse Region Classifier (SDR), a method based on Region Ranking SVM (RRSVM) [29], which imposes diversity on the regions selected by RRSVM when training an image classifier. Diversity in the selected regions is introduced by a mechanism called "Inhibition of Return", which is in charge of only selecting regions with low spatial overlap. This overlap is computed using the traditional intersection over union criterion used in non-maximun suppresion (NMS).   Experiments are conducted on the POET[26], MIT900 [8], PET [13] datasets, using a RRSVM and SDR models trained on the Pascal VOC 2007 dataset. The evaluation covers different scenarios: a) Single-target is present, b) the target is absent, and c) when multiple-targets are present. The Judd variation of the Area Under Curve (AUC-Judd) is used as a performance metric, where the ability of the method to recover visual fixation points is measured. Experimental results show that the introduced Inhibition Return mechanism introduced via NMS, brings a boost in performance of ~4 performance. 
- The content of the paper is well presented and accompanied by clear illustrations which make the content of the paper easy to follow. Likewise, the experimental protocol followed in the paper is quite clear, which eases reproducibility of the results obtained in this work.  Having said this, I consider there are two major flaws in the paper:  - I appreciate the comparison wrt. to methods that use the combination of the output of object detectors to create the priority maps. However, in my opinion, despite the fact that R-CNN takes into account object annotations during training, it is not a strong baseline as is claimed in the manuscript due to the following reasons. First, R-CNN uses which has the weakness of using Selective Search (Uijlings et al., IJCV'13) which is a generic method for generating object proposals. R-CNN is not anymore the state-of-the-art, I recommend repeating the experiment with Faster-RCNN, Ren et al. CVPR'15 which addresses that weakness. Second, and more critical, R-CNN is trained to optimizing the object detection which is measured by the area under the curve where matching between predicted object regions and ground-truth is computed by taking into account object bounding boxes (not the fixation points). Since these are two different tasks, ie. object detection (localizing the full extent of the object)  VS. human visual attention (localizing the specific regions that define the object), in my opinion it is not surprising that R-CNN has sub-standard performance.  - Regarding the AnnoBoxes baselines, I agree that it is exploiting more information (annotated bounding boxes on test images). However, I believe that the way in which this information is used, reduces significantly the advantage that it can bring to the task of visual attention prediction. As stated in Sec. 4.2: "for this method, the priority map is created by applying a Gaussian filter to a binary map where the center of the bounding box over the target(s) is set to 1 and everywhere else 0", this implies that the points in the binary map with values==1 (of high priority) are mostly located at the center of the bounding box of each object, and, as can be observed in the figures showing animal-related classes, this is not the same as the fixation points. This is further confirmed in Table 1, where it can be noted that is in the animal-related classes (cat, cow, dog and horse) where the proposed SDR method has a significant advantage over AnnoBoxes (with boosts of 5, 4, 8 and 6 percentage points, respectively).   In my opinion, a comparison against the two methods listed above (R-CNN and AnnoBoxes) is not a strong empirical evidence that the proposed method is good at predicting visual attention (fixation points). Perhaps, from the results of the previous two comparisons, it is better to say that solely focusing on learning how to localize objects does not imply learning how to predict visual attention (fixation points). This could serve as a motivation for the diversity introduced by SDR via NMS.  I believe the paper can be strengthened by performing experiments to evaluate the effect that some of the parameters (e.g., overlap considered when doing NMS for SDR, width of the Gaussian blur kernel) have on performance. In addition, I suggest making more concrete the conclusions made during this work (At this point they were not so clear to me).   In addition, regarding to the comparisons wrt. to existing work, I suggest considering the method from Li et al. arXiv:1506.06343 which tries to detected visual patterns on image regions that can be informative for image classification. In my opinion this method has some similarities wrt. the method proposed in the paper.   In my opinion the paper is still at a premature stage for publication at this point .  Given the promising potential and links that the proposed method has in different fields, I encourage the authors to properly position their work wrt. existing work, revise the baselines considered during their evaluation, and, through an ablation study, provide an insight on the effect of different parameters that are part of the proposed method.