Action from Still Image Dataset and Inverse Optimal Control to Learn Task Specific Visual Scanpaths
Human eye movements provide a rich source of information into the human visual information processing. The complex interplay between the task and the visual stimulus is believed to determine human eye movements, yet it is not fully understood, making it difficult to develop reliable eye movement prediction systems. Our work makes three contributions towards addressing this problem. First, we complement one of the largest and most challenging static computer vision datasets, VOC 2012 Actions, with human eye movement recordings collected under the primary task constraint of action recognition, as well as, separately, for context recognition, in order to analyze the impact of different tasks. Our dataset is unique among the eyetracking datasets of still images in terms of large scale (over 1 million fixations recorded in 9157 images) and different task controls. Second, we propose Markov models to automatically discover areas of interest (AOI) and introduce novel sequential consistency metrics based on them. Our methods can automatically determine the number, the spatial support and the transitions between AOIs, in addition to their locations. Based on such encodings, we quantitatively show that given unconstrained read-world stimuli, task instructions have significant influence on the human visual search patterns and are stable across subjects. Finally, we leverage powerful machine learning techniques and computer vision features in order to learn task-sensitive reward functions from eye movement data within models that allow to effectively predict the human visual search patterns based on inverse optimal control. The methodology achieves state of the art scanpath modeling results.
This paper presents work on human eye movements when 
      viewing still images. A new dataset is collected, measuring eye fixations 
      under task-specific instructions -- the same images are viewed by subjects 
      instructed to determine the human action or the scene context. The paper 
      presents analyses of these data, a model for scanpaths, a model for 
      fixations, and a temporal model of saccades. The analyses measure 
      consistency of different subjects' viewing of an image-task pair. The 
      model for scanpaths builds a task-image pair-specific HMM. The model for 
      fixations uses HOG descriptors and an SVM to predict salient regions of an 
      image. The temporal model uses reinforcement learning to generate a model 
      for viewing of an image. Overall, the paper makes contributions to 
      eye movement analysis -- expanding on previous work [16] to clearly show 
      task matters for gaze. In my view this is the strongest part of the paper. 
      On the other hand, various computational models are described, the HMM for 
      scanpaths, the HOG-SVM saliency predictor, and the reinforcement learning. 
      These are reasonable, but there are concerns about their 
      applicability/importance (details below). On the whole, the paper is 
      solid, and could be accepted as a poster. 1) The HMM for scanpaths 
      seems quite limited in its use. The model is a task-image pair-specific 
      HMM -- given an image for which many subjects' eye movements have been 
      observed given a task, an HMM that predicts how other people will view 
      this image when asked to do this task is learned. While HMMs are a 
      reasonable model, this task doesn't seem very common. Further, it seems 
      any temporal model would be appropriate for this task, and no particular 
      baselines are presented. 2) A HOG-SVM saliency predictor is 
      defined. This uses dataset and task-specific training data to build a 
      model of where a subject will look. The performance is good, and better 
      than baselines. However, it seems only marginally better than [12], and 
      this could either be due to the use of different features or a non-linear 
      classifier. It is necessary to clarify from which of these the performance 
      gain comes. 3) A similar comment applies to the reinforcement 
      learning and comparison to baselines. Different features (HOG) and 
      learning strategy are used in this method. While overall performance is 
      important, understanding where these gains come from is important too. 
      Minor comments: - The paper needs proofreading, there 
      are many typos and spelling errors. - Why does the dataset have 3s 
      exposures for action recognition and 2.5 for context? Won't this 
      difference bias comparative studies? - What are the "three key 
      presses" (line 135)? From the description, it seems only one key press is 
      made by a subject. - Line 200 "locating people"? - +/- 
      std. deviation should be added to numbers in Table 1. - I don't 
      think the cross-stimulus baseline for gaze prediction is defined in the 
      text. A sentence could be added, especially pointing out that the 
      performance is significantly lower, which bolsters a main message of the 
      paper.