Action is in the Eye of the Beholder: Eye-gaze Driven Model for Spatio-Temporal Action Localization
We propose a weakly-supervised structured learning approach for recognition and spatio-temporal localization of actions in video. As part of the proposed approach, we develop a generalization of the Max-Path search algorithm which allows us to efficiently search over a structured space of multiple spatio-temporal paths while also incorporating context information into the model. Instead of using spatial annotations in the form of bounding boxes to guide the latent model during training, we utilize human gaze data in the form of a weak supervisory signal. This is achieved by incorporating eye gaze, along with the classification, into the structured loss within the latent SVM learning framework. Experiments on a challenging benchmark dataset, UCF-Sports, show that our model is more accurate, in terms of classification, and achieves state-of-the-art results in localization. In addition, our model can produce top-down saliency maps conditioned on the classification label and localized latent paths.
This paper proposes the use of eye-gaze data for 
      spatio-temporal action localization in videos, showing improved action 
      classification accuracy and state-of-the-art action localization in 
      videos. The method of incorporating this information is a latent SVM 
      framework with modified constraints. The authors conduct various 
      experiments to show that their method performs better than existing 
      methods for localization and using eye gaze data could serve as a better 
      prior than bounding boxes as done by previous works. Overall, I 
      found the paper to be well written but the experimental results a little 
      lacking. The concerns are explained below. Pros: - Interesting 
      problem formulation of using eye-gaze data instead of bounding box data 
      for learning - Experimental results are thorough and various problem 
      settings are evaluated such as action classification, localization and 
      gaze prediction. The experimental results suggest that the method proposed 
      in the paper outperforms existing methods on the given tasks. 
      Cons: - The cost and difficulty of collecting eye-gaze data is 
      not well justified by the observed improvements. While eye-gaze data may 
      be more natural in some ways, it is significantly harder to collect as 
      compared to bounding box annotations which can be easily collected via 
      crowdsourcing as no specialized equipment is required. This makes it 
      significantly harder to scale this approach to other datasets. - How 
      important is it to model gaze in a category specific manner? If gaze could 
      be used for general saliency prediction, then the same method could be 
      applied to other datasets more easily. - It is not clear why the 
      authors do not compare their results to other existing methods for action 
      classification such as [Z1] and [Z2] (mentioned below). The performance 
      achieved by them is 85.6% and 86.5% as compared to the reported 
      performance of 82.1%. It should be noted that [Z1] and [Z2] do not report 
      results on action localization but the same can be said for [16] and [12] 
      which are reported here. While [Z1] and [Z2] use chi-square kernels for 
      learning, it is an advantage of their methods and it is not clear if the 
      same set of kernels can be applied in the formulation presented here. 
      - What is the corresponding localization result of the proposed method 
      on the 3 categories used by Tran and Yuan[18]? It would be nice to include 
      this in the figure caption. - There are no baselines provided for gaze 
      localization and prediction. It is possible to use some simple methods 
      such as exemplar-based detectors[Z3] for gaze prediction to show the 
      necessity/strength of your method. References (I use Z to be 
      distinct from paper references) [Z1] Evaluation of local 
      spatio-temporal features for action recognition. In BMVC, 2010. [Z2] 
      Learning hierarchical invariant spatio-temporal features for action 
      recognition with independent subspace analysis. In CVPR, 2011. 
      [Z3] Ensemble of Exemplar-SVMs for Object Detection and Beyond. In 
      ICCV, 2011. The paper is generally well-written but here are some 
      minor language errors: - L51: this comes at expense --> add "the" 
      - L69-70: miss --> mis - L82: proved --> have proven - 
      L134: extend --> extended