A flexible model for training action localization with varying levels of supervision
Spatio-temporal action detection in videos is typically addressed in a fullysupervised setup with manual annotation of training videos required at every frame. Since such annotation is extremely tedious and prohibits scalability, there is a clear need to minimize the amount of manual supervision. In this work we propose a unifying framework that can handle and combine varying types of less-demanding weak supervision. Our model is based on discriminative clustering and integrates different types of supervision as constraints on the optimization. We investigate applications of such a model to training setups with alternative supervisory signals ranging from video-level class labels to the full per-frame annotation of action bounding boxes. Experiments on the challenging UCF101-24 and DALY datasets demonstrate competitive performance of our method at a fraction of supervision used by previous methods. The flexibility of our model enables joint learning from data with different levels of annotation. Experimental results demonstrate a significant gain by adding a few fully supervised examples to otherwise weakly labeled videos.
Paper Summary: The paper describes a method for spatio-temporal human action localization in temporally untrimmed videos based on discriminative clustering [3, 47]. The main contribution of this paper is a new action detection approach which is flexible in the sense that it can be trained with various levels and amounts of supervision. For example, the model can be trained with very weak level of supervision, i.e., train the model for action detection only using ground truth video-level action labels; and also it can be trained with full supervision i.e. with dense per frame bounding box and their class labels.  Experimental results demonstrate the strengths and weaknesses for a wide range of supervisory signals such as, video level action labels, single temporal point, one GT bounding box, temporal bounds etc. The method is experimentally evaluated on the UCF-101-24 and DALY action detection datasets.  Paper Strengths: - The paper is clear and easy to understand. - The problem formulation is interesting and described with enough details. - The experimental results are interesting and promising, clearly demonstrate the significance of varying level of supervision on the detection performance - Table 1. - On DALY dataset, as expected, the detection performance increases with access to more supervision. - The proposed approach outperforms the SOA [46] by a large margin of 18%  (video mAP) on DALY dataset at all levels of supervision. - On UCF-101-24, the proposed approach outperforms the SOA [46] when bounding box annotations are available at any level, i.e., Temp.+1 BB, Temp. + 3 BBs, Fully Spervised (cf. Table 1).  - The visuals are helpful, support well the paper, and the qualitative experiments (in supplementary material) are interesting and convincing.   Paper Weaknesses: I haven't noticed any major weakness in this paper, however would like to mention that - on UCF-101-24, the proposed method has drop in performance as compared to the SOA [46]  when supervision level is "Temporal + spatial points".  This work addresses one of the major problems associated with action detection approaches based on fully supervised learning, i.e., these methods require dense frame level GT bounding box annotations and their labels, which is impractical for large scale video datasets and also highly expensive. The proposed unified action detection framework provides a way to train a ML model with weak supervision at various levels, contributing significantly to address the aforementioned problem. Thus, I vote for a clear accept.