Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential. Whereas prior approaches, largely unsupervised in nature, focus on sampling useful frames and assembling them as summaries, we consider video summarization as a supervised subset selection problem. Our idea is to teach the system to learn from human-created summaries how to select informative and diverse subsets, so as to best meet evaluation metrics derived from human-perceived quality. To this end, we propose the sequential determinantal point process (seqDPP), a probabilistic model for diverse sequential subset selection. Our novel seqDPP heeds the inherent sequential structures in video data, thus overcoming the deficiency of the standard DPP, which treats video frames as randomly permutable items. Meanwhile, seqDPP retains the power of modeling diverse subsets, essential for summarization. Our extensive results of summarizing videos from 3 datasets demonstrate the superior performance of our method, compared to not only existing unsupervised methods but also naive applications of the standard DPP model.
This paper presents a framework for supervised video summarization. The proposed method includes a new Sequential Determinantal Point Process (seqDPP) model, which learns to select the best subsets of the raw input data as components of the output summarization. Unlike the most common approaches, that focus on unsupervised summarization by selecting most important/representative frames and features, this paper attempts to directly learn from human-created summaries.The technical formulation of the method seems sound and correct. The new model seqDPP extends DPP by incorporating sequential dependencies. The model, learning and inference formulations are updated to account for the extension.The performance evaluation is adequate, including quantitative results (Precision, Recall and F-Score) from 3 datasets (youtube, OVP, kodak consumer). The results are good and mostly outperform the related work.The readability of the document is good. However the order of the sections could be improved. For instance, the Related Work could be presented earlier to help understand the context of this paper.In terms of originality, the seqDPP model is an extension of DPP which is used to summarize text data. The extension is sound to tackle the problem of video summarization. Also, the authors use a novel way to derive ground truth, by combining multiple human-created summaries into a unique ‘oracle’ summary, which is used for learning and evaluation.