Learning Mixtures of Submodular Functions for Image Collection Summarization
We address the problem of image collection summarization by learning mixtures of submodular functions. Submodularity is useful for this problem since it naturally represents characteristics such as fidelity and diversity, desirable for any summary. Several previously proposed image summarization scoring methodologies, in fact, instinctively arrived at submodularity. We provide classes of submodular component functions (including some which are instantiated via a deep neural network) over which mixtures may be learnt. We formulate the learning of such mixtures as a supervised problem via large-margin structured prediction. As a loss function, and for automatic summary scoring, we introduce a novel summary evaluation method called V-ROUGE, and test both submodular and non-submodular optimization (using the submodular-supermodular procedure) to learn a mixture of submodular functions. Interestingly, using non-submodular optimization to learn submodular functions provides the best results. We also provide a new data set consisting of 14 real-world image collections along with many human-generated ground truth summaries collected using Amazon Mechanical Turk. We compare our method with previous work on this problem and show that our learning approach outperforms all competitors on this new data set. This paper provides, to our knowledge, the first systematic approach for quantifying the problem of image collection summarization, along with a new data set of image collections and human summaries.
This paper formulates the task of summarizing personal photo collections as submodular optimization, and proposes a max-margin formulation with multiple choices for loss functions to learn weights to these. A dataset of image collections with several human-generated summaries is presented, together with a scoring method, using which the proposed approach is evaluated. Experiments show that on this dataset, the method has a very good performance, even outperforming human labelers on average.QualityThis paper is a through investigation of the possible application of submodularity for image summarization. Extensive experiments were conducted to compare different optimization methods (suitable for different choices of loss functions). The authors also provide an ample amount of supplementary material, giving even more details.ClarityThe paper is well organized, the explanations are clear and also well-written in terms of English usage.A few remarks:- on line 144, does the term "compatible" have a special meaning, justifying the italics? (also, similar was the term "calibrate" on line 155.)- the relation between the components in section 6 of the supplementary material and the summary in section 5 of the paper could be clarified (or at least it could be mentioned that further information can be found in the supplementary material).OriginalityA major contribution of the paper is posing image set summarization as a submodular optimization problem; to the knowledge of this reviewer, this is a novel view of the problem. Together with a new dataset and the adaptation of ROUGE to a significantly different application domain, this paper has several novel contributions to the state of the art.SignificanceAccording to this reviewer, this work fits well in the topics of interest for NIPS, to which it makes a significant contribution.