  Video summarization aims to select the most informative subset of frames in a
video to facilitate efficient video browsing. Unsupervised methods usually rely
on heuristic training objectives such as diversity and representativeness.
However, such methods need to bootstrap the online-generated summaries to
compute the objectives for importance score regression. We consider such a
pipeline inefficient and seek to directly quantify the frame-level importance
with the help of contrastive losses in the representation learning literature.
Leveraging the contrastive losses, we propose three metrics featuring a
desirable key frame: local dissimilarity, global consistency, and uniqueness.
With features pre-trained on the image classification task, the metrics can
already yield high-quality importance scores, demonstrating competitive or
better performance than past heavily-trained methods. We show that by refining
the pre-trained features with a lightweight contrastively learned projection
module, the frame-level importance scores can be further improved, and the
model can also leverage a large number of random videos and generalize to test
videos with decent performance. Code available at
https://github.com/pangzss/pytorch-CTVSUM.

Computer Vision and Pattern Recognition