UBiSS: A Unified Framework for Bimodal Semantic Summarization of Videos
<|reference_start|>UBiSS: A Unified Framework for Bimodal Semantic Summarization of Videos: With the surge in the amount of video data, video summarization techniques, including visual-modal(VM) and textual-modal(TM) summarization, are attracting more and more attention. However, unimodal summarization inevitably loses the rich semantics of the video. In this paper, we focus on a more comprehensive video summarization task named Bimodal Semantic Summarization of Videos (BiSSV). Specifically, we first construct a large-scale dataset, BIDS, in (video, VM-Summary, TM-Summary) triplet format. Unlike traditional processing methods, our construction procedure contains a VM-Summary extraction algorithm aiming to preserve the most salient content within long videos. Based on BIDS, we propose a Unified framework UBiSS for the BiSSV task, which models the saliency information in the video and generates a TM-summary and VM-summary simultaneously. We further optimize our model with a list-wise ranking-based objective to improve its capacity to capture highlights. Lastly, we propose a metric, $NDCG_{MS}$, to provide a joint evaluation of the bimodal summary. Experiments show that our unified framework achieves better performance than multi-stage summarization pipelines. Code and data are available at https://github.com/MeiYutingg/UBiSS.<|reference_end|>
@article{mei2024ubiss:,
    title={UBiSS: A Unified Framework for Bimodal Semantic Summarization of Videos},
    author={Yuting Mei, Linli Yao, Qin Jin},
    journal={Proceedings of the 2024 International Conference on Multimedia
  Retrieval, May 2024, Pages 1034-1042},
    year={2024},
    doi={10.1145/3652583.3658038},
    archivePrefix={arXiv},
    eprint={2406.16301},
    primaryClass={cs.CV cs.AI cs.MM}
}