TeaserGen: Generating Teasers for Long Documentaries
<|reference_start|>TeaserGen: Generating Teasers for Long Documentaries: Teasers are an effective tool for promoting content in entertainment, commercial and educational fields. However, creating an effective teaser for long videos is challenging for it requires long-range multimodal modeling on the input videos, while necessitating maintaining audiovisual alignments, managing scene changes and preserving factual accuracy for the output teasers. Due to the lack of a publicly-available dataset, progress along this research direction has been hindered. In this work, we present DocumentaryNet, a collection of 1,269 documentaries paired with their teasers, featuring multimodal data streams of video, speech, music, sound effects and narrations. With DocumentaryNet, we propose a new two-stage system for generating teasers from long documentaries. The proposed TeaserGen system first generates the teaser narration from the transcribed narration of the documentary using a pretrained large language model, and then selects the most relevant visual content to accompany the generated narration through language-vision models. For narration-video matching, we explore two approaches: a pretraining-based model using pretrained contrastive language-vision models and a deep sequential model that learns the mapping between the narrations and visuals. Our experimental results show that the pretraining-based approach is more effective at identifying relevant visual content than directly trained deep autoregressive models.<|reference_end|>
@article{xu2024teasergen:,
    title={TeaserGen: Generating Teasers for Long Documentaries},
    author={Weihan Xu, Paul Pu Liang, Haven Kim, Julian McAuley, Taylor
  Berg-Kirkpatrick, Hao-Wen Dong},
    journal={arXiv preprint arXiv:2410.05586},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.05586},
    primaryClass={cs.CV cs.AI}
}