AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description
<|reference_start|>AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description: Our objective is to generate Audio Descriptions (ADs) for both movies and TV series in a training-free manner. We use the power of off-the-shelf Visual-Language Models (VLMs) and Large Language Models (LLMs), and develop visual and text prompting strategies for this task. Our contributions are three-fold: (i) We demonstrate that a VLM can successfully name and refer to characters if directly prompted with character information through visual indications without requiring any fine-tuning; (ii) A two-stage process is developed to generate ADs, with the first stage asking the VLM to comprehensively describe the video, followed by a second stage utilising a LLM to summarise dense textual information into one succinct AD sentence; (iii) A new dataset for TV audio description is formulated. Our approach, named AutoAD-Zero, demonstrates outstanding performance (even competitive with some models fine-tuned on ground truth ADs) in AD generation for both movies and TV series, achieving state-of-the-art CRITIC scores.<|reference_end|>
@article{xie2024autoad-zero:,
    title={AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description},
    author={Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, G"ul Varol, Weidi
  Xie, Andrew Zisserman},
    journal={arXiv preprint arXiv:2407.15850},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.15850},
    primaryClass={cs.CV}
}