AutoAD III: The Prequel -- Back to the Pixels
<|reference_start|>AutoAD III: The Prequel -- Back to the Pixels: Generating Audio Description (AD) for movies is a challenging task that requires fine-grained visual understanding and an awareness of the characters and their names. Currently, visual language models for AD generation are limited by a lack of suitable training data, and also their evaluation is hampered by using performance measures not specialized to the AD domain. In this paper, we make three contributions: (i) We propose two approaches for constructing AD datasets with aligned video data, and build training and evaluation datasets using these. These datasets will be publicly released; (ii) We develop a Q-former-based architecture which ingests raw video and generates AD, using frozen pre-trained visual encoders and large language models; and (iii) We provide new evaluation metrics to benchmark AD quality that are well-matched to human performance. Taken together, we improve the state of the art on AD generation.<|reference_end|>
@article{han2024autoad,
    title={AutoAD III: The Prequel -- Back to the Pixels},
    author={Tengda Han, Max Bain, Arsha Nagrani, G"ul Varol, Weidi Xie, Andrew
  Zisserman},
    journal={arXiv preprint arXiv:2404.14412},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.14412},
    primaryClass={cs.CV}
}