FocusedAD: Character-centric Movie Audio Description
Movie Audio Description (AD) aims to narrate visual content during
dialogue-free segments, particularly benefiting blind and visually impaired
(BVI) audiences. Compared with general video captioning, AD demands
plot-relevant narration with explicit character name references, posing unique
challenges in movie understanding.To identify active main characters and focus
on storyline-relevant regions, we propose FocusedAD, a novel framework that
delivers character-centric movie audio descriptions. It includes: (i) a
Character Perception Module(CPM) for tracking character regions and linking
them to names; (ii) a Dynamic Prior Module(DPM) that injects contextual cues
from prior ADs and subtitles via learnable soft prompts; and (iii) a Focused
Caption Module(FCM) that generates narrations enriched with plot-relevant
details and named characters. To overcome limitations in character
identification, we also introduce an automated pipeline for building character
query banks. FocusedAD achieves state-of-the-art performance on multiple
benchmarks, including strong zero-shot results on MAD-eval-Named and our newly
proposed Cinepile-AD dataset. Code and data will be released at
https://github.com/Thorin215/FocusedAD .
6801d4446d2188af01a9b6b6
Xiaojun Ye
6801d4446d2188af01a9b6b7
Chun Wang
6801d4446d2188af01a9b6b8
Yiren Song
6801d4446d2188af01a9b6b9
Sheng Zhou
6801d4446d2188af01a9b6ba
Liangcheng Li
6801d4446d2188af01a9b6bb
Jiajun Bu
FocusedAD: Character-centric Movie Audio Description
Movie Audio Description (AD) aims to narrate visual content during
dialogue-free segments, particularly benefiting blind and visually impaired
(BVI) audiences. Compared with general video captioning, AD demands
plot-relevant narration with explicit character name references, posing unique
challenges in movie understanding.To identify active main characters and focus
on storyline-relevant regions, we propose FocusedAD, a novel framework that
delivers character-centric movie audio descriptions. It includes: (i) a
Character Perception Module(CPM) for tracking character regions and linking
them to names; (ii) a Dynamic Prior Module(DPM) that injects contextual cues
from prior ADs and subtitles via learnable soft prompts; and (iii) a Focused
Caption Module(FCM) that generates narrations enriched with plot-relevant
details and named characters. To overcome limitations in character
identification, we also introduce an automated pipeline for building character
query banks. FocusedAD achieves state-of-the-art performance on multiple
benchmarks, including strong zero-shot results on MAD-eval-Named and our newly
proposed Cinepile-AD dataset. Code and data will be released at
https://github.com/Thorin215/FocusedAD .
FocusedAD, a novel framework for movie audio descriptions, tracks characters, incorporates contextual cues, and generates plot-relevant narrations, achieving top performance on benchmarks and zero-shot results.
Character Perception Module Dynamic Prior Module Focused Caption Module character identification character query banks MAD-eval-Named Cinepile-AD dataset