Learning to Highlight Audio by Watching Movies
Recent years have seen a significant increase in video content creation and
consumption. Crafting engaging content requires the careful curation of both
visual and audio elements. While visual cue curation, through techniques like
optimal viewpoint selection or post-editing, has been central to media
production, its natural counterpart, audio, has not undergone equivalent
advancements. This often results in a disconnect between visual and acoustic
saliency. To bridge this gap, we introduce a novel task: visually-guided
acoustic highlighting, which aims to transform audio to deliver appropriate
highlighting effects guided by the accompanying video, ultimately creating a
more harmonious audio-visual experience. We propose a flexible,
transformer-based multimodal framework to solve this task. To train our model,
we also introduce a new dataset -- the muddy mix dataset, leveraging the
meticulous audio and video crafting found in movies, which provides a form of
free supervision. We develop a pseudo-data generation process to simulate
poorly mixed audio, mimicking real-world scenarios through a three-step process
-- separation, adjustment, and remixing. Our approach consistently outperforms
several baselines in both quantitative and subjective evaluation. We also
systematically study the impact of different types of contextual guidance and
difficulty levels of the dataset. Our project page is here:
https://wikichao.github.io/VisAH/.
682dfacb59d73d96c5f2dfd4
Chao Huang
682dfacb59d73d96c5f2dfd5
Ruohan Gao
682dfacb59d73d96c5f2dfd6
J. M. F. Tsang
682dfacb59d73d96c5f2dfd7
Jan Kurcius
682dfacb59d73d96c5f2dfd8
Cagdas Bilen
682dfacb59d73d96c5f2dfd9
Chenliang Xu
682dfacb59d73d96c5f2dfda
Anurag Kumar
682dfacb59d73d96c5f2dfdb
Sanjeel Parekh
Learning to Highlight Audio by Watching Movies
Recent years have seen a significant increase in video content creation and
consumption. Crafting engaging content requires the careful curation of both
visual and audio elements. While visual cue curation, through techniques like
optimal viewpoint selection or post-editing, has been central to media
production, its natural counterpart, audio, has not undergone equivalent
advancements. This often results in a disconnect between visual and acoustic
saliency. To bridge this gap, we introduce a novel task: visually-guided
acoustic highlighting, which aims to transform audio to deliver appropriate
highlighting effects guided by the accompanying video, ultimately creating a
more harmonious audio-visual experience. We propose a flexible,
transformer-based multimodal framework to solve this task. To train our model,
we also introduce a new dataset -- the muddy mix dataset, leveraging the
meticulous audio and video crafting found in movies, which provides a form of
free supervision. We develop a pseudo-data generation process to simulate
poorly mixed audio, mimicking real-world scenarios through a three-step process
-- separation, adjustment, and remixing. Our approach consistently outperforms
several baselines in both quantitative and subjective evaluation. We also
systematically study the impact of different types of contextual guidance and
difficulty levels of the dataset. Our project page is here:
https://wikichao.github.io/VisAH/.
A transformer-based multimodal framework addresses the challenge of visually-guided acoustic highlighting, creating harmonious audio-visual experiences using a newly introduced dataset and pseudo-data generation process.
visually-guided acoustic highlighting transformer-based multimodal framework muddy mix dataset audio visualization contextual guidance