title: Learning to Highlight Audio by Watching Movies
summary: Recent years have seen a significant increase in video content creation and
  consumption. Crafting engaging content requires the careful curation of both
  visual and audio elements. While visual cue curation, through techniques like
  optimal viewpoint selection or post-editing, has been central to media
  production, its natural counterpart, audio, has not undergone equivalent
  advancements. This often results in a disconnect between visual and acoustic
  saliency. To bridge this gap, we introduce a novel task: visually-guided
  acoustic highlighting, which aims to transform audio to deliver appropriate
  highlighting effects guided by the accompanying video, ultimately creating a
  more harmonious audio-visual experience. We propose a flexible,
  transformer-based multimodal framework to solve this task. To train our model,
  we also introduce a new dataset -- the muddy mix dataset, leveraging the
  meticulous audio and video crafting found in movies, which provides a form of
  free supervision. We develop a pseudo-data generation process to simulate
  poorly mixed audio, mimicking real-world scenarios through a three-step process
  -- separation, adjustment, and remixing. Our approach consistently outperforms
  several baselines in both quantitative and subjective evaluation. We also
  systematically study the impact of different types of contextual guidance and
  difficulty levels of the dataset. Our project page is here:
  https://wikichao.github.io/VisAH/.
paper_authors._id: 682dfacb59d73d96c5f2dfd4
paper_authors.name: Chao Huang
paper_authors._id: 682dfacb59d73d96c5f2dfd5
paper_authors.name: Ruohan Gao
paper_authors._id: 682dfacb59d73d96c5f2dfd6
paper_authors.name: J. M. F. Tsang
paper_authors._id: 682dfacb59d73d96c5f2dfd7
paper_authors.name: Jan Kurcius
paper_authors._id: 682dfacb59d73d96c5f2dfd8
paper_authors.name: Cagdas Bilen
paper_authors._id: 682dfacb59d73d96c5f2dfd9
paper_authors.name: Chenliang Xu
paper_authors._id: 682dfacb59d73d96c5f2dfda
paper_authors.name: Anurag Kumar
paper_authors._id: 682dfacb59d73d96c5f2dfdb
paper_authors.name: Sanjeel Parekh
paper_title: Learning to Highlight Audio by Watching Movies
paper_summary: Recent years have seen a significant increase in video content creation and
  consumption. Crafting engaging content requires the careful curation of both
  visual and audio elements. While visual cue curation, through techniques like
  optimal viewpoint selection or post-editing, has been central to media
  production, its natural counterpart, audio, has not undergone equivalent
  advancements. This often results in a disconnect between visual and acoustic
  saliency. To bridge this gap, we introduce a novel task: visually-guided
  acoustic highlighting, which aims to transform audio to deliver appropriate
  highlighting effects guided by the accompanying video, ultimately creating a
  more harmonious audio-visual experience. We propose a flexible,
  transformer-based multimodal framework to solve this task. To train our model,
  we also introduce a new dataset -- the muddy mix dataset, leveraging the
  meticulous audio and video crafting found in movies, which provides a form of
  free supervision. We develop a pseudo-data generation process to simulate
  poorly mixed audio, mimicking real-world scenarios through a three-step process
  -- separation, adjustment, and remixing. Our approach consistently outperforms
  several baselines in both quantitative and subjective evaluation. We also
  systematically study the impact of different types of contextual guidance and
  difficulty levels of the dataset. Our project page is here:
  https://wikichao.github.io/VisAH/.
paper_ai_summary: A transformer-based multimodal framework addresses the challenge of visually-guided acoustic highlighting, creating harmonious audio-visual experiences using a newly introduced dataset and pseudo-data generation process.
paper_ai_keywords: visually-guided acoustic highlighting transformer-based multimodal framework muddy mix dataset audio visualization contextual guidance