title: Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
summary: Traditional reference segmentation tasks have predominantly focused on silent
  visual scenes, neglecting the integral role of multimodal perception and
  interaction in human experiences. In this work, we introduce a novel task
  called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment
  objects within the visual domain based on expressions containing multimodal
  cues. Such expressions are articulated in natural language forms but are
  enriched with multimodal cues, including audio and visual descriptions. To
  facilitate this research, we construct the first Ref-AVS benchmark, which
  provides pixel-level annotations for objects described in corresponding
  multimodal-cue expressions. To tackle the Ref-AVS task, we propose a new method
  that adequately utilizes multimodal cues to offer precise segmentation
  guidance. Finally, we conduct quantitative and qualitative experiments on three
  test subsets to compare our approach with existing methods from related tasks.
  The results demonstrate the effectiveness of our method, highlighting its
  capability to precisely segment objects using multimodal-cue expressions.
  Dataset is available at
  https://gewu-lab.github.io/Ref-AVS{https://gewu-lab.github.io/Ref-AVS}.
paper_authors._id: 6697980e12d67a42d78b10d2
paper_authors.name: Yaoting Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-17T10:08:17.268Z
paper_authors._id: 64ff369d9abcc85a5519b33e
paper_authors.avatarUrl: /avatars/4b99cdaf5f970d930b196eddf1e5e499.svg
paper_authors.fullname: Yaoting Wang
paper_authors.type: user
paper_authors.user: Gh0stAR
paper_authors._id: 6697980e12d67a42d78b10d3
paper_authors.name: Peiwen Sun
paper_authors._id: 6697980e12d67a42d78b10d4
paper_authors.name: Dongzhan Zhou
paper_authors._id: 6697980e12d67a42d78b10d5
paper_authors.name: Guangyao Li
paper_authors._id: 6697980e12d67a42d78b10d6
paper_authors.name: Honggang Zhang
paper_authors._id: 6697980e12d67a42d78b10d7
paper_authors.name: Di Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:06:17.697Z
paper_authors._id: 669799b26d173ec7382848d1
paper_authors.avatarUrl: /avatars/a45490aa5c9d30985b8126e3381dbabb.svg
paper_authors.fullname: DTao Hu
paper_authors.type: user
paper_authors.user: DTaoo
paper_title: Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
paper_summary: Traditional reference segmentation tasks have predominantly focused on silent
  visual scenes, neglecting the integral role of multimodal perception and
  interaction in human experiences. In this work, we introduce a novel task
  called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment
  objects within the visual domain based on expressions containing multimodal
  cues. Such expressions are articulated in natural language forms but are
  enriched with multimodal cues, including audio and visual descriptions. To
  facilitate this research, we construct the first Ref-AVS benchmark, which
  provides pixel-level annotations for objects described in corresponding
  multimodal-cue expressions. To tackle the Ref-AVS task, we propose a new method
  that adequately utilizes multimodal cues to offer precise segmentation
  guidance. Finally, we conduct quantitative and qualitative experiments on three
  test subsets to compare our approach with existing methods from related tasks.
  The results demonstrate the effectiveness of our method, highlighting its
  capability to precisely segment objects using multimodal-cue expressions.
  Dataset is available at
  https://gewu-lab.github.io/Ref-AVS{https://gewu-lab.github.io/Ref-AVS}.
paper_ai_summary: A new task, Reference Audio-Visual Segmentation (Ref-AVS), is introduced to segment objects using multimodal cues, and a method leveraging these cues outperforms existing approaches in experiments.
paper_ai_keywords: Reference Audio-Visual Segmentation Ref-AVS multimodal perception multimodal cues pixel-level annotations quantitative experiments qualitative experiments