Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
Traditional reference segmentation tasks have predominantly focused on silent
visual scenes, neglecting the integral role of multimodal perception and
interaction in human experiences. In this work, we introduce a novel task
called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment
objects within the visual domain based on expressions containing multimodal
cues. Such expressions are articulated in natural language forms but are
enriched with multimodal cues, including audio and visual descriptions. To
facilitate this research, we construct the first Ref-AVS benchmark, which
provides pixel-level annotations for objects described in corresponding
multimodal-cue expressions. To tackle the Ref-AVS task, we propose a new method
that adequately utilizes multimodal cues to offer precise segmentation
guidance. Finally, we conduct quantitative and qualitative experiments on three
test subsets to compare our approach with existing methods from related tasks.
The results demonstrate the effectiveness of our method, highlighting its
capability to precisely segment objects using multimodal-cue expressions.
Dataset is available at
https://gewu-lab.github.io/Ref-AVS{https://gewu-lab.github.io/Ref-AVS}.
6697980e12d67a42d78b10d2
Yaoting Wang
extracted_pending
2024-07-17T10:08:17.268Z
64ff369d9abcc85a5519b33e
/avatars/4b99cdaf5f970d930b196eddf1e5e499.svg
Yaoting Wang
user
Gh0stAR
6697980e12d67a42d78b10d3
Peiwen Sun
6697980e12d67a42d78b10d4
Dongzhan Zhou
6697980e12d67a42d78b10d5
Guangyao Li
6697980e12d67a42d78b10d6
Honggang Zhang
6697980e12d67a42d78b10d7
Di Hu
claimed_verified
2024-07-18T09:06:17.697Z
669799b26d173ec7382848d1
/avatars/a45490aa5c9d30985b8126e3381dbabb.svg
DTao Hu
user
DTaoo
Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
Traditional reference segmentation tasks have predominantly focused on silent
visual scenes, neglecting the integral role of multimodal perception and
interaction in human experiences. In this work, we introduce a novel task
called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment
objects within the visual domain based on expressions containing multimodal
cues. Such expressions are articulated in natural language forms but are
enriched with multimodal cues, including audio and visual descriptions. To
facilitate this research, we construct the first Ref-AVS benchmark, which
provides pixel-level annotations for objects described in corresponding
multimodal-cue expressions. To tackle the Ref-AVS task, we propose a new method
that adequately utilizes multimodal cues to offer precise segmentation
guidance. Finally, we conduct quantitative and qualitative experiments on three
test subsets to compare our approach with existing methods from related tasks.
The results demonstrate the effectiveness of our method, highlighting its
capability to precisely segment objects using multimodal-cue expressions.
Dataset is available at
https://gewu-lab.github.io/Ref-AVS{https://gewu-lab.github.io/Ref-AVS}.
A new task, Reference Audio-Visual Segmentation (Ref-AVS), is introduced to segment objects using multimodal cues, and a method leveraging these cues outperforms existing approaches in experiments.
Reference Audio-Visual Segmentation Ref-AVS multimodal perception multimodal cues pixel-level annotations quantitative experiments qualitative experiments