title: Open-Vocabulary Audio-Visual Semantic Segmentation
summary: Audio-visual semantic segmentation (AVSS) aims to segment and classify
  sounding objects in videos with acoustic cues. However, most approaches operate
  on the close-set assumption and only identify pre-defined categories from
  training data, lacking the generalization ability to detect novel categories in
  practical applications. In this paper, we introduce a new task: open-vocabulary
  audio-visual semantic segmentation, extending AVSS task to open-world scenarios
  beyond the annotated label space. This is a more challenging task that requires
  recognizing all categories, even those that have never been seen nor heard
  during training. Moreover, we propose the first open-vocabulary AVSS framework,
  OV-AVSS, which mainly consists of two parts: 1) a universal sound source
  localization module to perform audio-visual fusion and locate all potential
  sounding objects and 2) an open-vocabulary classification module to predict
  categories with the help of the prior knowledge from large-scale pre-trained
  vision-language models. To properly evaluate the open-vocabulary AVSS, we split
  zero-shot training and testing subsets based on the AVSBench-semantic
  benchmark, namely AVSBench-OV. Extensive experiments demonstrate the strong
  segmentation and zero-shot generalization ability of our model on all
  categories. On the AVSBench-OV dataset, OV-AVSS achieves 55.43% mIoU on base
  categories and 29.14% mIoU on novel categories, exceeding the state-of-the-art
  zero-shot method by 41.88%/20.61% and open-vocabulary method by 10.2%/11.6%.
  The code is available at https://github.com/ruohaoguo/ovavss.
paper_authors._id: 66ab2e17c5f3bec3ef2eec13
paper_authors.name: Ruohao Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T15:30:32.120Z
paper_authors._id: 628711eb40423ef48fbc07af
paper_authors.avatarUrl: /avatars/1a0755f880c6b0b4942cbfd6a6631c34.svg
paper_authors.fullname: Ruohao Guo
paper_authors.type: user
paper_authors.user: Ruohao
paper_authors._id: 66ab2e17c5f3bec3ef2eec14
paper_authors.name: Liao Qu
paper_authors._id: 66ab2e17c5f3bec3ef2eec15
paper_authors.name: Dantong Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T15:31:17.561Z
paper_authors._id: 65c3fdf79d062be813813e45
paper_authors.avatarUrl: /avatars/52528a61abe5bbbef4a4a431944973cd.svg
paper_authors.fullname: Dantong Niu
paper_authors.type: user
paper_authors.user: NdtSoCool
paper_authors._id: 66ab2e17c5f3bec3ef2eec16
paper_authors.name: Yanyu Qi
paper_authors._id: 66ab2e17c5f3bec3ef2eec17
paper_authors.name: Wenzhen Yue
paper_authors._id: 66ab2e17c5f3bec3ef2eec18
paper_authors.name: Ji Shi
paper_authors._id: 66ab2e17c5f3bec3ef2eec19
paper_authors.name: Bowei Xing
paper_authors._id: 66ab2e17c5f3bec3ef2eec1a
paper_authors.name: Xianghua Ying
paper_title: Open-Vocabulary Audio-Visual Semantic Segmentation
paper_summary: Audio-visual semantic segmentation (AVSS) aims to segment and classify
  sounding objects in videos with acoustic cues. However, most approaches operate
  on the close-set assumption and only identify pre-defined categories from
  training data, lacking the generalization ability to detect novel categories in
  practical applications. In this paper, we introduce a new task: open-vocabulary
  audio-visual semantic segmentation, extending AVSS task to open-world scenarios
  beyond the annotated label space. This is a more challenging task that requires
  recognizing all categories, even those that have never been seen nor heard
  during training. Moreover, we propose the first open-vocabulary AVSS framework,
  OV-AVSS, which mainly consists of two parts: 1) a universal sound source
  localization module to perform audio-visual fusion and locate all potential
  sounding objects and 2) an open-vocabulary classification module to predict
  categories with the help of the prior knowledge from large-scale pre-trained
  vision-language models. To properly evaluate the open-vocabulary AVSS, we split
  zero-shot training and testing subsets based on the AVSBench-semantic
  benchmark, namely AVSBench-OV. Extensive experiments demonstrate the strong
  segmentation and zero-shot generalization ability of our model on all
  categories. On the AVSBench-OV dataset, OV-AVSS achieves 55.43% mIoU on base
  categories and 29.14% mIoU on novel categories, exceeding the state-of-the-art
  zero-shot method by 41.88%/20.61% and open-vocabulary method by 10.2%/11.6%.
  The code is available at https://github.com/ruohaoguo/ovavss.
paper_ai_keywords: audio-visual semantic segmentation open-vocabulary audio-visual semantic segmentation sound source localization audio-visual fusion open-vocabulary classification vision-language models zero-shot training zero-shot generalization AVSBench-semantic AVSBench-OV base categories novel categories mIoU