title: AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual
    Segmentation
summary: Recently, transformer-based models have demonstrated remarkable performance
  on audio-visual segmentation (AVS) tasks. However, their expensive
  computational cost makes real-time inference impractical. By characterizing
  attention maps of the network, we identify two key obstacles in AVS models: 1)
  attention dissipation, corresponding to the over-concentrated attention weights
  by Softmax within restricted frames, and 2) inefficient, burdensome transformer
  decoder, caused by narrow focus patterns in early stages. In this paper, we
  introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation
  transformer that achieves fast, efficient and light-weight simultaneously. Our
  model leverages an efficient prompt query generator to correct the behaviour of
  cross-attention. Additionally, we propose ELF decoder to bring greater
  efficiency by facilitating convolutions suitable for local features to reduce
  computational burdens. Extensive experiments demonstrate that our AVESFormer
  significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3
  and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an
  excellent trade-off between performance and speed. Code can be found at
  https://github.com/MarkXCloud/AVESFormer.git.
paper_authors._id: 66b3095cc80e840a1f7be6e0
paper_authors.name: Zili Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T07:40:28.854Z
paper_authors._id: 6410998db27543634e3aacc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: MarkWang
paper_authors._id: 66b3095cc80e840a1f7be6e1
paper_authors.name: Qi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T09:57:05.011Z
paper_authors._id: 643e45c4c639f8bc9727810a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643e45c4c639f8bc9727810a/BJR1cvSCxcqxr08iS7GsI.jpeg
paper_authors.fullname: YannQi
paper_authors.type: user
paper_authors.user: YannQi
paper_authors._id: 66b3095cc80e840a1f7be6e2
paper_authors.name: Linsu Shi
paper_authors._id: 66b3095cc80e840a1f7be6e3
paper_authors.name: Jiazhong Yu
paper_authors._id: 66b3095cc80e840a1f7be6e4
paper_authors.name: Qinghua Liang
paper_authors._id: 66b3095cc80e840a1f7be6e5
paper_authors.name: Fei Li
paper_authors._id: 66b3095cc80e840a1f7be6e6
paper_authors.name: Shiming Xiang
paper_title: AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual
    Segmentation
paper_summary: Recently, transformer-based models have demonstrated remarkable performance
  on audio-visual segmentation (AVS) tasks. However, their expensive
  computational cost makes real-time inference impractical. By characterizing
  attention maps of the network, we identify two key obstacles in AVS models: 1)
  attention dissipation, corresponding to the over-concentrated attention weights
  by Softmax within restricted frames, and 2) inefficient, burdensome transformer
  decoder, caused by narrow focus patterns in early stages. In this paper, we
  introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation
  transformer that achieves fast, efficient and light-weight simultaneously. Our
  model leverages an efficient prompt query generator to correct the behaviour of
  cross-attention. Additionally, we propose ELF decoder to bring greater
  efficiency by facilitating convolutions suitable for local features to reduce
  computational burdens. Extensive experiments demonstrate that our AVESFormer
  significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3
  and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an
  excellent trade-off between performance and speed. Code can be found at
  https://github.com/MarkXCloud/AVESFormer.git.
paper_ai_keywords: attention maps AVS models attention dissipation Softmax transformer decoder AVESFormer real-time Audio-Visual Efficient Segmentation prompt query generator cross-attention ELF decoder convolutions local features S4 MS3 AVSS