AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual
  Segmentation
Recently, transformer-based models have demonstrated remarkable performance
on audio-visual segmentation (AVS) tasks. However, their expensive
computational cost makes real-time inference impractical. By characterizing
attention maps of the network, we identify two key obstacles in AVS models: 1)
attention dissipation, corresponding to the over-concentrated attention weights
by Softmax within restricted frames, and 2) inefficient, burdensome transformer
decoder, caused by narrow focus patterns in early stages. In this paper, we
introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation
transformer that achieves fast, efficient and light-weight simultaneously. Our
model leverages an efficient prompt query generator to correct the behaviour of
cross-attention. Additionally, we propose ELF decoder to bring greater
efficiency by facilitating convolutions suitable for local features to reduce
computational burdens. Extensive experiments demonstrate that our AVESFormer
significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3
and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an
excellent trade-off between performance and speed. Code can be found at
https://github.com/MarkXCloud/AVESFormer.git.
66b3095cc80e840a1f7be6e0
Zili Wang
claimed_verified
2024-08-08T07:40:28.854Z
6410998db27543634e3aacc3
https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
Zili Wang
user
MarkWang
66b3095cc80e840a1f7be6e1
Qi Yang
claimed_verified
2024-09-11T09:57:05.011Z
643e45c4c639f8bc9727810a
https://cdn-avatars.huggingface.co/v1/production/uploads/643e45c4c639f8bc9727810a/BJR1cvSCxcqxr08iS7GsI.jpeg
YannQi
user
YannQi
66b3095cc80e840a1f7be6e2
Linsu Shi
66b3095cc80e840a1f7be6e3
Jiazhong Yu
66b3095cc80e840a1f7be6e4
Qinghua Liang
66b3095cc80e840a1f7be6e5
Fei Li
66b3095cc80e840a1f7be6e6
Shiming Xiang
AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual
  Segmentation
Recently, transformer-based models have demonstrated remarkable performance
on audio-visual segmentation (AVS) tasks. However, their expensive
computational cost makes real-time inference impractical. By characterizing
attention maps of the network, we identify two key obstacles in AVS models: 1)
attention dissipation, corresponding to the over-concentrated attention weights
by Softmax within restricted frames, and 2) inefficient, burdensome transformer
decoder, caused by narrow focus patterns in early stages. In this paper, we
introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation
transformer that achieves fast, efficient and light-weight simultaneously. Our
model leverages an efficient prompt query generator to correct the behaviour of
cross-attention. Additionally, we propose ELF decoder to bring greater
efficiency by facilitating convolutions suitable for local features to reduce
computational burdens. Extensive experiments demonstrate that our AVESFormer
significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3
and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an
excellent trade-off between performance and speed. Code can be found at
https://github.com/MarkXCloud/AVESFormer.git.
AVESFormer addresses computational inefficiencies in transformer-based audio-visual segmentation through a prompt query generator and ELF decoder, enabling real-time performance with improved accuracy.
transformer-based models attention maps attention dissipation Softmax transformer decoder AVESFormer Audio-Visual Efficient Segmentation prompt query generator cross-attention ELF decoder S4 MS3 AVSS