Attention or Convolution: Transformer Encoders in Audio Language Models
  for Inference Efficiency
In this paper, we show that a simple self-supervised pre-trained audio model
can achieve comparable inference efficiency to more complicated pre-trained
models with speech transformer encoders. These speech transformers rely on
mixing convolutional modules with self-attention modules. They achieve
state-of-the-art performance on ASR with top efficiency. We first show that
employing these speech transformers as an encoder significantly improves the
efficiency of pre-trained audio models as well. However, our study shows that
we can achieve comparable efficiency with advanced self-attention solely. We
demonstrate that this simpler approach is particularly beneficial with a
low-bit weight quantization technique of a neural network to improve
efficiency. We hypothesize that it prevents propagating the errors between
different quantized modules compared to recent speech transformers mixing
quantized convolution and the quantized self-attention modules.
6549c3df8fde27109bcfc4fb
Sungho Jeon
admin_assigned
2023-11-07T17:21:22.022Z
64f897a25515d7dccea84b3a
/avatars/7b7ec64f453e45be7811905befb4c967.svg
Sungho Jeon
user
sdeva14
6549c3df8fde27109bcfc4fc
Ching-Feng Yeh
6549c3df8fde27109bcfc4fd
Hakan Inan
6549c3df8fde27109bcfc4fe
Wei-Ning Hsu
admin_assigned
2023-11-07T17:21:42.093Z
64c143d484887aa2b43bac0d
/avatars/817dba24739676aa0f34692702bb00d6.svg
Wei-Ning Hsu
user
wnhsu
6549c3df8fde27109bcfc4ff
Rashi Rungta
admin_assigned
2023-11-07T17:21:48.570Z
6227b906e78328793ae0edd9
/avatars/7ed1773abb419d7862940005b7970d03.svg
Rashi Rungta
user
rr654
6549c3df8fde27109bcfc500
Yashar Mehdad
6549c3df8fde27109bcfc501
Daniel Bikel
Attention or Convolution: Transformer Encoders in Audio Language Models
  for Inference Efficiency
In this paper, we show that a simple self-supervised pre-trained audio model
can achieve comparable inference efficiency to more complicated pre-trained
models with speech transformer encoders. These speech transformers rely on
mixing convolutional modules with self-attention modules. They achieve
state-of-the-art performance on ASR with top efficiency. We first show that
employing these speech transformers as an encoder significantly improves the
efficiency of pre-trained audio models as well. However, our study shows that
we can achieve comparable efficiency with advanced self-attention solely. We
demonstrate that this simpler approach is particularly beneficial with a
low-bit weight quantization technique of a neural network to improve
efficiency. We hypothesize that it prevents propagating the errors between
different quantized modules compared to recent speech transformers mixing
quantized convolution and the quantized self-attention modules.
A simpler self-attention-only approach in pre-trained audio models matches the efficiency of more complex speech transformer models, especially with low-bit weight quantization.
self-supervised pre-trained audio model speech transformer convolutional modules self-attention modules ASR efficiency encoder low-bit weight quantization error propagation quantized modules