title: Attention or Convolution: Transformer Encoders in Audio Language Models
    for Inference Efficiency
summary: In this paper, we show that a simple self-supervised pre-trained audio model
  can achieve comparable inference efficiency to more complicated pre-trained
  models with speech transformer encoders. These speech transformers rely on
  mixing convolutional modules with self-attention modules. They achieve
  state-of-the-art performance on ASR with top efficiency. We first show that
  employing these speech transformers as an encoder significantly improves the
  efficiency of pre-trained audio models as well. However, our study shows that
  we can achieve comparable efficiency with advanced self-attention solely. We
  demonstrate that this simpler approach is particularly beneficial with a
  low-bit weight quantization technique of a neural network to improve
  efficiency. We hypothesize that it prevents propagating the errors between
  different quantized modules compared to recent speech transformers mixing
  quantized convolution and the quantized self-attention modules.
paper_authors._id: 6549c3df8fde27109bcfc4fb
paper_authors.name: Sungho Jeon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:21:22.022Z
paper_authors._id: 64f897a25515d7dccea84b3a
paper_authors.avatarUrl: /avatars/7b7ec64f453e45be7811905befb4c967.svg
paper_authors.fullname: Sungho Jeon
paper_authors.type: user
paper_authors.user: sdeva14
paper_authors._id: 6549c3df8fde27109bcfc4fc
paper_authors.name: Ching-Feng Yeh
paper_authors._id: 6549c3df8fde27109bcfc4fd
paper_authors.name: Hakan Inan
paper_authors._id: 6549c3df8fde27109bcfc4fe
paper_authors.name: Wei-Ning Hsu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:21:42.093Z
paper_authors._id: 64c143d484887aa2b43bac0d
paper_authors.avatarUrl: /avatars/817dba24739676aa0f34692702bb00d6.svg
paper_authors.fullname: Wei-Ning Hsu
paper_authors.type: user
paper_authors.user: wnhsu
paper_authors._id: 6549c3df8fde27109bcfc4ff
paper_authors.name: Rashi Rungta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:21:48.570Z
paper_authors._id: 6227b906e78328793ae0edd9
paper_authors.avatarUrl: /avatars/7ed1773abb419d7862940005b7970d03.svg
paper_authors.fullname: Rashi Rungta
paper_authors.type: user
paper_authors.user: rr654
paper_authors._id: 6549c3df8fde27109bcfc500
paper_authors.name: Yashar Mehdad
paper_authors._id: 6549c3df8fde27109bcfc501
paper_authors.name: Daniel Bikel
paper_title: Attention or Convolution: Transformer Encoders in Audio Language Models
    for Inference Efficiency
paper_summary: In this paper, we show that a simple self-supervised pre-trained audio model
  can achieve comparable inference efficiency to more complicated pre-trained
  models with speech transformer encoders. These speech transformers rely on
  mixing convolutional modules with self-attention modules. They achieve
  state-of-the-art performance on ASR with top efficiency. We first show that
  employing these speech transformers as an encoder significantly improves the
  efficiency of pre-trained audio models as well. However, our study shows that
  we can achieve comparable efficiency with advanced self-attention solely. We
  demonstrate that this simpler approach is particularly beneficial with a
  low-bit weight quantization technique of a neural network to improve
  efficiency. We hypothesize that it prevents propagating the errors between
  different quantized modules compared to recent speech transformers mixing
  quantized convolution and the quantized self-attention modules.
paper_ai_keywords: speech transformer encoders convolutional modules self-attention modules ASR (Automatic Speech Recognition) state-of-the-art performance low-bit weight quantization quantized modules quantized convolution quantized self-attention modules