title: Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong
    General Audio Event Taggers
summary: In this paper, we focus on Whisper, a recent automatic speech recognition
  model trained with a massive 680k hour labeled speech corpus recorded in
  diverse conditions. We first show an interesting finding that while Whisper is
  very robust against real-world background sounds (e.g., music), its audio
  representation is actually not noise-invariant, but is instead highly
  correlated to non-speech sounds, indicating that Whisper recognizes speech
  conditioned on the noise type. With this finding, we build a unified audio
  tagging and speech recognition model Whisper-AT by freezing the backbone of
  Whisper, and training a lightweight audio tagging model on top of it. With <1%
  extra computational cost, Whisper-AT can recognize audio events, in addition to
  spoken text, in a single forward pass.
paper_authors._id: 64a77fb21fc2c199e62f7b97
paper_authors.name: Yuan Gong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-10T09:24:30.286Z
paper_authors._id: 63b5db8ebda8d44adf35285e
paper_authors.avatarUrl: /avatars/24c98c05509fa7b1ac3371ceb1b81f67.svg
paper_authors.fullname: Yuan Gong
paper_authors.type: user
paper_authors.user: yuangongfdu
paper_authors._id: 64a77fb21fc2c199e62f7b98
paper_authors.name: Sameer Khurana
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-07T13:16:54.592Z
paper_authors._id: 62050bb6a03b2e13bf5401cc
paper_authors.avatarUrl: /avatars/d45580d0a858e6194dc169169b0a2d57.svg
paper_authors.fullname: Sameer Khurana
paper_authors.type: user
paper_authors.user: skhurana
paper_authors._id: 64a77fb21fc2c199e62f7b99
paper_authors.name: Leonid Karlinsky
paper_authors._id: 64a77fb21fc2c199e62f7b9a
paper_authors.name: James Glass
paper_title: Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong
    General Audio Event Taggers
paper_summary: In this paper, we focus on Whisper, a recent automatic speech recognition
  model trained with a massive 680k hour labeled speech corpus recorded in
  diverse conditions. We first show an interesting finding that while Whisper is
  very robust against real-world background sounds (e.g., music), its audio
  representation is actually not noise-invariant, but is instead highly
  correlated to non-speech sounds, indicating that Whisper recognizes speech
  conditioned on the noise type. With this finding, we build a unified audio
  tagging and speech recognition model Whisper-AT by freezing the backbone of
  Whisper, and training a lightweight audio tagging model on top of it. With <1%
  extra computational cost, Whisper-AT can recognize audio events, in addition to
  spoken text, in a single forward pass.
paper_ai_keywords: Whisper automatic speech recognition massive 680k hour labeled speech corpus noise-invariant audio representation Whisper-AT backbone lightweight audio tagging model audio events