title: SALMONN: Towards Generic Hearing Abilities for Large Language Models
summary: Hearing is arguably an essential ability of artificial intelligence (AI)
  agents in the physical world, which refers to the perception and understanding
  of general auditory information consisting of at least three types of sounds:
  speech, audio events, and music. In this paper, we propose SALMONN, a speech
  audio language music open neural network, built by integrating a pre-trained
  text-based large language model (LLM) with speech and audio encoders into a
  single multimodal model. SALMONN enables the LLM to directly process and
  understand general audio inputs and achieve competitive performances on a
  number of speech and audio tasks used in training, such as automatic speech
  recognition and translation, auditory-information-based question answering,
  emotion recognition, speaker verification, and music and audio captioning
  etc. SALMONN also has a diverse set of emergent abilities unseen in
  the training, which includes but is not limited to speech translation to
  untrained languages, speech-based slot filling, spoken-query-based question
  answering, audio-based storytelling, and speech audio co-reasoning
  etc. The presence of the cross-modal emergent abilities is studied,
  and a novel few-shot activation tuning approach is proposed to activate such
  abilities of SALMONN. To our knowledge, SALMONN is the first model of its type
  and can be regarded as a step towards AI with generic hearing abilities. An
  interactive demo of SALMONN is available at
  \url{https://github.com/bytedance/SALMONN}, and the training code and
  model checkpoints will be released upon acceptance.
paper_authors._id: 6535eb4699d8bba294550d3c
paper_authors.name: Changli Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:55:43.692Z
paper_authors._id: 63770389cdcc1bf630870758
paper_authors.avatarUrl: /avatars/6bc7c602e79688bc42a4c79ecf5b6d2d.svg
paper_authors.fullname: Changli Tang
paper_authors.type: user
paper_authors.user: Changli
paper_authors._id: 6535eb4699d8bba294550d3d
paper_authors.name: Wenyi Yu
paper_authors._id: 6535eb4699d8bba294550d3e
paper_authors.name: Guangzhi Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:56:14.406Z
paper_authors._id: 64ab544489aa67e4a2505eeb
paper_authors.avatarUrl: /avatars/f1a9def3afbec2f8b89ef4450770d67e.svg
paper_authors.fullname: Guangzhi Sun
paper_authors.type: user
paper_authors.user: BrianatCambridge
paper_authors._id: 6535eb4699d8bba294550d3f
paper_authors.name: Xianzhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:56:56.476Z
paper_authors._id: 646617cfcf550af36eb2e773
paper_authors.avatarUrl: /avatars/963271ee1a69b5524dafe5d90c93ff3f.svg
paper_authors.fullname: chenxianzhao
paper_authors.type: user
paper_authors.user: chenxianzhao
paper_authors._id: 6535eb4699d8bba294550d40
paper_authors.name: Tian Tan
paper_authors._id: 6535eb4699d8bba294550d41
paper_authors.name: Wei Li
paper_authors._id: 6535eb4699d8bba294550d42
paper_authors.name: Lu Lu
paper_authors._id: 6535eb4699d8bba294550d43
paper_authors.name: Zejun Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:58:02.291Z
paper_authors._id: 64cfc1d4228324a28bd411c2
paper_authors.avatarUrl: /avatars/2d53afb1f32e8620568862681ce3bd27.svg
paper_authors.fullname: MA
paper_authors.type: user
paper_authors.user: Zejun68
paper_authors._id: 6535eb4699d8bba294550d44
paper_authors.name: Chao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-23T09:43:47.915Z
paper_authors._id: 65363f359326d6da5f651f2e
paper_authors.avatarUrl: /avatars/13c0dce7c5ee2f1bd5eb451e8707979e.svg
paper_authors.fullname: Chao Zhang
paper_authors.type: user
paper_authors.user: cz277
paper_title: SALMONN: Towards Generic Hearing Abilities for Large Language Models
paper_summary: Hearing is arguably an essential ability of artificial intelligence (AI)
  agents in the physical world, which refers to the perception and understanding
  of general auditory information consisting of at least three types of sounds:
  speech, audio events, and music. In this paper, we propose SALMONN, a speech
  audio language music open neural network, built by integrating a pre-trained
  text-based large language model (LLM) with speech and audio encoders into a
  single multimodal model. SALMONN enables the LLM to directly process and
  understand general audio inputs and achieve competitive performances on a
  number of speech and audio tasks used in training, such as automatic speech
  recognition and translation, auditory-information-based question answering,
  emotion recognition, speaker verification, and music and audio captioning
  etc. SALMONN also has a diverse set of emergent abilities unseen in
  the training, which includes but is not limited to speech translation to
  untrained languages, speech-based slot filling, spoken-query-based question
  answering, audio-based storytelling, and speech audio co-reasoning
  etc. The presence of the cross-modal emergent abilities is studied,
  and a novel few-shot activation tuning approach is proposed to activate such
  abilities of SALMONN. To our knowledge, SALMONN is the first model of its type
  and can be regarded as a step towards AI with generic hearing abilities. An
  interactive demo of SALMONN is available at
  \url{https://github.com/bytedance/SALMONN}, and the training code and
  model checkpoints will be released upon acceptance.
paper_ai_keywords: SALMONN large language model (LLM) speech encoders audio encoders multimodal model automatic speech recognition translation auditory-information-based question answering emotion recognition speaker verification music captioning audio captioning speech translation slot filling spoken-query-based question answering audio-based storytelling speech audio co-reasoning cross-modal emergent abilities few-shot activation tuning