title: On decoder-only architecture for speech-to-text and large language model
    integration
summary: Large language models (LLMs) have achieved remarkable success in the field of
  natural language processing, enabling better human-computer interaction using
  natural language. However, the seamless integration of speech signals into LLMs
  has not been explored well. The "decoder-only" architecture has also not been
  well studied for speech processing tasks. In this research, we introduce
  Speech-LLaMA, a novel approach that effectively incorporates acoustic
  information into text-based large language models. Our method leverages
  Connectionist Temporal Classification and a simple audio encoder to map the
  compressed acoustic features to the continuous semantic space of the LLM. In
  addition, we further probe the decoder-only architecture for speech-to-text
  tasks by training a smaller scale randomly initialized speech-LLaMA model from
  speech-text paired data alone. We conduct experiments on multilingual
  speech-to-text translation tasks and demonstrate a significant improvement over
  strong baselines, highlighting the potential advantages of decoder-only models
  for speech-to-text conversion.
paper_authors._id: 64acb4b667c1cee8dfde2349
paper_authors.name: Jian Wu
paper_authors._id: 64acb4b667c1cee8dfde234a
paper_authors.name: Yashesh Gaur
paper_authors._id: 64acb4b667c1cee8dfde234b
paper_authors.name: Zhuo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T07:29:41.641Z
paper_authors._id: 642635d22a423daad52bf3e2
paper_authors.avatarUrl: /avatars/dbcb757428a1b9d275c116dd56852f98.svg
paper_authors.fullname: Zhuo Chen
paper_authors.type: user
paper_authors.user: pqpq123
paper_authors._id: 64acb4b667c1cee8dfde234c
paper_authors.name: Long Zhou
paper_authors._id: 64acb4b667c1cee8dfde234d
paper_authors.name: Yimeng Zhu
paper_authors._id: 64acb4b667c1cee8dfde234e
paper_authors.name: Tianrui Wang
paper_authors._id: 64acb4b667c1cee8dfde234f
paper_authors.name: Jinyu Li
paper_authors._id: 64acb4b667c1cee8dfde2350
paper_authors.name: Shujie Liu
paper_authors._id: 64acb4b667c1cee8dfde2351
paper_authors.name: Bo Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:54:12.648Z
paper_authors._id: 668dcf92835bf7e64bbca904
paper_authors.avatarUrl: /avatars/416eb3a3c5318a6a45aad87012296470.svg
paper_authors.fullname: Bo Ren
paper_authors.type: user
paper_authors.user: rosrad
paper_authors._id: 64acb4b667c1cee8dfde2352
paper_authors.name: Linquan Liu
paper_authors._id: 64acb4b667c1cee8dfde2353
paper_authors.name: Yu Wu
paper_title: On decoder-only architecture for speech-to-text and large language model
    integration
paper_summary: Large language models (LLMs) have achieved remarkable success in the field of
  natural language processing, enabling better human-computer interaction using
  natural language. However, the seamless integration of speech signals into LLMs
  has not been explored well. The "decoder-only" architecture has also not been
  well studied for speech processing tasks. In this research, we introduce
  Speech-LLaMA, a novel approach that effectively incorporates acoustic
  information into text-based large language models. Our method leverages
  Connectionist Temporal Classification and a simple audio encoder to map the
  compressed acoustic features to the continuous semantic space of the LLM. In
  addition, we further probe the decoder-only architecture for speech-to-text
  tasks by training a smaller scale randomly initialized speech-LLaMA model from
  speech-text paired data alone. We conduct experiments on multilingual
  speech-to-text translation tasks and demonstrate a significant improvement over
  strong baselines, highlighting the potential advantages of decoder-only models
  for speech-to-text conversion.
paper_ai_keywords: Speech-LLaMA Connectionist Temporal Classification audio encoder compressed acoustic features continuous semantic space decoder-only architecture speech-to-text tasks multilingual speech-to-text translation tasks