title: Roadmap towards Superhuman Speech Understanding using Large Language
    Models
summary: The success of large language models (LLMs) has prompted efforts to integrate
  speech and audio data, aiming to create general foundation models capable of
  processing both textual and non-textual inputs. Recent advances, such as
  GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves
  non-semantic information and world knowledge for deeper speech understanding.
  To guide the development of speech LLMs, we propose a five-level roadmap,
  ranging from basic automatic speech recognition (ASR) to advanced superhuman
  models capable of integrating non-semantic information with abstract acoustic
  knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark,
  that standardizes critical aspects across various tasks in these five levels,
  uncovering challenges in using abstract acoustic knowledge and completeness of
  capability. Our findings reveal gaps in handling paralinguistic cues and
  abstract acoustic knowledge, and we offer future directions. This paper
  outlines a roadmap for advancing speech LLMs, introduces a benchmark for
  evaluation, and provides key insights into their current limitations and
  potential.
paper_authors._id: 671224b95fcebe9dcaa7b73d
paper_authors.name: Fan Bu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T09:24:58.638Z
paper_authors._id: 668e7f46c243a12604035758
paper_authors.avatarUrl: /avatars/35bd20032fafb7d7603266cf9a72d1e0.svg
paper_authors.fullname: Fan Bu
paper_authors.type: user
paper_authors.user: FanBuCUHK
paper_authors._id: 671224b95fcebe9dcaa7b73e
paper_authors.name: Yuhao Zhang
paper_authors._id: 671224b95fcebe9dcaa7b73f
paper_authors.name: Xidong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T14:07:58.649Z
paper_authors._id: 640ed3e9f2d7c41a1e9a9fde
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678693486092-640ed3e9f2d7c41a1e9a9fde.jpeg
paper_authors.fullname: Xidong Wang
paper_authors.type: user
paper_authors.user: Xidong
paper_authors._id: 671224b95fcebe9dcaa7b740
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T14:08:12.097Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 671224b95fcebe9dcaa7b741
paper_authors.name: Qun Liu
paper_authors._id: 671224b95fcebe9dcaa7b742
paper_authors.name: Haizhou Li
paper_title: Roadmap towards Superhuman Speech Understanding using Large Language
    Models
paper_summary: The success of large language models (LLMs) has prompted efforts to integrate
  speech and audio data, aiming to create general foundation models capable of
  processing both textual and non-textual inputs. Recent advances, such as
  GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves
  non-semantic information and world knowledge for deeper speech understanding.
  To guide the development of speech LLMs, we propose a five-level roadmap,
  ranging from basic automatic speech recognition (ASR) to advanced superhuman
  models capable of integrating non-semantic information with abstract acoustic
  knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark,
  that standardizes critical aspects across various tasks in these five levels,
  uncovering challenges in using abstract acoustic knowledge and completeness of
  capability. Our findings reveal gaps in handling paralinguistic cues and
  abstract acoustic knowledge, and we offer future directions. This paper
  outlines a roadmap for advancing speech LLMs, introduces a benchmark for
  evaluation, and provides key insights into their current limitations and
  potential.
paper_ai_summary: The paper outlines a roadmap and introduces a benchmark for advancing speech LLMs to integrate non-semantic information and abstract acoustic knowledge, highlighting current limitations and future directions.
paper_ai_keywords: large language models LLMs end-to-end speech LLMs automatic speech recognition ASR SAGI Benchmark paralinguistic cues abstract acoustic knowledge