Roadmap towards Superhuman Speech Understanding using Large Language
  Models
The success of large language models (LLMs) has prompted efforts to integrate
speech and audio data, aiming to create general foundation models capable of
processing both textual and non-textual inputs. Recent advances, such as
GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves
non-semantic information and world knowledge for deeper speech understanding.
To guide the development of speech LLMs, we propose a five-level roadmap,
ranging from basic automatic speech recognition (ASR) to advanced superhuman
models capable of integrating non-semantic information with abstract acoustic
knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark,
that standardizes critical aspects across various tasks in these five levels,
uncovering challenges in using abstract acoustic knowledge and completeness of
capability. Our findings reveal gaps in handling paralinguistic cues and
abstract acoustic knowledge, and we offer future directions. This paper
outlines a roadmap for advancing speech LLMs, introduces a benchmark for
evaluation, and provides key insights into their current limitations and
potential.
671224b95fcebe9dcaa7b73d
Fan Bu
claimed_verified
2024-10-18T09:24:58.638Z
668e7f46c243a12604035758
/avatars/35bd20032fafb7d7603266cf9a72d1e0.svg
Fan Bu
user
FanBuCUHK
671224b95fcebe9dcaa7b73e
Yuhao Zhang
671224b95fcebe9dcaa7b73f
Xidong Wang
admin_assigned
2024-10-22T14:07:58.649Z
640ed3e9f2d7c41a1e9a9fde
https://cdn-avatars.huggingface.co/v1/production/uploads/1678693486092-640ed3e9f2d7c41a1e9a9fde.jpeg
Xidong Wang
user
Xidong
671224b95fcebe9dcaa7b740
Benyou Wang
admin_assigned
2024-10-22T14:08:12.097Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
671224b95fcebe9dcaa7b741
Qun Liu
671224b95fcebe9dcaa7b742
Haizhou Li
Roadmap towards Superhuman Speech Understanding using Large Language
  Models
The success of large language models (LLMs) has prompted efforts to integrate
speech and audio data, aiming to create general foundation models capable of
processing both textual and non-textual inputs. Recent advances, such as
GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves
non-semantic information and world knowledge for deeper speech understanding.
To guide the development of speech LLMs, we propose a five-level roadmap,
ranging from basic automatic speech recognition (ASR) to advanced superhuman
models capable of integrating non-semantic information with abstract acoustic
knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark,
that standardizes critical aspects across various tasks in these five levels,
uncovering challenges in using abstract acoustic knowledge and completeness of
capability. Our findings reveal gaps in handling paralinguistic cues and
abstract acoustic knowledge, and we offer future directions. This paper
outlines a roadmap for advancing speech LLMs, introduces a benchmark for
evaluation, and provides key insights into their current limitations and
potential.
The paper outlines a roadmap and introduces a benchmark for advancing speech LLMs to integrate non-semantic information and abstract acoustic knowledge, highlighting current limitations and future directions.
large language models LLMs end-to-end speech LLMs automatic speech recognition ASR SAGI Benchmark paralinguistic cues abstract acoustic knowledge