title: S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following
    with Paralinguistic Information
summary: The rapid development of large language models (LLMs) has brought significant
  attention to speech models, particularly recent progress in speech2speech
  protocols supporting speech input and output. However, the existing benchmarks
  adopt automatic text-based evaluators for evaluating the instruction following
  ability of these models lack consideration for paralinguistic information in
  both speech understanding and generation. To address these issues, we introduce
  S2S-Arena, a novel arena-style S2S benchmark that evaluates
  instruction-following capabilities with paralinguistic information in both
  speech-in and speech-out across real-world tasks. We design 154 samples that
  fused TTS and live recordings in four domains with 21 tasks and manually
  evaluate existing popular speech models in an arena-style manner. The
  experimental results show that: (1) in addition to the superior performance of
  GPT-4o, the speech model of cascaded ASR, LLM, and TTS outperforms the jointly
  trained model after text-speech alignment in speech2speech protocols; (2)
  considering paralinguistic information, the knowledgeability of the speech
  model mainly depends on the LLM backbone, and the multilingual support of that
  is limited by the speech module; (3) excellent speech models can already
  understand the paralinguistic information in speech input, but generating
  appropriate audio with paralinguistic information is still a challenge.
paper_authors._id: 67ced88fa0db83a841e09be6
paper_authors.name: Feng Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T12:46:04.417Z
paper_authors._id: 62d4cdbdb9e2ec814c0e0659
paper_authors.avatarUrl: /avatars/616dd89be0fe8937b32c3f23c69a4e15.svg
paper_authors.fullname: feng jiang
paper_authors.type: user
paper_authors.user: liuxuan320
paper_authors._id: 67ced88fa0db83a841e09be7
paper_authors.name: Zhiyu Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T13:35:13.930Z
paper_authors._id: 67061847e343e345b777d574
paper_authors.avatarUrl: /avatars/6b6f35aa563dcf32937c4b6a3cdd870e.svg
paper_authors.fullname: Zhiyu
paper_authors.type: user
paper_authors.user: zylin1
paper_authors._id: 67ced88fa0db83a841e09be8
paper_authors.name: Fan Bu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T13:35:16.297Z
paper_authors._id: 668e7f46c243a12604035758
paper_authors.avatarUrl: /avatars/35bd20032fafb7d7603266cf9a72d1e0.svg
paper_authors.fullname: Fan Bu
paper_authors.type: user
paper_authors.user: FanBuCUHK
paper_authors._id: 67ced88fa0db83a841e09be9
paper_authors.name: Yuhao Du
paper_authors._id: 67ced88fa0db83a841e09bea
paper_authors.name: Benyou Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T13:35:11.668Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 67ced88fa0db83a841e09beb
paper_authors.name: Haizhou Li
paper_title: S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following
    with Paralinguistic Information
paper_summary: The rapid development of large language models (LLMs) has brought significant
  attention to speech models, particularly recent progress in speech2speech
  protocols supporting speech input and output. However, the existing benchmarks
  adopt automatic text-based evaluators for evaluating the instruction following
  ability of these models lack consideration for paralinguistic information in
  both speech understanding and generation. To address these issues, we introduce
  S2S-Arena, a novel arena-style S2S benchmark that evaluates
  instruction-following capabilities with paralinguistic information in both
  speech-in and speech-out across real-world tasks. We design 154 samples that
  fused TTS and live recordings in four domains with 21 tasks and manually
  evaluate existing popular speech models in an arena-style manner. The
  experimental results show that: (1) in addition to the superior performance of
  GPT-4o, the speech model of cascaded ASR, LLM, and TTS outperforms the jointly
  trained model after text-speech alignment in speech2speech protocols; (2)
  considering paralinguistic information, the knowledgeability of the speech
  model mainly depends on the LLM backbone, and the multilingual support of that
  is limited by the speech module; (3) excellent speech models can already
  understand the paralinguistic information in speech input, but generating
  appropriate audio with paralinguistic information is still a challenge.
paper_ai_summary: S2S-Arena is introduced to evaluate speech models' instruction-following abilities with paralinguistic information, revealing that cascaded ASR, LLM, and TTS outperform jointly trained models in speech2speech protocols and that generating appropriate audio with paralinguistic information remains challenging.
paper_ai_keywords: large language models LLMs speech2speech S2S-Arena paralinguistic information speech understanding speech generation TTS ASR task evaluation speech model knowledgeability multilingual support