title: EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic,
    Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
summary: Text-to-Speech (TTS) benchmarks often fail to capture how well models handle
  nuanced and semantically complex text. Building on EmergentTTS, we
  introduce EmergentTTS-Eval, a comprehensive benchmark covering six
  challenging TTS scenarios: emotions, paralinguistics, foreign words, syntactic
  complexity, complex pronunciation (e.g. URLs, formulas), and questions.
  Crucially, our framework automates both test-case generation and evaluation,
  making the benchmark easily extensible. Starting from a small set of
  human-written seed prompts, we iteratively extend them using LLMs to target
  specific structural, phonetic and prosodic challenges, resulting in 1,645
  diverse test cases. Moreover, we employ a model-as-a-judge approach, using a
  Large Audio Language Model (LALM) to assess the speech across multiple
  dimensions such as expressed emotion, prosodic, intonational, and pronunciation
  accuracy. We evaluate state-of-the-art open-source and proprietary TTS systems,
  such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval,
  demonstrating its ability to reveal fine-grained performance differences.
  Results show that the model-as-a-judge approach offers robust TTS assessment
  and a high correlation with human preferences. We open source the evaluation
  https://github.com/boson-ai/EmergentTTS-Eval-public{code} and the
  https://huggingface.co/datasets/bosonai/EmergentTTS-Eval{dataset}.
paper_authors._id: 683916c60df60182c0dee89d
paper_authors.name: Ruskin Raj Manku
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:12.390Z
paper_authors._id: 66958c29d4ca2767b9c41005
paper_authors.avatarUrl: /avatars/c81b65c1ad345c48c252773ea78b7607.svg
paper_authors.fullname: Ruskin Raj Manku
paper_authors.type: user
paper_authors.user: ruskinmanku
paper_authors._id: 683916c60df60182c0dee89e
paper_authors.name: Yuzhi Tang
paper_authors._id: 683916c60df60182c0dee89f
paper_authors.name: Xingjian Shi
paper_authors._id: 683916c60df60182c0dee8a0
paper_authors.name: Mu Li
paper_authors._id: 683916c60df60182c0dee8a1
paper_authors.name: Alex Smola
paper_title: EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic,
    Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
paper_summary: Text-to-Speech (TTS) benchmarks often fail to capture how well models handle
  nuanced and semantically complex text. Building on EmergentTTS, we
  introduce EmergentTTS-Eval, a comprehensive benchmark covering six
  challenging TTS scenarios: emotions, paralinguistics, foreign words, syntactic
  complexity, complex pronunciation (e.g. URLs, formulas), and questions.
  Crucially, our framework automates both test-case generation and evaluation,
  making the benchmark easily extensible. Starting from a small set of
  human-written seed prompts, we iteratively extend them using LLMs to target
  specific structural, phonetic and prosodic challenges, resulting in 1,645
  diverse test cases. Moreover, we employ a model-as-a-judge approach, using a
  Large Audio Language Model (LALM) to assess the speech across multiple
  dimensions such as expressed emotion, prosodic, intonational, and pronunciation
  accuracy. We evaluate state-of-the-art open-source and proprietary TTS systems,
  such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval,
  demonstrating its ability to reveal fine-grained performance differences.
  Results show that the model-as-a-judge approach offers robust TTS assessment
  and a high correlation with human preferences. We open source the evaluation
  https://github.com/boson-ai/EmergentTTS-Eval-public{code} and the
  https://huggingface.co/datasets/bosonai/EmergentTTS-Eval{dataset}.
paper_ai_summary: A comprehensive TTS benchmark, EmergentTTS-Eval, automates test-case generation and evaluation using LLMs and LALM to assess nuanced and semantically complex text in speech outputs.
paper_ai_keywords: EmergentTTS-Eval LLMs Large Audio Language Model (LALM) expressed emotion prosodic intonational pronunciation accuracy TTS systems model-as-a-judge