EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic,
  Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
Text-to-Speech (TTS) benchmarks often fail to capture how well models handle
nuanced and semantically complex text. Building on EmergentTTS, we
introduce EmergentTTS-Eval, a comprehensive benchmark covering six
challenging TTS scenarios: emotions, paralinguistics, foreign words, syntactic
complexity, complex pronunciation (e.g. URLs, formulas), and questions.
Crucially, our framework automates both test-case generation and evaluation,
making the benchmark easily extensible. Starting from a small set of
human-written seed prompts, we iteratively extend them using LLMs to target
specific structural, phonetic and prosodic challenges, resulting in 1,645
diverse test cases. Moreover, we employ a model-as-a-judge approach, using a
Large Audio Language Model (LALM) to assess the speech across multiple
dimensions such as expressed emotion, prosodic, intonational, and pronunciation
accuracy. We evaluate state-of-the-art open-source and proprietary TTS systems,
such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval,
demonstrating its ability to reveal fine-grained performance differences.
Results show that the model-as-a-judge approach offers robust TTS assessment
and a high correlation with human preferences. We open source the evaluation
https://github.com/boson-ai/EmergentTTS-Eval-public{code} and the
https://huggingface.co/datasets/bosonai/EmergentTTS-Eval{dataset}.
683916c60df60182c0dee89d
Ruskin Raj Manku
claimed_verified
2025-06-02T07:47:12.390Z
66958c29d4ca2767b9c41005
/avatars/c81b65c1ad345c48c252773ea78b7607.svg
Ruskin Raj Manku
user
ruskinmanku
683916c60df60182c0dee89e
Yuzhi Tang
683916c60df60182c0dee89f
Xingjian Shi
683916c60df60182c0dee8a0
Mu Li
683916c60df60182c0dee8a1
Alex Smola
EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic,
  Expressiveness, and Linguistic Challenges Using Model-as-a-Judge
Text-to-Speech (TTS) benchmarks often fail to capture how well models handle
nuanced and semantically complex text. Building on EmergentTTS, we
introduce EmergentTTS-Eval, a comprehensive benchmark covering six
challenging TTS scenarios: emotions, paralinguistics, foreign words, syntactic
complexity, complex pronunciation (e.g. URLs, formulas), and questions.
Crucially, our framework automates both test-case generation and evaluation,
making the benchmark easily extensible. Starting from a small set of
human-written seed prompts, we iteratively extend them using LLMs to target
specific structural, phonetic and prosodic challenges, resulting in 1,645
diverse test cases. Moreover, we employ a model-as-a-judge approach, using a
Large Audio Language Model (LALM) to assess the speech across multiple
dimensions such as expressed emotion, prosodic, intonational, and pronunciation
accuracy. We evaluate state-of-the-art open-source and proprietary TTS systems,
such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval,
demonstrating its ability to reveal fine-grained performance differences.
Results show that the model-as-a-judge approach offers robust TTS assessment
and a high correlation with human preferences. We open source the evaluation
https://github.com/boson-ai/EmergentTTS-Eval-public{code} and the
https://huggingface.co/datasets/bosonai/EmergentTTS-Eval{dataset}.
A comprehensive TTS benchmark, EmergentTTS-Eval, automates test-case generation and evaluation using LLMs and LALM to assess nuanced and semantically complex text in speech outputs.
EmergentTTS-Eval LLMs Large Audio Language Model (LALM) expressed emotion prosodic intonational pronunciation accuracy TTS systems model-as-a-judge