Audio-Aware Large Language Models as Judges for Speaking Styles
Audio-aware large language models (ALLMs) can understand the textual and
non-textual information in the audio input. In this paper, we explore using
ALLMs as an automatic judge to assess the speaking styles of speeches. We use
ALLM judges to evaluate the speeches generated by SLMs on two tasks: voice
style instruction following and role-playing. The speaking style we consider
includes emotion, volume, speaking pace, word emphasis, pitch control, and
non-verbal elements. We use four spoken language models (SLMs) to complete the
two tasks and use humans and ALLMs to judge the SLMs' responses. We compare two
ALLM judges, GPT-4o-audio and Gemini-2.5-pro, with human evaluation results and
show that the agreement between Gemini and human judges is comparable to the
agreement between human evaluators. These promising results show that ALLMs can
be used as a judge to evaluate SLMs. Our results also reveal that current SLMs,
even GPT-4o-audio, still have room for improvement in controlling the speaking
style and generating natural dialogues.
68463ee43ec10bdd8ab4da6f
Cheng-Han Chiang
claimed_verified
2025-06-09T10:11:26.041Z
622326ae0129f2097d69a3e2
/avatars/7665223e3fc8b820ce001e6003daf4d2.svg
Cheng-Han Chiang
user
dcml0714
68463ee43ec10bdd8ab4da70
Xiaofei Wang
extracted_confirmed
2025-07-22T03:32:18.975Z
64dc191bc307ee5369fbcb04
/avatars/5a8a0db63a187e85d4ae2fff93a838f0.svg
Xiaofei Wang
user
xiaofei-wang
68463ee43ec10bdd8ab4da71
Chung-Ching Lin
claimed_verified
2025-07-23T08:37:38.901Z
6669f406d21cf7d39b1d98ba
/avatars/5ec270cb8bcb68786379d8a8bc411aaa.svg
CC Lin
user
cclin10
68463ee43ec10bdd8ab4da72
Kevin Lin
68463ee43ec10bdd8ab4da73
Linjie Li
68463ee43ec10bdd8ab4da74
Radu Kopetz
68463ee43ec10bdd8ab4da75
Yao Qian
68463ee43ec10bdd8ab4da76
Zhendong Wang
68463ee43ec10bdd8ab4da77
Zhengyuan Yang
68463ee43ec10bdd8ab4da78
Hung-yi Lee
68463ee43ec10bdd8ab4da79
Lijuan Wang
Audio-Aware Large Language Models as Judges for Speaking Styles
Audio-aware large language models (ALLMs) can understand the textual and
non-textual information in the audio input. In this paper, we explore using
ALLMs as an automatic judge to assess the speaking styles of speeches. We use
ALLM judges to evaluate the speeches generated by SLMs on two tasks: voice
style instruction following and role-playing. The speaking style we consider
includes emotion, volume, speaking pace, word emphasis, pitch control, and
non-verbal elements. We use four spoken language models (SLMs) to complete the
two tasks and use humans and ALLMs to judge the SLMs' responses. We compare two
ALLM judges, GPT-4o-audio and Gemini-2.5-pro, with human evaluation results and
show that the agreement between Gemini and human judges is comparable to the
agreement between human evaluators. These promising results show that ALLMs can
be used as a judge to evaluate SLMs. Our results also reveal that current SLMs,
even GPT-4o-audio, still have room for improvement in controlling the speaking
style and generating natural dialogues.
Audio-aware large language models can assess speaking styles in audio inputs, demonstrating performance comparable to human judges in evaluating synthesized speech along dimensions like emotion, volume, and pitch.
audio-aware large language models ALLMs speaking styles SLMs voice style instruction role-playing emotion volume speaking pace word emphasis pitch control non-verbal elements GPT-4o-audio Gemini-2.5-pro human evaluation agreement speaking style control natural dialogues