CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
The Automated Audio Captioning (AAC) task asks models to generate natural
language descriptions of an audio input. Evaluating these machine-generated
audio captions is a complex task that requires considering diverse factors,
among them, auditory scene understanding, sound-object inference, temporal
coherence, and the environmental context of the scene. While current methods
focus on specific aspects, they often fail to provide an overall score that
aligns well with human judgment. In this work, we propose CLAIR-A, a simple and
flexible method that leverages the zero-shot capabilities of large language
models (LLMs) to evaluate candidate audio captions by directly asking LLMs for
a semantic distance score. In our evaluations, CLAIR-A better predicts human
judgements of quality compared to traditional metrics, with a 5.8% relative
accuracy improvement compared to the domain-specific FENSE metric and up to 11%
over the best general-purpose measure on the Clotho-Eval dataset. Moreover,
CLAIR-A offers more transparency by allowing the language model to explain the
reasoning behind its scores, with these explanations rated up to 30% better by
human evaluators than those provided by baseline methods. CLAIR-A is made
publicly available at https://github.com/DavidMChan/clair-a.
66ed982d518ccb4e628a365a
Tsung-Han Wu
claimed_verified
2025-04-18T09:42:06.142Z
644a767044b75fd95805d232
https://cdn-avatars.huggingface.co/v1/production/uploads/644a767044b75fd95805d232/vHA2vI_B3CpXapdBEwspB.jpeg
Patrick (Tsung-Han) Wu
user
tsunghanwu
66ed982d518ccb4e628a365b
Joseph E. Gonzalez
66ed982d518ccb4e628a365c
Trevor Darrell
66ed982d518ccb4e628a365d
David M. Chan
CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
The Automated Audio Captioning (AAC) task asks models to generate natural
language descriptions of an audio input. Evaluating these machine-generated
audio captions is a complex task that requires considering diverse factors,
among them, auditory scene understanding, sound-object inference, temporal
coherence, and the environmental context of the scene. While current methods
focus on specific aspects, they often fail to provide an overall score that
aligns well with human judgment. In this work, we propose CLAIR-A, a simple and
flexible method that leverages the zero-shot capabilities of large language
models (LLMs) to evaluate candidate audio captions by directly asking LLMs for
a semantic distance score. In our evaluations, CLAIR-A better predicts human
judgements of quality compared to traditional metrics, with a 5.8% relative
accuracy improvement compared to the domain-specific FENSE metric and up to 11%
over the best general-purpose measure on the Clotho-Eval dataset. Moreover,
CLAIR-A offers more transparency by allowing the language model to explain the
reasoning behind its scores, with these explanations rated up to 30% better by
human evaluators than those provided by baseline methods. CLAIR-A is made
publicly available at https://github.com/DavidMChan/clair-a.
CLAIR-A uses large language models to evaluate audio captions by providing semantic distance scores, outperforming traditional metrics and offering Explainable AI benefits.
large language models zero-shot capabilities semantic distance score Explainable AI