title: CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
summary: The Automated Audio Captioning (AAC) task asks models to generate natural
  language descriptions of an audio input. Evaluating these machine-generated
  audio captions is a complex task that requires considering diverse factors,
  among them, auditory scene understanding, sound-object inference, temporal
  coherence, and the environmental context of the scene. While current methods
  focus on specific aspects, they often fail to provide an overall score that
  aligns well with human judgment. In this work, we propose CLAIR-A, a simple and
  flexible method that leverages the zero-shot capabilities of large language
  models (LLMs) to evaluate candidate audio captions by directly asking LLMs for
  a semantic distance score. In our evaluations, CLAIR-A better predicts human
  judgements of quality compared to traditional metrics, with a 5.8% relative
  accuracy improvement compared to the domain-specific FENSE metric and up to 11%
  over the best general-purpose measure on the Clotho-Eval dataset. Moreover,
  CLAIR-A offers more transparency by allowing the language model to explain the
  reasoning behind its scores, with these explanations rated up to 30% better by
  human evaluators than those provided by baseline methods. CLAIR-A is made
  publicly available at https://github.com/DavidMChan/clair-a.
paper_authors._id: 66ed982d518ccb4e628a365a
paper_authors.name: Tsung-Han Wu
paper_authors._id: 66ed982d518ccb4e628a365b
paper_authors.name: Joseph E. Gonzalez
paper_authors._id: 66ed982d518ccb4e628a365c
paper_authors.name: Trevor Darrell
paper_authors._id: 66ed982d518ccb4e628a365d
paper_authors.name: David M. Chan
paper_title: CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
paper_summary: The Automated Audio Captioning (AAC) task asks models to generate natural
  language descriptions of an audio input. Evaluating these machine-generated
  audio captions is a complex task that requires considering diverse factors,
  among them, auditory scene understanding, sound-object inference, temporal
  coherence, and the environmental context of the scene. While current methods
  focus on specific aspects, they often fail to provide an overall score that
  aligns well with human judgment. In this work, we propose CLAIR-A, a simple and
  flexible method that leverages the zero-shot capabilities of large language
  models (LLMs) to evaluate candidate audio captions by directly asking LLMs for
  a semantic distance score. In our evaluations, CLAIR-A better predicts human
  judgements of quality compared to traditional metrics, with a 5.8% relative
  accuracy improvement compared to the domain-specific FENSE metric and up to 11%
  over the best general-purpose measure on the Clotho-Eval dataset. Moreover,
  CLAIR-A offers more transparency by allowing the language model to explain the
  reasoning behind its scores, with these explanations rated up to 30% better by
  human evaluators than those provided by baseline methods. CLAIR-A is made
  publicly available at https://github.com/DavidMChan/clair-a.
paper_ai_keywords: auditory scene understanding sound-object inference temporal coherence environmental context zero-shot capabilities large language models (LLMs) semantic distance score FENSE metric Clotho-Eval dataset human evaluators