title: Mind the Gap! Static and Interactive Evaluations of Large Audio Models
summary: As AI chatbots become ubiquitous, voice interaction presents a compelling way
  to enable rapid, high-bandwidth communication for both semantic and social
  signals. This has driven research into Large Audio Models (LAMs) to power
  voice-native experiences. However, aligning LAM development with user goals
  requires a clear understanding of user needs and preferences to establish
  reliable progress metrics. This study addresses these challenges by introducing
  an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions
  from 484 participants. Through topic modeling of user queries, we identify
  primary use cases for audio interfaces. We then analyze user preference
  rankings and qualitative feedback to determine which models best align with
  user needs. Finally, we evaluate how static benchmarks predict interactive
  performance - our analysis reveals no individual benchmark strongly correlates
  with interactive results (tau leq 0.33 for all benchmarks). While combining
  multiple coarse-grained features yields modest predictive power (R^2=0.30),
  only two out of twenty datasets on spoken question answering and age prediction
  show significantly positive correlations. This suggests a clear need to develop
  LAM evaluations that better correlate with user preferences.
paper_authors._id: 67be33ffe30b2f126c599413
paper_authors.name: Minzhi Li
paper_authors._id: 67be33ffe30b2f126c599414
paper_authors.name: William Barr Held
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:28:38.756Z
paper_authors._id: 632116accafe12f481a473cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666207676653-632116accafe12f481a473cb.jpeg
paper_authors.fullname: Will Held
paper_authors.type: user
paper_authors.user: WillHeld
paper_authors._id: 67be33ffe30b2f126c599415
paper_authors.name: Michael J Ryan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:52:35.827Z
paper_authors._id: 63878fa2e40346f68ede7fc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63878fa2e40346f68ede7fc4/MOdQbApvhwpVzPo5FWLVV.jpeg
paper_authors.fullname: Michael Ryan
paper_authors.type: user
paper_authors.user: MichaelR207
paper_authors._id: 67be33ffe30b2f126c599416
paper_authors.name: Kunat Pipatanakul
paper_authors._id: 67be33ffe30b2f126c599417
paper_authors.name: Potsawee Manakul
paper_authors._id: 67be33ffe30b2f126c599418
paper_authors.name: Hao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T16:07:29.874Z
paper_authors._id: 61aa376688c20eebf1e8deb3
paper_authors.avatarUrl: /avatars/7c11dcb232c73547d7d87834be287822.svg
paper_authors.fullname: Hao Zhu
paper_authors.type: user
paper_authors.user: ProKil
paper_authors._id: 67be33ffe30b2f126c599419
paper_authors.name: Diyi Yang
paper_title: Mind the Gap! Static and Interactive Evaluations of Large Audio Models
paper_summary: As AI chatbots become ubiquitous, voice interaction presents a compelling way
  to enable rapid, high-bandwidth communication for both semantic and social
  signals. This has driven research into Large Audio Models (LAMs) to power
  voice-native experiences. However, aligning LAM development with user goals
  requires a clear understanding of user needs and preferences to establish
  reliable progress metrics. This study addresses these challenges by introducing
  an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions
  from 484 participants. Through topic modeling of user queries, we identify
  primary use cases for audio interfaces. We then analyze user preference
  rankings and qualitative feedback to determine which models best align with
  user needs. Finally, we evaluate how static benchmarks predict interactive
  performance - our analysis reveals no individual benchmark strongly correlates
  with interactive results (tau leq 0.33 for all benchmarks). While combining
  multiple coarse-grained features yields modest predictive power (R^2=0.30),
  only two out of twenty datasets on spoken question answering and age prediction
  show significantly positive correlations. This suggests a clear need to develop
  LAM evaluations that better correlate with user preferences.
paper_ai_keywords: Large Audio Models (LAMs) voice-native experiences topic modeling user queries user preference rankings qualitative feedback interactive approach static benchmarks interactive performance spoken question answering age prediction