title: Report Cards: Qualitative Evaluation of Language Models Using Natural
    Language Summaries
summary: The rapid development and dynamic nature of large language models (LLMs) make
  it difficult for conventional quantitative benchmarks to accurately assess
  their capabilities. We propose report cards, which are human-interpretable,
  natural language summaries of model behavior for specific skills or topics. We
  develop a framework to evaluate report cards based on three criteria:
  specificity (ability to distinguish between models), faithfulness (accurate
  representation of model capabilities), and interpretability (clarity and
  relevance to humans). We also propose an iterative algorithm for generating
  report cards without human supervision and explore its efficacy by ablating
  various design choices. Through experimentation with popular LLMs, we
  demonstrate that report cards provide insights beyond traditional benchmarks
  and can help address the need for a more interpretable and holistic evaluation
  of LLMs.
paper_authors._id: 66d8b48d8c67ac5bdf0056d8
paper_authors.name: Blair Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:24:13.084Z
paper_authors._id: 64dcdecfe0a1a37596576e46
paper_authors.avatarUrl: /avatars/3e57d706108d92a4b3c5ee4e2b20398c.svg
paper_authors.fullname: Blair Yang
paper_authors.type: user
paper_authors.user: loveblairsky
paper_authors._id: 66d8b48d8c67ac5bdf0056d9
paper_authors.name: Fuyang Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T16:16:38.565Z
paper_authors._id: 66db0da2947b2e8abc3f93dd
paper_authors.avatarUrl: /avatars/9a8cdc10071329ce178c05380b90cc17.svg
paper_authors.fullname: Fuyang Cui
paper_authors.type: user
paper_authors.user: scottcfy
paper_authors._id: 66d8b48d8c67ac5bdf0056da
paper_authors.name: Keiran Paster
paper_authors._id: 66d8b48d8c67ac5bdf0056db
paper_authors.name: Jimmy Ba
paper_authors._id: 66d8b48d8c67ac5bdf0056dc
paper_authors.name: Pashootan Vaezipoor
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T13:34:13.198Z
paper_authors._id: 647630d4288784ce9ab9c2e3
paper_authors.avatarUrl: /avatars/7ff4a595a2a2e4fccd8319a8cd8f236f.svg
paper_authors.fullname: Pashootan V
paper_authors.type: user
paper_authors.user: PashaV
paper_authors._id: 66d8b48d8c67ac5bdf0056dd
paper_authors.name: Silviu Pitis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T09:54:02.283Z
paper_authors._id: 62d1cf75c40bde8b986e1c84
paper_authors.avatarUrl: /avatars/4cf2b2557c37e65662250fc83fc83ab0.svg
paper_authors.fullname: Silviu Pitis
paper_authors.type: user
paper_authors.user: spitis
paper_authors._id: 66d8b48d8c67ac5bdf0056de
paper_authors.name: Michael R. Zhang
paper_title: Report Cards: Qualitative Evaluation of Language Models Using Natural
    Language Summaries
paper_summary: The rapid development and dynamic nature of large language models (LLMs) make
  it difficult for conventional quantitative benchmarks to accurately assess
  their capabilities. We propose report cards, which are human-interpretable,
  natural language summaries of model behavior for specific skills or topics. We
  develop a framework to evaluate report cards based on three criteria:
  specificity (ability to distinguish between models), faithfulness (accurate
  representation of model capabilities), and interpretability (clarity and
  relevance to humans). We also propose an iterative algorithm for generating
  report cards without human supervision and explore its efficacy by ablating
  various design choices. Through experimentation with popular LLMs, we
  demonstrate that report cards provide insights beyond traditional benchmarks
  and can help address the need for a more interpretable and holistic evaluation
  of LLMs.
paper_ai_keywords: large language models (LLMs) report cards human-interpretable natural language summaries model behavior specificity faithfulness interpretability iterative algorithm ablation