title: EvalTree: Profiling Language Model Weaknesses via Hierarchical
    Capability Trees
summary: An ideal model evaluation should achieve two goals: identifying where the
  model fails and providing actionable improvement guidance. Toward these goals
  for Language Model (LM) evaluations, we formulate the problem of generating a
  weakness profile, a set of weaknesses expressed in natural language, given an
  LM's performance on every individual instance in a benchmark. We introduce a
  suite of quantitative assessments to compare different weakness profiling
  methods. We also propose a weakness profiling method EvalTree. It constructs a
  capability tree where each node represents a capability described in natural
  language and is linked to a subset of benchmark instances that specifically
  evaluate this capability; it then extracts nodes where the LM performs poorly
  to generate a weakness profile. On the MATH and WildChat benchmarks, we show
  that EvalTree outperforms baseline weakness profiling methods by identifying
  weaknesses more precisely and comprehensively. Weakness profiling further
  enables weakness-guided data collection, and training data collection guided by
  EvalTree-identified weaknesses improves LM performance more than other data
  collection strategies. We also show how EvalTree exposes flaws in Chatbot
  Arena's human-voter-based evaluation practice. To facilitate future work, we
  release our code and an interface that allows practitioners to interactively
  explore the capability trees built by EvalTree.
paper_authors._id: 67dade8168f86d8f53993ed8
paper_authors.name: Zhiyuan Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:00.793Z
paper_authors._id: 64a85e23b6512b8328f9d9e2
paper_authors.avatarUrl: /avatars/4a6b35752d3f76cb03278f52b3b43426.svg
paper_authors.fullname: Zhiyuan Zeng
paper_authors.type: user
paper_authors.user: ZhiyuanZeng
paper_authors._id: 67dade8168f86d8f53993ed9
paper_authors.name: Yizhong Wang
paper_authors._id: 67dade8168f86d8f53993eda
paper_authors.name: Hannaneh Hajishirzi
paper_authors._id: 67dade8168f86d8f53993edb
paper_authors.name: Pang Wei Koh
paper_title: EvalTree: Profiling Language Model Weaknesses via Hierarchical
    Capability Trees
paper_summary: An ideal model evaluation should achieve two goals: identifying where the
  model fails and providing actionable improvement guidance. Toward these goals
  for Language Model (LM) evaluations, we formulate the problem of generating a
  weakness profile, a set of weaknesses expressed in natural language, given an
  LM's performance on every individual instance in a benchmark. We introduce a
  suite of quantitative assessments to compare different weakness profiling
  methods. We also propose a weakness profiling method EvalTree. It constructs a
  capability tree where each node represents a capability described in natural
  language and is linked to a subset of benchmark instances that specifically
  evaluate this capability; it then extracts nodes where the LM performs poorly
  to generate a weakness profile. On the MATH and WildChat benchmarks, we show
  that EvalTree outperforms baseline weakness profiling methods by identifying
  weaknesses more precisely and comprehensively. Weakness profiling further
  enables weakness-guided data collection, and training data collection guided by
  EvalTree-identified weaknesses improves LM performance more than other data
  collection strategies. We also show how EvalTree exposes flaws in Chatbot
  Arena's human-voter-based evaluation practice. To facilitate future work, we
  release our code and an interface that allows practitioners to interactively
  explore the capability trees built by EvalTree.
paper_ai_summary: EvalTree generates precise and comprehensive weakness profiles for language models by constructing a capability tree and identifying areas of poor performance, leading to improved data collection and model performance.
paper_ai_keywords: Language Model weakness profile capability tree MATH benchmark WildChat benchmark Chatbot Arena human-voter-based evaluation