title: BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM
    Evaluation
summary: As large language models (LLMs) continue to advance, the need for up-to-date
  and well-organized benchmarks becomes increasingly critical. However, many
  existing datasets are scattered, difficult to manage, and make it challenging
  to perform evaluations tailored to specific needs or domains, despite the
  growing importance of domain-specific models in areas such as math or code. In
  this paper, we introduce BenchHub, a dynamic benchmark repository that empowers
  researchers and developers to evaluate LLMs more effectively. BenchHub
  aggregates and automatically classifies benchmark datasets from diverse
  domains, integrating 303K questions across 38 benchmarks. It is designed to
  support continuous updates and scalable data management, enabling flexible and
  customizable evaluation tailored to various domains or use cases. Through
  extensive experiments with various LLM families, we demonstrate that model
  performance varies significantly across domain-specific subsets, emphasizing
  the importance of domain-aware benchmarking. We believe BenchHub can encourage
  better dataset reuse, more transparent model comparisons, and easier
  identification of underrepresented areas in existing benchmarks, offering a
  critical infrastructure for advancing LLM evaluation research.
paper_authors._id: 68402986a50b67f983749710
paper_authors.name: Eunsu Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:27:49.056Z
paper_authors._id: 6576ace7769f3ee9bd7b1b88
paper_authors.avatarUrl: /avatars/5b5921e54413a37afde6ce017809c86e.svg
paper_authors.fullname: Eunsu Kim
paper_authors.type: user
paper_authors.user: EunsuKim
paper_authors._id: 68402986a50b67f983749711
paper_authors.name: Haneul Yoo
paper_authors._id: 68402986a50b67f983749712
paper_authors.name: Guijin Son
paper_authors._id: 68402986a50b67f983749713
paper_authors.name: Hitesh Patel
paper_authors._id: 68402986a50b67f983749714
paper_authors.name: Amit Agarwal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:26.835Z
paper_authors._id: 6321a50a6b1992383fa00db1
paper_authors.avatarUrl: /avatars/92311b4a5e8bcc8f5477747621334fc8.svg
paper_authors.fullname: Amit Agarwal
paper_authors.type: user
paper_authors.user: amitbcp
paper_authors._id: 68402986a50b67f983749715
paper_authors.name: Alice Oh
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-05T06:37:21.889Z
paper_authors._id: 60e0251ea9b5d8282481f2b7
paper_authors.avatarUrl: /avatars/43441373af054a6184c22097bfeb97e4.svg
paper_authors.fullname: Alice Oh
paper_authors.type: user
paper_authors.user: aliceoh
paper_title: BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM
    Evaluation
paper_summary: As large language models (LLMs) continue to advance, the need for up-to-date
  and well-organized benchmarks becomes increasingly critical. However, many
  existing datasets are scattered, difficult to manage, and make it challenging
  to perform evaluations tailored to specific needs or domains, despite the
  growing importance of domain-specific models in areas such as math or code. In
  this paper, we introduce BenchHub, a dynamic benchmark repository that empowers
  researchers and developers to evaluate LLMs more effectively. BenchHub
  aggregates and automatically classifies benchmark datasets from diverse
  domains, integrating 303K questions across 38 benchmarks. It is designed to
  support continuous updates and scalable data management, enabling flexible and
  customizable evaluation tailored to various domains or use cases. Through
  extensive experiments with various LLM families, we demonstrate that model
  performance varies significantly across domain-specific subsets, emphasizing
  the importance of domain-aware benchmarking. We believe BenchHub can encourage
  better dataset reuse, more transparent model comparisons, and easier
  identification of underrepresented areas in existing benchmarks, offering a
  critical infrastructure for advancing LLM evaluation research.
paper_ai_summary: BenchHub is a dynamic benchmark repository that aggregates and classifies datasets for large language models, facilitating domain-specific evaluations and improving model comparisons.
paper_ai_keywords: large language models LLMs BenchHub benchmark repository domain-specific models benchmark datasets continuous updates scalable data management model performance domain-aware benchmarking