BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM
  Evaluation
As large language models (LLMs) continue to advance, the need for up-to-date
and well-organized benchmarks becomes increasingly critical. However, many
existing datasets are scattered, difficult to manage, and make it challenging
to perform evaluations tailored to specific needs or domains, despite the
growing importance of domain-specific models in areas such as math or code. In
this paper, we introduce BenchHub, a dynamic benchmark repository that empowers
researchers and developers to evaluate LLMs more effectively. BenchHub
aggregates and automatically classifies benchmark datasets from diverse
domains, integrating 303K questions across 38 benchmarks. It is designed to
support continuous updates and scalable data management, enabling flexible and
customizable evaluation tailored to various domains or use cases. Through
extensive experiments with various LLM families, we demonstrate that model
performance varies significantly across domain-specific subsets, emphasizing
the importance of domain-aware benchmarking. We believe BenchHub can encourage
better dataset reuse, more transparent model comparisons, and easier
identification of underrepresented areas in existing benchmarks, offering a
critical infrastructure for advancing LLM evaluation research.
68402986a50b67f983749710
Eunsu Kim
claimed_verified
2025-06-05T07:27:49.056Z
6576ace7769f3ee9bd7b1b88
/avatars/5b5921e54413a37afde6ce017809c86e.svg
Eunsu Kim
user
EunsuKim
68402986a50b67f983749711
Haneul Yoo
68402986a50b67f983749712
Guijin Son
68402986a50b67f983749713
Hitesh Patel
68402986a50b67f983749714
Amit Agarwal
claimed_verified
2025-06-07T05:49:26.835Z
6321a50a6b1992383fa00db1
/avatars/92311b4a5e8bcc8f5477747621334fc8.svg
Amit Agarwal
user
amitbcp
68402986a50b67f983749715
Alice Oh
extracted_confirmed
2025-06-05T06:37:21.889Z
60e0251ea9b5d8282481f2b7
/avatars/43441373af054a6184c22097bfeb97e4.svg
Alice Oh
user
aliceoh
BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM
  Evaluation
As large language models (LLMs) continue to advance, the need for up-to-date
and well-organized benchmarks becomes increasingly critical. However, many
existing datasets are scattered, difficult to manage, and make it challenging
to perform evaluations tailored to specific needs or domains, despite the
growing importance of domain-specific models in areas such as math or code. In
this paper, we introduce BenchHub, a dynamic benchmark repository that empowers
researchers and developers to evaluate LLMs more effectively. BenchHub
aggregates and automatically classifies benchmark datasets from diverse
domains, integrating 303K questions across 38 benchmarks. It is designed to
support continuous updates and scalable data management, enabling flexible and
customizable evaluation tailored to various domains or use cases. Through
extensive experiments with various LLM families, we demonstrate that model
performance varies significantly across domain-specific subsets, emphasizing
the importance of domain-aware benchmarking. We believe BenchHub can encourage
better dataset reuse, more transparent model comparisons, and easier
identification of underrepresented areas in existing benchmarks, offering a
critical infrastructure for advancing LLM evaluation research.
BenchHub is a dynamic benchmark repository that aggregates and classifies datasets for large language models, facilitating domain-specific evaluations and improving model comparisons.
large language models LLMs BenchHub benchmark repository domain-specific models benchmark datasets continuous updates scalable data management model performance domain-aware benchmarking