HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language
  Models on cross-domain multi-file project problems
Evaluating the real-world applicability of large language models (LLMs)
provides valuable insights for their development and use in software
development tasks. Existing benchmarks often focus on standalone coding
problems or specific libraries, overlooking multi-file, project-based scenarios
and lacking a rigorous evaluation of consistency. The HackerRank-ASTRA
Benchmark introduces project-based coding problems that mirror real-world
scenarios. It evaluates model consistency through 32 runs (k = 32) and median
standard deviation while incorporating taxonomy-level analysis to assess
sub-skill capabilities. Initial evaluations on 65 problems show that the top
three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved
comparable average scores of 75%, with no statistically significant differences
in performance. Notably, Claude-3.5-Sonnet-1022 demonstrated the highest
consistency across problems, with low variability (SD = 0.0497), which was
statistically significant compared to other models, highlighting its
reliability for real-world software development tasks.
67a3d37e2d9a08978848c657
Jun Xing
claimed_verified
2025-02-06T14:15:09.857Z
63eff09f4a788ed1dd863b09
/avatars/b557d83cf6d6b3b46dfbe9b7727ae16d.svg
Jun
user
oldteacherjoy
67a3d37e2d9a08978848c658
Mayur Bhatia
67a3d37e2d9a08978848c659
Sahil Phulwani
67a3d37e2d9a08978848c65a
Darshan Suresh
67a3d37e2d9a08978848c65b
Rafik Matta
HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language
  Models on cross-domain multi-file project problems
Evaluating the real-world applicability of large language models (LLMs)
provides valuable insights for their development and use in software
development tasks. Existing benchmarks often focus on standalone coding
problems or specific libraries, overlooking multi-file, project-based scenarios
and lacking a rigorous evaluation of consistency. The HackerRank-ASTRA
Benchmark introduces project-based coding problems that mirror real-world
scenarios. It evaluates model consistency through 32 runs (k = 32) and median
standard deviation while incorporating taxonomy-level analysis to assess
sub-skill capabilities. Initial evaluations on 65 problems show that the top
three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved
comparable average scores of 75%, with no statistically significant differences
in performance. Notably, Claude-3.5-Sonnet-1022 demonstrated the highest
consistency across problems, with low variability (SD = 0.0497), which was
statistically significant compared to other models, highlighting its
reliability for real-world software development tasks.
The HackerRank-ASTRA Benchmark evaluates large language models for real-world software development by assessing project-based coding problems, consistency, and sub-skill capabilities.
large language models LLMs HackerRank-ASTRA Benchmark project-based coding problems consistency median standard deviation taxonomy-level analysis sub-skill capabilities statistical significance reliability