title: HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language
    Models on cross-domain multi-file project problems
summary: Evaluating the real-world applicability of large language models (LLMs)
  provides valuable insights for their development and use in software
  development tasks. Existing benchmarks often focus on standalone coding
  problems or specific libraries, overlooking multi-file, project-based scenarios
  and lacking a rigorous evaluation of consistency. The HackerRank-ASTRA
  Benchmark introduces project-based coding problems that mirror real-world
  scenarios. It evaluates model consistency through 32 runs (k = 32) and median
  standard deviation while incorporating taxonomy-level analysis to assess
  sub-skill capabilities. Initial evaluations on 65 problems show that the top
  three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved
  comparable average scores of 75%, with no statistically significant differences
  in performance. Notably, Claude-3.5-Sonnet-1022 demonstrated the highest
  consistency across problems, with low variability (SD = 0.0497), which was
  statistically significant compared to other models, highlighting its
  reliability for real-world software development tasks.
paper_authors._id: 67a3d37e2d9a08978848c657
paper_authors.name: Jun Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:15:09.857Z
paper_authors._id: 63eff09f4a788ed1dd863b09
paper_authors.avatarUrl: /avatars/b557d83cf6d6b3b46dfbe9b7727ae16d.svg
paper_authors.fullname: Jun
paper_authors.type: user
paper_authors.user: oldteacherjoy
paper_authors._id: 67a3d37e2d9a08978848c658
paper_authors.name: Mayur Bhatia
paper_authors._id: 67a3d37e2d9a08978848c659
paper_authors.name: Sahil Phulwani
paper_authors._id: 67a3d37e2d9a08978848c65a
paper_authors.name: Darshan Suresh
paper_authors._id: 67a3d37e2d9a08978848c65b
paper_authors.name: Rafik Matta
paper_title: HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language
    Models on cross-domain multi-file project problems
paper_summary: Evaluating the real-world applicability of large language models (LLMs)
  provides valuable insights for their development and use in software
  development tasks. Existing benchmarks often focus on standalone coding
  problems or specific libraries, overlooking multi-file, project-based scenarios
  and lacking a rigorous evaluation of consistency. The HackerRank-ASTRA
  Benchmark introduces project-based coding problems that mirror real-world
  scenarios. It evaluates model consistency through 32 runs (k = 32) and median
  standard deviation while incorporating taxonomy-level analysis to assess
  sub-skill capabilities. Initial evaluations on 65 problems show that the top
  three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved
  comparable average scores of 75%, with no statistically significant differences
  in performance. Notably, Claude-3.5-Sonnet-1022 demonstrated the highest
  consistency across problems, with low variability (SD = 0.0497), which was
  statistically significant compared to other models, highlighting its
  reliability for real-world software development tasks.
paper_ai_summary: The HackerRank-ASTRA Benchmark evaluates large language models for real-world software development by assessing project-based coding problems, consistency, and sub-skill capabilities.
paper_ai_keywords: large language models LLMs HackerRank-ASTRA Benchmark project-based coding problems consistency median standard deviation taxonomy-level analysis sub-skill capabilities statistical significance reliability