Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysis
<|reference_start|>Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysis: This study explores the performance of large language models (LLMs) in solving competitive programming problems from the Romanian Informatics Olympiad at the county level. Romania, a leading nation in computer science competitions, provides an ideal environment for evaluating LLM capabilities due to its rich history and stringent competition standards. We collected and analyzed a dataset comprising 304 challenges from 2002 to 2023, focusing on solutions written by LLMs in C++ and Python for these problems. Our primary goal is to understand why LLMs perform well or poorly on different tasks. We evaluated various models, including closed-source models like GPT-4 and open-weight models such as CodeLlama and RoMistral, using a standardized process involving multiple attempts and feedback rounds. The analysis revealed significant variations in LLM performance across different grades and problem types. Notably, GPT-4 showed strong performance, indicating its potential use as an educational tool for middle school students. We also observed differences in code quality and style across various LLMs<|reference_end|>
@article{dumitran2024evaluating,
    title={Evaluating the Performance of Large Language Models in Competitive
  Programming: A Multi-Year, Multi-Grade Analysis},
    author={Adrian Marius Dumitran, Adrian Catalin Badea, Stefan-Gabriel Muscalu},
    journal={arXiv preprint arXiv:2409.09054},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09054},
    primaryClass={cs.SE cs.AI cs.PL}
}