Coding Triangle: How Does Large Language Model Understand Code?
Large language models (LLMs) have achieved remarkable progress in code
generation, yet their true programming competence remains underexplored. We
introduce the Code Triangle framework, which systematically evaluates LLMs
across three fundamental dimensions: editorial analysis, code implementation,
and test case generation. Through extensive experiments on competitive
programming benchmarks, we reveal that while LLMs can form a self-consistent
system across these dimensions, their solutions often lack the diversity and
robustness of human programmers. We identify a significant distribution shift
between model cognition and human expertise, with model errors tending to
cluster due to training data biases and limited reasoning transfer. Our study
demonstrates that incorporating human-generated editorials, solutions, and
diverse test cases, as well as leveraging model mixtures, can substantially
enhance both the performance and robustness of LLMs. Furthermore, we reveal
both the consistency and inconsistency in the cognition of LLMs that may
facilitate self-reflection and self-improvement, providing a potential
direction for developing more powerful coding models.
686ddd6acb5725779c60b43c
Taolin Zhang
claimed_verified
2025-07-10T13:14:37.851Z
64c8835a4cc48498134364be
/avatars/a26120216483c6bf3b830c0cf9c4008e.svg
TaolinZhang
user
iridescentttt
686ddd6acb5725779c60b43d
Zihan Ma
claimed_verified
2025-07-09T08:50:01.962Z
677e869467f3bb8d8215eec6
https://cdn-avatars.huggingface.co/v1/production/uploads/677e869467f3bb8d8215eec6/kEC6JOKObgLHA22jRcP4H.jpeg
Zihan Ma
user
MichaelErchi
686ddd6acb5725779c60b43e
Maosong Cao
686ddd6acb5725779c60b43f
Junnan Liu
claimed_verified
2025-07-09T08:49:59.943Z
643d26979347842571bc9613
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3heFf7h3jbhhJWJ4JfGfh.jpeg
Junnan Liu
user
jnanliu
686ddd6acb5725779c60b440
Songyang Zhang
claimed_verified
2025-07-10T13:14:35.879Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
686ddd6acb5725779c60b441
Kai Chen
Coding Triangle: How Does Large Language Model Understand Code?
Large language models (LLMs) have achieved remarkable progress in code
generation, yet their true programming competence remains underexplored. We
introduce the Code Triangle framework, which systematically evaluates LLMs
across three fundamental dimensions: editorial analysis, code implementation,
and test case generation. Through extensive experiments on competitive
programming benchmarks, we reveal that while LLMs can form a self-consistent
system across these dimensions, their solutions often lack the diversity and
robustness of human programmers. We identify a significant distribution shift
between model cognition and human expertise, with model errors tending to
cluster due to training data biases and limited reasoning transfer. Our study
demonstrates that incorporating human-generated editorials, solutions, and
diverse test cases, as well as leveraging model mixtures, can substantially
enhance both the performance and robustness of LLMs. Furthermore, we reveal
both the consistency and inconsistency in the cognition of LLMs that may
facilitate self-reflection and self-improvement, providing a potential
direction for developing more powerful coding models.
The Code Triangle framework evaluates LLMs in code generation across editorial analysis, implementation, and test case generation, revealing areas for improvement through human-generated inputs and model mixtures.
Code Triangle framework LLMs code generation editorial analysis code implementation test case generation competitive programming benchmarks model cognition human expertise distribution shift training data biases reasoning transfer model mixtures self-reflection self-improvement