GitChameleon: Unmasking the Version-Switching Capabilities of Code
  Generation Models
The rapid evolution of software libraries presents a significant challenge
for code generation models, which must adapt to frequent version updates while
maintaining compatibility with previous versions. Existing code completion
benchmarks often overlook this dynamic aspect, and the one that does consider
it relies on static code prediction tasks without execution-based evaluation,
offering a limited perspective on a model's practical usability. To address
this gap, we introduce \GitChameleon{}, a novel, manually curated
dataset comprising 116 Python code completion problems, each conditioned on
specific library versions and accompanied by executable unit tests.
 is designed to rigorously assess the ability of modern large
language models (LLMs) to generate version-specific code that is not only
syntactically correct but also functionally accurate upon execution. Our
comprehensive evaluations reveal that state-of-the-art LLMs struggle with this
task; for instance, GPT-4o achieves a pass@10 of only 39.9\% (43.7\%
when provided with error feedback), highlighting the complexity of the problem
and the limitations of current models. By providing an execution-based
benchmark that emphasizes the dynamic nature of code libraries, 
serves as a critical tool to advance the development of more adaptable and
reliable code generation models. For facilitation for further exploration of
version-conditioned code generation, we make our code repository publicly
accessible at https://github.com/NizarIslah/GitChameleon.
673360a07c79bb4d069dcc57
Nizar Islah
673360a07c79bb4d069dcc58
Justine Gehring
claimed_verified
2024-11-15T09:25:48.365Z
6421f59be61513ec1c66f277
/avatars/7792f5bad78dcae548125e6d6d6c22e7.svg
Justine Gehring
user
justine-gehring
673360a07c79bb4d069dcc59
Diganta Misra
673360a07c79bb4d069dcc5a
Eilif Muller
673360a07c79bb4d069dcc5b
Irina Rish
673360a07c79bb4d069dcc5c
Terry Yue Zhuo
673360a07c79bb4d069dcc5d
Massimo Caccia
claimed_verified
2024-12-13T15:23:25.890Z
64b04320e5000ae8a57570a7
/avatars/226f57ea23b9dd861e71b7ba7821adce.svg
Massimo Caccia
user
optimass
GitChameleon: Unmasking the Version-Switching Capabilities of Code
  Generation Models
The rapid evolution of software libraries presents a significant challenge
for code generation models, which must adapt to frequent version updates while
maintaining compatibility with previous versions. Existing code completion
benchmarks often overlook this dynamic aspect, and the one that does consider
it relies on static code prediction tasks without execution-based evaluation,
offering a limited perspective on a model's practical usability. To address
this gap, we introduce \GitChameleon{}, a novel, manually curated
dataset comprising 116 Python code completion problems, each conditioned on
specific library versions and accompanied by executable unit tests.
 is designed to rigorously assess the ability of modern large
language models (LLMs) to generate version-specific code that is not only
syntactically correct but also functionally accurate upon execution. Our
comprehensive evaluations reveal that state-of-the-art LLMs struggle with this
task; for instance, GPT-4o achieves a pass@10 of only 39.9\% (43.7\%
when provided with error feedback), highlighting the complexity of the problem
and the limitations of current models. By providing an execution-based
benchmark that emphasizes the dynamic nature of code libraries, 
serves as a critical tool to advance the development of more adaptable and
reliable code generation models. For facilitation for further exploration of
version-conditioned code generation, we make our code repository publicly
accessible at https://github.com/NizarIslah/GitChameleon.
A new dataset, GitChameleon, evaluates large language models' ability to generate version-specific and executable code, highlighting their limitations.
code generation models large language models code completion GitChameleon version-specific code executable unit tests pass@10 GPT-4o