title: GitChameleon: Unmasking the Version-Switching Capabilities of Code
    Generation Models
summary: The rapid evolution of software libraries presents a significant challenge
  for code generation models, which must adapt to frequent version updates while
  maintaining compatibility with previous versions. Existing code completion
  benchmarks often overlook this dynamic aspect, and the one that does consider
  it relies on static code prediction tasks without execution-based evaluation,
  offering a limited perspective on a model's practical usability. To address
  this gap, we introduce \GitChameleon{}, a novel, manually curated
  dataset comprising 116 Python code completion problems, each conditioned on
  specific library versions and accompanied by executable unit tests.
   is designed to rigorously assess the ability of modern large
  language models (LLMs) to generate version-specific code that is not only
  syntactically correct but also functionally accurate upon execution. Our
  comprehensive evaluations reveal that state-of-the-art LLMs struggle with this
  task; for instance, GPT-4o achieves a pass@10 of only 39.9\% (43.7\%
  when provided with error feedback), highlighting the complexity of the problem
  and the limitations of current models. By providing an execution-based
  benchmark that emphasizes the dynamic nature of code libraries, 
  serves as a critical tool to advance the development of more adaptable and
  reliable code generation models. For facilitation for further exploration of
  version-conditioned code generation, we make our code repository publicly
  accessible at https://github.com/NizarIslah/GitChameleon.
paper_authors._id: 673360a07c79bb4d069dcc57
paper_authors.name: Nizar Islah
paper_authors._id: 673360a07c79bb4d069dcc58
paper_authors.name: Justine Gehring
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:25:48.365Z
paper_authors._id: 6421f59be61513ec1c66f277
paper_authors.avatarUrl: /avatars/7792f5bad78dcae548125e6d6d6c22e7.svg
paper_authors.fullname: Justine Gehring
paper_authors.type: user
paper_authors.user: justine-gehring
paper_authors._id: 673360a07c79bb4d069dcc59
paper_authors.name: Diganta Misra
paper_authors._id: 673360a07c79bb4d069dcc5a
paper_authors.name: Eilif Muller
paper_authors._id: 673360a07c79bb4d069dcc5b
paper_authors.name: Irina Rish
paper_authors._id: 673360a07c79bb4d069dcc5c
paper_authors.name: Terry Yue Zhuo
paper_authors._id: 673360a07c79bb4d069dcc5d
paper_authors.name: Massimo Caccia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:23:25.890Z
paper_authors._id: 64b04320e5000ae8a57570a7
paper_authors.avatarUrl: /avatars/226f57ea23b9dd861e71b7ba7821adce.svg
paper_authors.fullname: Massimo Caccia
paper_authors.type: user
paper_authors.user: optimass
paper_title: GitChameleon: Unmasking the Version-Switching Capabilities of Code
    Generation Models
paper_summary: The rapid evolution of software libraries presents a significant challenge
  for code generation models, which must adapt to frequent version updates while
  maintaining compatibility with previous versions. Existing code completion
  benchmarks often overlook this dynamic aspect, and the one that does consider
  it relies on static code prediction tasks without execution-based evaluation,
  offering a limited perspective on a model's practical usability. To address
  this gap, we introduce \GitChameleon{}, a novel, manually curated
  dataset comprising 116 Python code completion problems, each conditioned on
  specific library versions and accompanied by executable unit tests.
   is designed to rigorously assess the ability of modern large
  language models (LLMs) to generate version-specific code that is not only
  syntactically correct but also functionally accurate upon execution. Our
  comprehensive evaluations reveal that state-of-the-art LLMs struggle with this
  task; for instance, GPT-4o achieves a pass@10 of only 39.9\% (43.7\%
  when provided with error feedback), highlighting the complexity of the problem
  and the limitations of current models. By providing an execution-based
  benchmark that emphasizes the dynamic nature of code libraries, 
  serves as a critical tool to advance the development of more adaptable and
  reliable code generation models. For facilitation for further exploration of
  version-conditioned code generation, we make our code repository publicly
  accessible at https://github.com/NizarIslah/GitChameleon.