title: CODESYNC: Synchronizing Large Language Models with Dynamic Code
    Evolution at Scale
summary: Large Language Models (LLMs) have exhibited exceptional performance in
  software engineering yet face challenges in adapting to continually evolving
  code knowledge, particularly regarding the frequent updates of third-party
  library APIs. This limitation, stemming from static pre-training datasets,
  often results in non-executable code or implementations with suboptimal safety
  and efficiency. To this end, this paper introduces CODESYNC, a data engine for
  identifying outdated code patterns and collecting real-time code knowledge
  updates from Python third-party libraries. Building upon CODESYNC, we develop
  CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay
  synchronized with code evolution, which covers real-world updates for 220 APIs
  from six Python libraries. Our benchmark offers 3,300 test cases across three
  evaluation tasks and an update-aware instruction tuning dataset consisting of
  2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs
  reveal that they struggle with dynamic code evolution, even with the support of
  advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe
  that our benchmark can offer a strong foundation for the development of more
  effective methods for real-time code knowledge updating in the future. The
  experimental code and dataset are publicly available at:
  https://github.com/Lucky-voyage/Code-Sync.
paper_authors._id: 67c12e60d8247a49b805694f
paper_authors.name: Chenlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:29:50.564Z
paper_authors._id: 6441270ead24e9b2cfbc45e0
paper_authors.avatarUrl: /avatars/92eab1ae50efaaee070674ae20244fc0.svg
paper_authors.fullname: Wang Chenlong
paper_authors.type: user
paper_authors.user: Wildxxxxx75
paper_authors._id: 67c12e60d8247a49b8056950
paper_authors.name: Zhaoyang Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:29:56.482Z
paper_authors._id: 64fb128552e82dd432682b06
paper_authors.avatarUrl: /avatars/c141326a5d8c17d35be40e12579810bb.svg
paper_authors.fullname: Zhaoyang Chu
paper_authors.type: user
paper_authors.user: chuzy
paper_authors._id: 67c12e60d8247a49b8056951
paper_authors.name: Zhengxiang Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T09:28:33.569Z
paper_authors._id: 669096da35cddb688a352ca8
paper_authors.avatarUrl: /avatars/5dd096cb7360682016d0fca909ab9744.svg
paper_authors.fullname: zxiang
paper_authors.type: user
paper_authors.user: zx10086
paper_authors._id: 67c12e60d8247a49b8056952
paper_authors.name: Xuyi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T09:28:31.564Z
paper_authors._id: 6743e9d4303e7ce5b9d13e9b
paper_authors.avatarUrl: /avatars/cdaf150380e9c8916547185b968a2670.svg
paper_authors.fullname: xy
paper_authors.type: user
paper_authors.user: yxy0807
paper_authors._id: 67c12e60d8247a49b8056953
paper_authors.name: Kaiyue Qiu
paper_authors._id: 67c12e60d8247a49b8056954
paper_authors.name: Yao Wan
paper_authors._id: 67c12e60d8247a49b8056955
paper_authors.name: Zhou Zhao
paper_authors._id: 67c12e60d8247a49b8056956
paper_authors.name: Xuanhua Shi
paper_authors._id: 67c12e60d8247a49b8056957
paper_authors.name: Dongping Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:30:19.705Z
paper_authors._id: 65e2be1e630e2db23829ee8d
paper_authors.avatarUrl: /avatars/294f9ba909037f03669dc0bb80cabfe3.svg
paper_authors.fullname: Dongping Chen
paper_authors.type: user
paper_authors.user: fjchendp
paper_title: CODESYNC: Synchronizing Large Language Models with Dynamic Code
    Evolution at Scale
paper_summary: Large Language Models (LLMs) have exhibited exceptional performance in
  software engineering yet face challenges in adapting to continually evolving
  code knowledge, particularly regarding the frequent updates of third-party
  library APIs. This limitation, stemming from static pre-training datasets,
  often results in non-executable code or implementations with suboptimal safety
  and efficiency. To this end, this paper introduces CODESYNC, a data engine for
  identifying outdated code patterns and collecting real-time code knowledge
  updates from Python third-party libraries. Building upon CODESYNC, we develop
  CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay
  synchronized with code evolution, which covers real-world updates for 220 APIs
  from six Python libraries. Our benchmark offers 3,300 test cases across three
  evaluation tasks and an update-aware instruction tuning dataset consisting of
  2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs
  reveal that they struggle with dynamic code evolution, even with the support of
  advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe
  that our benchmark can offer a strong foundation for the development of more
  effective methods for real-time code knowledge updating in the future. The
  experimental code and dataset are publicly available at:
  https://github.com/Lucky-voyage/Code-Sync.
paper_ai_summary: CODESYNCBENCH is introduced to evaluate LLMs' adaptation to evolving third-party library APIs, revealing their challenges with dynamic code changes.
paper_ai_keywords: LARGE LANGUAGE MODELS (LLMs) CODESYNC CODESYNCBENCH third-party library APIs real-time code knowledge updates non-executable code suboptimal safety and efficiency DPO ORPO SimPO