CODESYNC: Synchronizing Large Language Models with Dynamic Code
  Evolution at Scale
Large Language Models (LLMs) have exhibited exceptional performance in
software engineering yet face challenges in adapting to continually evolving
code knowledge, particularly regarding the frequent updates of third-party
library APIs. This limitation, stemming from static pre-training datasets,
often results in non-executable code or implementations with suboptimal safety
and efficiency. To this end, this paper introduces CODESYNC, a data engine for
identifying outdated code patterns and collecting real-time code knowledge
updates from Python third-party libraries. Building upon CODESYNC, we develop
CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay
synchronized with code evolution, which covers real-world updates for 220 APIs
from six Python libraries. Our benchmark offers 3,300 test cases across three
evaluation tasks and an update-aware instruction tuning dataset consisting of
2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs
reveal that they struggle with dynamic code evolution, even with the support of
advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe
that our benchmark can offer a strong foundation for the development of more
effective methods for real-time code knowledge updating in the future. The
experimental code and dataset are publicly available at:
https://github.com/Lucky-voyage/Code-Sync.
67c12e60d8247a49b805694f
Chenlong Wang
admin_assigned
2025-02-28T12:29:50.564Z
6441270ead24e9b2cfbc45e0
/avatars/92eab1ae50efaaee070674ae20244fc0.svg
Wang Chenlong
user
Wildxxxxx75
67c12e60d8247a49b8056950
Zhaoyang Chu
admin_assigned
2025-02-28T12:29:56.482Z
64fb128552e82dd432682b06
/avatars/c141326a5d8c17d35be40e12579810bb.svg
Zhaoyang Chu
user
chuzy
67c12e60d8247a49b8056951
Zhengxiang Cheng
claimed_verified
2025-02-28T09:28:33.569Z
669096da35cddb688a352ca8
/avatars/5dd096cb7360682016d0fca909ab9744.svg
zxiang
user
zx10086
67c12e60d8247a49b8056952
Xuyi Yang
claimed_verified
2025-02-28T09:28:31.564Z
6743e9d4303e7ce5b9d13e9b
/avatars/cdaf150380e9c8916547185b968a2670.svg
xy
user
yxy0807
67c12e60d8247a49b8056953
Kaiyue Qiu
67c12e60d8247a49b8056954
Yao Wan
67c12e60d8247a49b8056955
Zhou Zhao
67c12e60d8247a49b8056956
Xuanhua Shi
67c12e60d8247a49b8056957
Dongping Chen
admin_assigned
2025-02-28T12:30:19.705Z
65e2be1e630e2db23829ee8d
/avatars/294f9ba909037f03669dc0bb80cabfe3.svg
Dongping Chen
user
fjchendp
CODESYNC: Synchronizing Large Language Models with Dynamic Code
  Evolution at Scale
Large Language Models (LLMs) have exhibited exceptional performance in
software engineering yet face challenges in adapting to continually evolving
code knowledge, particularly regarding the frequent updates of third-party
library APIs. This limitation, stemming from static pre-training datasets,
often results in non-executable code or implementations with suboptimal safety
and efficiency. To this end, this paper introduces CODESYNC, a data engine for
identifying outdated code patterns and collecting real-time code knowledge
updates from Python third-party libraries. Building upon CODESYNC, we develop
CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay
synchronized with code evolution, which covers real-world updates for 220 APIs
from six Python libraries. Our benchmark offers 3,300 test cases across three
evaluation tasks and an update-aware instruction tuning dataset consisting of
2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs
reveal that they struggle with dynamic code evolution, even with the support of
advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe
that our benchmark can offer a strong foundation for the development of more
effective methods for real-time code knowledge updating in the future. The
experimental code and dataset are publicly available at:
https://github.com/Lucky-voyage/Code-Sync.
CODESYNCBENCH is introduced to evaluate LLMs' adaptation to evolving third-party library APIs, revealing their challenges with dynamic code changes.
LARGE LANGUAGE MODELS (LLMs) CODESYNC CODESYNCBENCH third-party library APIs real-time code knowledge updates non-executable code suboptimal safety and efficiency DPO ORPO SimPO