Unifying the Perspectives of NLP and Software Engineering: A Survey on
  Language Models for Code
In this work we systematically review the recent advancements in software
engineering with language models, covering 70+ models, 40+ evaluation tasks,
180+ datasets, and 900 related works. Unlike previous works, we integrate
software engineering (SE) with natural language processing (NLP) by discussing
the perspectives of both sides: SE applies language models for development
automation, while NLP adopts SE tasks for language model evaluation. We break
down code processing models into general language models represented by the GPT
family and specialized models that are specifically pretrained on code, often
with tailored objectives. We discuss the relations and differences between
these models, and highlight the historical transition of code modeling from
statistical models and RNNs to pretrained Transformers and LLMs, which is
exactly the same course that had been taken by NLP. We also go beyond
programming and review LLMs' application in other software engineering
activities including requirement engineering, testing, deployment, and
operations in an endeavor to provide a global view of NLP in SE, and identify
key challenges and potential future directions in this domain. We keep the
survey open and updated on GitHub at
https://github.com/codefuse-ai/Awesome-Code-LLM.
65543b3143cc10c9d094f5c1
Ziyin Zhang
admin_assigned
2024-09-13T18:03:46.110Z
6430bdd8cd31d174a9f900fb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
Ziyin Zhang
user
Geralt-Targaryen
65543b3143cc10c9d094f5c2
Chaoyu Chen
admin_assigned
2023-11-15T09:54:44.160Z
63f4860dcfd7ba6e26f3c694
/avatars/de1ac926d4395688e1da75568ac50c76.svg
Chaoyu Chen
user
chencyudel
65543b3143cc10c9d094f5c3
Bingchang Liu
claimed_verified
2025-04-08T08:47:04.451Z
640ec9fc8512ec51d7f64409
https://cdn-avatars.huggingface.co/v1/production/uploads/640ec9fc8512ec51d7f64409/LSq6prKfiWCsvAPcBI4Rm.jpeg
L BC
user
twelveand0
65543b3143cc10c9d094f5c4
Cong Liao
65543b3143cc10c9d094f5c5
Zi Gong
65543b3143cc10c9d094f5c6
Hang Yu
65543b3143cc10c9d094f5c7
Jianguo Li
admin_assigned
2023-11-15T09:55:37.006Z
6539fc3fe279d2fc80c88902
/avatars/432ac2287a7f7d04c2c7e505070a3331.svg
Jianguo Li
user
jglee2046
65543b3143cc10c9d094f5c8
Rui Wang
Unifying the Perspectives of NLP and Software Engineering: A Survey on
  Language Models for Code
In this work we systematically review the recent advancements in software
engineering with language models, covering 70+ models, 40+ evaluation tasks,
180+ datasets, and 900 related works. Unlike previous works, we integrate
software engineering (SE) with natural language processing (NLP) by discussing
the perspectives of both sides: SE applies language models for development
automation, while NLP adopts SE tasks for language model evaluation. We break
down code processing models into general language models represented by the GPT
family and specialized models that are specifically pretrained on code, often
with tailored objectives. We discuss the relations and differences between
these models, and highlight the historical transition of code modeling from
statistical models and RNNs to pretrained Transformers and LLMs, which is
exactly the same course that had been taken by NLP. We also go beyond
programming and review LLMs' application in other software engineering
activities including requirement engineering, testing, deployment, and
operations in an endeavor to provide a global view of NLP in SE, and identify
key challenges and potential future directions in this domain. We keep the
survey open and updated on GitHub at
https://github.com/codefuse-ai/Awesome-Code-LLM.
GPT family pretrained Transformers LLMs Statistical models RNNs code-specific features AST (Abstract Syntax Tree) CFG (Control Flow Graph) unit tests