title: A Survey on Language Models for Code
summary: In this work we systematically review the recent advancements in code
  processing with language models, covering 50+ models, 30+ evaluation tasks, and
  500 related works. We break down code processing models into general language
  models represented by the GPT family and specialized models that are
  specifically pretrained on code, often with tailored objectives. We discuss the
  relations and differences between these models, and highlight the historical
  transition of code modeling from statistical models and RNNs to pretrained
  Transformers and LLMs, which is exactly the same course that had been taken by
  NLP. We also discuss code-specific features such as AST, CFG, and unit tests,
  along with their application in training code language models, and identify key
  challenges and potential future directions in this domain. We keep the survey
  open and updated on github repository at
  https://github.com/codefuse-ai/Awesome-Code-LLM.
paper_authors._id: 65543b3143cc10c9d094f5c1
paper_authors.name: Ziyin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T18:03:46.110Z
paper_authors._id: 6430bdd8cd31d174a9f900fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
paper_authors.fullname: Ziyin Zhang
paper_authors.type: user
paper_authors.user: Geralt-Targaryen
paper_authors._id: 65543b3143cc10c9d094f5c2
paper_authors.name: Chaoyu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:54:44.160Z
paper_authors._id: 63f4860dcfd7ba6e26f3c694
paper_authors.avatarUrl: /avatars/de1ac926d4395688e1da75568ac50c76.svg
paper_authors.fullname: Chaoyu Chen
paper_authors.type: user
paper_authors.user: chencyudel
paper_authors._id: 65543b3143cc10c9d094f5c3
paper_authors.name: Bingchang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T08:47:04.451Z
paper_authors._id: 640ec9fc8512ec51d7f64409
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640ec9fc8512ec51d7f64409/LSq6prKfiWCsvAPcBI4Rm.jpeg
paper_authors.fullname: L BC
paper_authors.type: user
paper_authors.user: twelveand0
paper_authors._id: 65543b3143cc10c9d094f5c4
paper_authors.name: Cong Liao
paper_authors._id: 65543b3143cc10c9d094f5c5
paper_authors.name: Zi Gong
paper_authors._id: 65543b3143cc10c9d094f5c6
paper_authors.name: Hang Yu
paper_authors._id: 65543b3143cc10c9d094f5c7
paper_authors.name: Jianguo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:55:37.006Z
paper_authors._id: 6539fc3fe279d2fc80c88902
paper_authors.avatarUrl: /avatars/432ac2287a7f7d04c2c7e505070a3331.svg
paper_authors.fullname: Jianguo Li
paper_authors.type: user
paper_authors.user: jglee2046
paper_authors._id: 65543b3143cc10c9d094f5c8
paper_authors.name: Rui Wang
paper_title: A Survey on Language Models for Code
paper_summary: In this work we systematically review the recent advancements in code
  processing with language models, covering 50+ models, 30+ evaluation tasks, and
  500 related works. We break down code processing models into general language
  models represented by the GPT family and specialized models that are
  specifically pretrained on code, often with tailored objectives. We discuss the
  relations and differences between these models, and highlight the historical
  transition of code modeling from statistical models and RNNs to pretrained
  Transformers and LLMs, which is exactly the same course that had been taken by
  NLP. We also discuss code-specific features such as AST, CFG, and unit tests,
  along with their application in training code language models, and identify key
  challenges and potential future directions in this domain. We keep the survey
  open and updated on github repository at
  https://github.com/codefuse-ai/Awesome-Code-LLM.
paper_ai_keywords: GPT family pretrained Transformers LLMs Statistical models RNNs code-specific features AST (Abstract Syntax Tree) CFG (Control Flow Graph) unit tests