title: Guiding Language Models of Code with Global Context using Monitors
summary: Language models of code (LMs) work well when the surrounding code in the
  vicinity of generation provides sufficient context. This is not true when it
  becomes necessary to use types or functionality defined in another module or
  library, especially those not seen during training. LMs suffer from limited
  awareness of such global context and end up hallucinating, e.g., using types
  defined in other files incorrectly. Recent work tries to overcome this issue by
  retrieving global information to augment the local context. However, this
  bloats the prompt or requires architecture modifications and additional
  training.
    Integrated development environments (IDEs) assist developers by bringing the
  global context at their fingertips using static analysis. We extend this
  assistance, enjoyed by developers, to the LMs. We propose a notion of monitors
  that use static analysis in the background to guide the decoding. Unlike a
  priori retrieval, static analysis is invoked iteratively during the entire
  decoding process, providing the most relevant suggestions on demand. We
  demonstrate the usefulness of our proposal by monitoring for type-consistent
  use of identifiers whenever an LM generates code for object dereference.
    To evaluate our approach, we curate PragmaticCode, a dataset of open-source
  projects with their development environments. On models of varying parameter
  scale, we show that monitor-guided decoding consistently improves the ability
  of an LM to not only generate identifiers that match the ground truth but also
  improves compilation rates and agreement with ground truth. We find that LMs
  with fewer parameters, when guided with our monitor, can outperform larger LMs.
  With monitor-guided decoding, SantaCoder-1.1B achieves better compilation rate
  and next-identifier match than the much larger text-davinci-003 model. The
  datasets and code will be released at https://aka.ms/monitors4codegen .
paper_authors._id: 6492611c7bcb175b58b69046
paper_authors.name: Lakshya A Agrawal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T09:16:58.013Z
paper_authors._id: 63bf0cbb24e75d1efdfdf8bd
paper_authors.avatarUrl: /avatars/3bb13e989db31820734cbe6b13be130a.svg
paper_authors.fullname: Lakshya A Agrawal
paper_authors.type: user
paper_authors.user: LakshyAAAgrawal
paper_authors._id: 6492611c7bcb175b58b69047
paper_authors.name: Aditya Kanade
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-02T12:51:57.419Z
paper_authors._id: 649e62e953777d8950749c72
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649e62e953777d8950749c72/PHigcJ-TQ7HXwriZkBhLV.jpeg
paper_authors.fullname: AK
paper_authors.type: user
paper_authors.user: adityakanade
paper_authors._id: 6492611c7bcb175b58b69048
paper_authors.name: Navin Goyal
paper_authors._id: 6492611c7bcb175b58b69049
paper_authors.name: Shuvendu K. Lahiri
paper_authors._id: 6492611c7bcb175b58b6904a
paper_authors.name: Sriram K. Rajamani
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-21T03:48:09.988Z
paper_authors._id: 63fb54480aab060792f71151
paper_authors.avatarUrl: /avatars/97edbaed65e87a555bf05f770fa60b23.svg
paper_authors.fullname: Sriram Rajamani
paper_authors.type: user
paper_authors.user: sriramkr
paper_title: Guiding Language Models of Code with Global Context using Monitors
paper_summary: Language models of code (LMs) work well when the surrounding code in the
  vicinity of generation provides sufficient context. This is not true when it
  becomes necessary to use types or functionality defined in another module or
  library, especially those not seen during training. LMs suffer from limited
  awareness of such global context and end up hallucinating, e.g., using types
  defined in other files incorrectly. Recent work tries to overcome this issue by
  retrieving global information to augment the local context. However, this
  bloats the prompt or requires architecture modifications and additional
  training.
    Integrated development environments (IDEs) assist developers by bringing the
  global context at their fingertips using static analysis. We extend this
  assistance, enjoyed by developers, to the LMs. We propose a notion of monitors
  that use static analysis in the background to guide the decoding. Unlike a
  priori retrieval, static analysis is invoked iteratively during the entire
  decoding process, providing the most relevant suggestions on demand. We
  demonstrate the usefulness of our proposal by monitoring for type-consistent
  use of identifiers whenever an LM generates code for object dereference.
    To evaluate our approach, we curate PragmaticCode, a dataset of open-source
  projects with their development environments. On models of varying parameter
  scale, we show that monitor-guided decoding consistently improves the ability
  of an LM to not only generate identifiers that match the ground truth but also
  improves compilation rates and agreement with ground truth. We find that LMs
  with fewer parameters, when guided with our monitor, can outperform larger LMs.
  With monitor-guided decoding, SantaCoder-1.1B achieves better compilation rate
  and next-identifier match than the much larger text-davinci-003 model. The
  datasets and code will be released at https://aka.ms/monitors4codegen .
paper_ai_summary: Using static analysis monitors during code generation improves the accuracy and compilation rates of language models by providing context-aware guidance.
paper_ai_keywords: language models of code static analysis decoding identifiers object dereference PragmaticCode compilation rates next-identifier match monitor-guided decoding SantaCoder-1.1B text-davinci-003