title: Statically Contextualizing Large Language Models with Typed Holes
summary: Large language models (LLMs) have reshaped the landscape of program
  synthesis. However, contemporary LLM-based code completion systems often
  hallucinate broken code because they lack appropriate context, particularly
  when working with definitions not in the training data nor near the cursor.
  This paper demonstrates that tight integration with the type and binding
  structure of a language, as exposed by its language server, can address this
  contextualization problem in a token-efficient manner. In short, we contend
  that AIs need IDEs, too! In particular, we integrate LLM code generation into
  the Hazel live program sketching environment. The Hazel Language Server
  identifies the type and typing context of the hole being filled, even in the
  presence of errors, ensuring that a meaningful program sketch is always
  available. This allows prompting with codebase-wide contextual information not
  lexically local to the cursor, nor necessarily in the same file, but that is
  likely to be semantically local to the developer's goal. Completions
  synthesized by the LLM are then iteratively refined via further dialog with the
  language server. To evaluate these techniques, we introduce MVUBench, a dataset
  of model-view-update (MVU) web applications. These applications serve as
  challenge problems due to their reliance on application-specific data
  structures. We find that contextualization with type definitions is
  particularly impactful. After introducing our ideas in the context of Hazel we
  duplicate our techniques and port MVUBench to TypeScript in order to validate
  the applicability of these methods to higher-resource languages. Finally, we
  outline ChatLSP, a conservative extension to the Language Server Protocol (LSP)
  that language servers can implement to expose capabilities that AI code
  completion systems of various designs can use to incorporate static context
  when generating prompts for an LLM.
paper_authors._id: 66d802247e6c9509bb107bd4
paper_authors.name: Andrew Blinn
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:14:58.426Z
paper_authors._id: 630407b7d14428368d1a936f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630407b7d14428368d1a936f/8u_a8SQq00pBIy0DKNS5R.jpeg
paper_authors.fullname: andrew blinn
paper_authors.type: user
paper_authors.user: disconcision
paper_authors._id: 66d802247e6c9509bb107bd5
paper_authors.name: Xiang Li
paper_authors._id: 66d802247e6c9509bb107bd6
paper_authors.name: June Hyung Kim
paper_authors._id: 66d802247e6c9509bb107bd7
paper_authors.name: Cyrus Omar
paper_title: Statically Contextualizing Large Language Models with Typed Holes
paper_summary: Large language models (LLMs) have reshaped the landscape of program
  synthesis. However, contemporary LLM-based code completion systems often
  hallucinate broken code because they lack appropriate context, particularly
  when working with definitions not in the training data nor near the cursor.
  This paper demonstrates that tight integration with the type and binding
  structure of a language, as exposed by its language server, can address this
  contextualization problem in a token-efficient manner. In short, we contend
  that AIs need IDEs, too! In particular, we integrate LLM code generation into
  the Hazel live program sketching environment. The Hazel Language Server
  identifies the type and typing context of the hole being filled, even in the
  presence of errors, ensuring that a meaningful program sketch is always
  available. This allows prompting with codebase-wide contextual information not
  lexically local to the cursor, nor necessarily in the same file, but that is
  likely to be semantically local to the developer's goal. Completions
  synthesized by the LLM are then iteratively refined via further dialog with the
  language server. To evaluate these techniques, we introduce MVUBench, a dataset
  of model-view-update (MVU) web applications. These applications serve as
  challenge problems due to their reliance on application-specific data
  structures. We find that contextualization with type definitions is
  particularly impactful. After introducing our ideas in the context of Hazel we
  duplicate our techniques and port MVUBench to TypeScript in order to validate
  the applicability of these methods to higher-resource languages. Finally, we
  outline ChatLSP, a conservative extension to the Language Server Protocol (LSP)
  that language servers can implement to expose capabilities that AI code
  completion systems of various designs can use to incorporate static context
  when generating prompts for an LLM.
paper_ai_keywords: type typing context live program sketching environment Hazel Language Server model-view-update (MVU) dataset type definitions MVUBench TypeScript ChatLSP Language Server Protocol (LSP) static context prompt generation AI code completion systems code generation dialog with the language server