title: Grammar Prompting for Domain-Specific Language Generation with Large
    Language Models
summary: Large language models (LLMs) can learn to perform a wide range of natural
  language tasks from just a handful of in-context examples. However, for
  generating strings from highly structured languages (e.g., semantic parsing to
  complex domain-specific languages), it is challenging for the LLM to generalize
  from just a few exemplars. We explore grammar prompting as a simple
  approach for enabling LLMs to use external knowledge and domain-specific
  constraints, expressed through a grammar expressed in Backus--Naur Form (BNF),
  during in-context learning. Grammar prompting augments each demonstration
  example with a specialized grammar that is minimally sufficient for generating
  the particular output example, where the specialized grammar is a subset of the
  full DSL grammar. For inference, the LLM first predicts a BNF grammar given a
  test input, and then generates the output according to the rules of the
  grammar. Experiments demonstrate that grammar prompting can enable LLMs to
  perform competitively on a diverse set of DSL generation tasks, including
  semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even
  molecule generation (SMILES).
paper_authors._id: 6476c72ea0bde42d3f94b45c
paper_authors.name: Bailin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T05:54:35.403Z
paper_authors._id: 642f4bfbc953ca48ccd162f2
paper_authors.avatarUrl: /avatars/390662485bbf2bbe3454ba9f416d1536.svg
paper_authors.fullname: Bailin
paper_authors.type: user
paper_authors.user: bailin28
paper_authors._id: 6476c72ea0bde42d3f94b45d
paper_authors.name: Zi Wang
paper_authors._id: 6476c72ea0bde42d3f94b45e
paper_authors.name: Xuezhi Wang
paper_authors._id: 6476c72ea0bde42d3f94b45f
paper_authors.name: Yuan Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T05:54:30.844Z
paper_authors._id: 6433a8c37b8247480105cf7f
paper_authors.avatarUrl: /avatars/a19536f3f3b8eae3609648bea112b381.svg
paper_authors.fullname: Yuan Cao
paper_authors.type: user
paper_authors.user: caoyuan33
paper_authors._id: 6476c72ea0bde42d3f94b460
paper_authors.name: Rif A. Saurous
paper_authors._id: 6476c72ea0bde42d3f94b461
paper_authors.name: Yoon Kim
paper_title: Grammar Prompting for Domain-Specific Language Generation with Large
    Language Models
paper_summary: Large language models (LLMs) can learn to perform a wide range of natural
  language tasks from just a handful of in-context examples. However, for
  generating strings from highly structured languages (e.g., semantic parsing to
  complex domain-specific languages), it is challenging for the LLM to generalize
  from just a few exemplars. We explore grammar prompting as a simple
  approach for enabling LLMs to use external knowledge and domain-specific
  constraints, expressed through a grammar expressed in Backus--Naur Form (BNF),
  during in-context learning. Grammar prompting augments each demonstration
  example with a specialized grammar that is minimally sufficient for generating
  the particular output example, where the specialized grammar is a subset of the
  full DSL grammar. For inference, the LLM first predicts a BNF grammar given a
  test input, and then generates the output according to the rules of the
  grammar. Experiments demonstrate that grammar prompting can enable LLMs to
  perform competitively on a diverse set of DSL generation tasks, including
  semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and even
  molecule generation (SMILES).
paper_ai_summary: Grammar prompting enhances LLMs' performance on DSL generation tasks by incorporating domain-specific grammars expressed in BNF.
paper_ai_keywords: grammar prompting Backus--Naur Form (BNF) semantic parsing DSL generation PDDL planning molecule generation SMCalFlow Overnight GeoQuery SMILES