title: Self-consistency for open-ended generations
summary: In this paper, we present a novel approach for improving the quality and
  consistency of generated outputs from large-scale pre-trained language models
  (LLMs). Self-consistency has emerged as an effective approach for prompts with
  fixed answers, selecting the answer with the highest number of votes. In this
  paper, we introduce a generalized framework for self-consistency that extends
  its applicability beyond problems that have fixed-answer answers. Through
  extensive simulations, we demonstrate that our approach consistently recovers
  the optimal or near-optimal generation from a set of candidates. We also
  propose lightweight parameter-free similarity functions that show significant
  and consistent improvements across code generation, autoformalization, and
  summarization tasks, even without access to token log probabilities. Our method
  incurs minimal computational overhead, requiring no auxiliary reranker models
  or modifications to the existing model.
paper_authors._id: 64b0b90a2a98a6d6494aa823
paper_authors.name: Siddhartha Jain
paper_authors._id: 64b0b90a2a98a6d6494aa824
paper_authors.name: Xiaofei Ma
paper_authors._id: 64b0b90a2a98a6d6494aa825
paper_authors.name: Anoop Deoras
paper_authors._id: 64b0b90a2a98a6d6494aa826
paper_authors.name: Bing Xiang
paper_title: Self-consistency for open-ended generations
paper_summary: In this paper, we present a novel approach for improving the quality and
  consistency of generated outputs from large-scale pre-trained language models
  (LLMs). Self-consistency has emerged as an effective approach for prompts with
  fixed answers, selecting the answer with the highest number of votes. In this
  paper, we introduce a generalized framework for self-consistency that extends
  its applicability beyond problems that have fixed-answer answers. Through
  extensive simulations, we demonstrate that our approach consistently recovers
  the optimal or near-optimal generation from a set of candidates. We also
  propose lightweight parameter-free similarity functions that show significant
  and consistent improvements across code generation, autoformalization, and
  summarization tasks, even without access to token log probabilities. Our method
  incurs minimal computational overhead, requiring no auxiliary reranker models
  or modifications to the existing model.
paper_ai_keywords: self-consistency pre-trained language models (LLMs) fixed-answer answers generalized framework extensive simulations optimal generation near-optimal generation lightweight parameter-free similarity functions codes generation autoformalization summarization tasks token log probabilities auxiliary reranker models