title: Stay on topic with Classifier-Free Guidance
summary: Classifier-Free Guidance (CFG) has recently emerged in text-to-image
  generation as a lightweight technique to encourage prompt-adherence in
  generations. In this work, we demonstrate that CFG can be used broadly as an
  inference-time technique in pure language modeling. We show that CFG (1)
  improves the performance of Pythia, GPT-2 and LLaMA-family models across an
  array of tasks: Q\&A, reasoning, code generation, and machine translation,
  achieving SOTA on LAMBADA with LLaMA-7B over PaLM-540B; (2) brings improvements
  equivalent to a model with twice the parameter-count; (3) can stack alongside
  other inference-time methods like Chain-of-Thought and Self-Consistency,
  yielding further improvements in difficult tasks; (4) can be used to increase
  the faithfulness and coherence of assistants in challenging form-driven and
  content-driven prompts: in a human evaluation we show a 75\% preference for
  GPT4All using CFG over baseline.
paper_authors._id: 64a222628856c528ea9be2fa
paper_authors.name: Guillaume Sanchez
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-07-03T11:45:46.176Z
paper_authors._id: 6318b35869ef5467e18f0277
paper_authors.avatarUrl: /avatars/1f89cd6a94e4a1e5fea6767a78ba5a0c.svg
paper_authors.fullname: Vermeille
paper_authors.type: user
paper_authors.user: Vermifuge
paper_authors._id: 64a222628856c528ea9be2fb
paper_authors.name: Honglu Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T11:47:18.848Z
paper_authors._id: 635bdaf7dc371b8f91029184
paper_authors.avatarUrl: /avatars/7e3769763c0e3f3ff83f3020aef60768.svg
paper_authors.fullname: Honglu
paper_authors.type: user
paper_authors.user: quintic
paper_authors._id: 64a222628856c528ea9be2fc
paper_authors.name: Alexander Spangher
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-02T18:29:31.416Z
paper_authors._id: 63b67943029518c6bbfa4af5
paper_authors.avatarUrl: /avatars/9ac6db8b8541162ee625a8929a2736ee.svg
paper_authors.fullname: Alexander Spangher
paper_authors.type: user
paper_authors.user: alex2awesome
paper_authors._id: 64a222628856c528ea9be2fd
paper_authors.name: Elad Levi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-07-03T05:01:41.486Z
paper_authors._id: 635d618494e5b275ca73b844
paper_authors.avatarUrl: /avatars/8cdaac6591a12b252612b99094e00959.svg
paper_authors.fullname: Levi
paper_authors.type: user
paper_authors.user: Eladlev
paper_authors._id: 64a222628856c528ea9be2fe
paper_authors.name: Pawan Sasanka Ammanamanchi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-07-03T01:20:34.881Z
paper_authors._id: 5f04beab4ec31d33a72116d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594146435878-noauth.jpeg
paper_authors.fullname: Pawan Sasanka Ammanamanchi
paper_authors.type: user
paper_authors.user: paws
paper_authors._id: 64a222628856c528ea9be2ff
paper_authors.name: Stella Biderman
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-13T05:23:37.865Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_title: Stay on topic with Classifier-Free Guidance
paper_summary: Classifier-Free Guidance (CFG) has recently emerged in text-to-image
  generation as a lightweight technique to encourage prompt-adherence in
  generations. In this work, we demonstrate that CFG can be used broadly as an
  inference-time technique in pure language modeling. We show that CFG (1)
  improves the performance of Pythia, GPT-2 and LLaMA-family models across an
  array of tasks: Q\&A, reasoning, code generation, and machine translation,
  achieving SOTA on LAMBADA with LLaMA-7B over PaLM-540B; (2) brings improvements
  equivalent to a model with twice the parameter-count; (3) can stack alongside
  other inference-time methods like Chain-of-Thought and Self-Consistency,
  yielding further improvements in difficult tasks; (4) can be used to increase
  the faithfulness and coherence of assistants in challenging form-driven and
  content-driven prompts: in a human evaluation we show a 75\% preference for
  GPT4All using CFG over baseline.
paper_ai_keywords: Classifier-Free Guidance (CFG) text-to-image generation prompt-adherence pure language modeling Pythia GPT-2 LLaMA-family models Q&A reasoning code generation machine translation LAMBADA LLaMA-7B PaLM-540B inference-time methods Chain-of-Thought Self-Consistency faithfulness coherence form-driven prompts content-driven prompts human evaluation GPT4All