title: Suppressing Pink Elephants with Direct Principle Feedback
summary: Existing methods for controlling language models, such as RLHF and
  Constitutional AI, involve determining which LLM behaviors are desirable and
  training them into a language model. However, in many cases, it is desirable
  for LLMs to be controllable at inference time, so that they can be
  used in multiple contexts with diverse needs. We illustrate this with the
  Pink Elephant Problem: instructing an LLM to avoid discussing a
  certain entity (a ``Pink Elephant''), and instead discuss a preferred entity
  (``Grey Elephant''). We apply a novel simplification of Constitutional AI,
  Direct Principle Feedback, which skips the ranking of responses and
  uses DPO directly on critiques and revisions. Our results show that after DPF
  fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2
  model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and
  performs as well as GPT-4 in on our curated test set assessing the Pink
  Elephant Problem.
paper_authors._id: 65cad2814913057ac8189f99
paper_authors.name: Louis Castricato
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:20:26.331Z
paper_authors._id: 60d35a0307da9c17c7270909
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665155571839-60d35a0307da9c17c7270909.jpeg
paper_authors.fullname: Louis Castricato
paper_authors.type: user
paper_authors.user: LouisCastricato
paper_authors._id: 65cad2814913057ac8189f9a
paper_authors.name: Nathan Lile
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T12:35:39.367Z
paper_authors._id: 61aa15fd8a9625ebfe284286
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61aa15fd8a9625ebfe284286/KaGzIeijcgcN15JErCqft.jpeg
paper_authors.fullname: nathan lile
paper_authors.type: user
paper_authors.user: nlile
paper_authors._id: 65cad2814913057ac8189f9b
paper_authors.name: Suraj Anand
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:21:10.203Z
paper_authors._id: 65a8015fc0e51d82e67e9167
paper_authors.avatarUrl: /avatars/e13f5bc548dbe129694e4d3536e8057a.svg
paper_authors.fullname: Suraj Anand
paper_authors.type: user
paper_authors.user: surajk610
paper_authors._id: 65cad2814913057ac8189f9c
paper_authors.name: Hailey Schoelkopf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T12:21:17.178Z
paper_authors._id: 62895a0215aeee85756062c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669665010552-62895a0215aeee85756062c4.jpeg
paper_authors.fullname: Hailey Schoelkopf
paper_authors.type: user
paper_authors.user: hails
paper_authors._id: 65cad2814913057ac8189f9d
paper_authors.name: Siddharth Verma
paper_authors._id: 65cad2814913057ac8189f9e
paper_authors.name: Stella Biderman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-13T08:10:12.517Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_title: Suppressing Pink Elephants with Direct Principle Feedback
paper_summary: Existing methods for controlling language models, such as RLHF and
  Constitutional AI, involve determining which LLM behaviors are desirable and
  training them into a language model. However, in many cases, it is desirable
  for LLMs to be controllable at inference time, so that they can be
  used in multiple contexts with diverse needs. We illustrate this with the
  Pink Elephant Problem: instructing an LLM to avoid discussing a
  certain entity (a ``Pink Elephant''), and instead discuss a preferred entity
  (``Grey Elephant''). We apply a novel simplification of Constitutional AI,
  Direct Principle Feedback, which skips the ranking of responses and
  uses DPO directly on critiques and revisions. Our results show that after DPF
  fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2
  model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and
  performs as well as GPT-4 in on our curated test set assessing the Pink
  Elephant Problem.
paper_ai_summary: A novel method called Direct Principle Feedback fine-tunes language models to become controllable at inference time, demonstrated on the Pink Elephant Problem.
paper_ai_keywords: RLHF Constitutional AI Direct Principle Feedback DPO Pink Elephant Problem fine-tuning LLaMA 2 GPT-4