title: Personas as a Way to Model Truthfulness in Language Models
summary: Large Language Models are trained on vast amounts of text from the internet,
  which contains both factual and misleading information about the world. Can
  language models discern truth from falsehood in this contradicting data?
  Expanding on the view that LLMs can model different agents producing the
  corpora, we hypothesize that they can cluster truthful text by modeling a
  truthful persona: a group of agents that are likely to produce truthful text
  and share similar features. For example, trustworthy sources like Wikipedia and
  Science usually use formal writing styles and make consistent claims. By
  modeling this persona, LLMs can generalize truthfulness beyond the specific
  contexts in which each agent generated the training text. For example, the
  model can infer that the agent "Wikipedia" will behave truthfully on topics
  that were only generated by "Science" because they share a persona. We first
  show evidence for the persona hypothesis via two observations: (1) we can probe
  whether a model's answer will be truthful before it is generated; (2)
  finetuning a model on a set of facts improves its truthfulness on unseen
  topics. Next, using arithmetics as a synthetic environment, we show that
  language models can separate true and false statements, and generalize
  truthfulness across agents; but only if agents in the training data share a
  truthful generative process that enables the creation of a truthful persona.
  Overall, our findings suggest that models can exploit hierarchical structures
  in the data to learn abstract concepts like truthfulness.
paper_authors._id: 653f18caec992c5c209b9947
paper_authors.name: Nitish Joishi
paper_authors._id: 653f18caec992c5c209b9948
paper_authors.name: Javier Rando
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T14:09:10.025Z
paper_authors._id: 6245b1b2d5895bdf34ef8c33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6245b1b2d5895bdf34ef8c33/hLmcppWpRS7s2rxMxdWGh.jpeg
paper_authors.fullname: Javier Rando
paper_authors.type: user
paper_authors.user: javirandor
paper_authors._id: 653f18caec992c5c209b9949
paper_authors.name: Abulhair Saparov
paper_authors._id: 653f18caec992c5c209b994a
paper_authors.name: Najoung Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T14:09:24.670Z
paper_authors._id: 627c0b0a7e62b4bf5c37ea65
paper_authors.avatarUrl: /avatars/7b459b6828bcdfd66bc3fbd922606d14.svg
paper_authors.fullname: Najoung Kim
paper_authors.type: user
paper_authors.user: najoungkim
paper_authors._id: 653f18caec992c5c209b994b
paper_authors.name: He He
paper_title: Personas as a Way to Model Truthfulness in Language Models
paper_summary: Large Language Models are trained on vast amounts of text from the internet,
  which contains both factual and misleading information about the world. Can
  language models discern truth from falsehood in this contradicting data?
  Expanding on the view that LLMs can model different agents producing the
  corpora, we hypothesize that they can cluster truthful text by modeling a
  truthful persona: a group of agents that are likely to produce truthful text
  and share similar features. For example, trustworthy sources like Wikipedia and
  Science usually use formal writing styles and make consistent claims. By
  modeling this persona, LLMs can generalize truthfulness beyond the specific
  contexts in which each agent generated the training text. For example, the
  model can infer that the agent "Wikipedia" will behave truthfully on topics
  that were only generated by "Science" because they share a persona. We first
  show evidence for the persona hypothesis via two observations: (1) we can probe
  whether a model's answer will be truthful before it is generated; (2)
  finetuning a model on a set of facts improves its truthfulness on unseen
  topics. Next, using arithmetics as a synthetic environment, we show that
  language models can separate true and false statements, and generalize
  truthfulness across agents; but only if agents in the training data share a
  truthful generative process that enables the creation of a truthful persona.
  Overall, our findings suggest that models can exploit hierarchical structures
  in the data to learn abstract concepts like truthfulness.
paper_ai_keywords: Large Language Models truthfulness factual misleading information visual corpora truthful persona agents formal writing styles consistent claims inference persona hypothesis synthetic environment true and false statements hierarchical structures abstract concepts