title: Towards Measuring the Representation of Subjective Global Opinions in
    Language Models
summary: Large language models (LLMs) may not equitably represent diverse global
  perspectives on societal issues. In this paper, we develop a quantitative
  framework to evaluate whose opinions model-generated responses are more similar
  to. We first build a dataset, GlobalOpinionQA, comprised of questions and
  answers from cross-national surveys designed to capture diverse opinions on
  global issues across different countries. Next, we define a metric that
  quantifies the similarity between LLM-generated survey responses and human
  responses, conditioned on country. With our framework, we run three experiments
  on an LLM trained to be helpful, honest, and harmless with Constitutional AI.
  By default, LLM responses tend to be more similar to the opinions of certain
  populations, such as those from the USA, and some European and South American
  countries, highlighting the potential for biases. When we prompt the model to
  consider a particular country's perspective, responses shift to be more similar
  to the opinions of the prompted populations, but can reflect harmful cultural
  stereotypes. When we translate GlobalOpinionQA questions to a target language,
  the model's responses do not necessarily become the most similar to the
  opinions of speakers of those languages. We release our dataset for others to
  use and build on. Our data is at
  https://huggingface.co/datasets/Anthropic/llm_global_opinions. We also provide
  an interactive visualization at https://llmglobalvalues.anthropic.com.
paper_authors._id: 649cd399dd87dd9ef76babcd
paper_authors.name: Esin Durmus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:32:29.464Z
paper_authors._id: 627dd7dbba19b19eff6aaa46
paper_authors.avatarUrl: /avatars/6e43025ccb96e48339d1db477da043b0.svg
paper_authors.fullname: Esin Durmus
paper_authors.type: user
paper_authors.user: esdurmus
paper_authors._id: 649cd399dd87dd9ef76babce
paper_authors.name: Karina Nyugen
paper_authors._id: 649cd399dd87dd9ef76babcf
paper_authors.name: Thomas I. Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:33:29.035Z
paper_authors._id: 627c1e283974b0ed6b2d0f64
paper_authors.avatarUrl: /avatars/9a828c6057ab4ea3d3cd8d939d12c2ae.svg
paper_authors.fullname: Thomas I. Liao
paper_authors.type: user
paper_authors.user: til
paper_authors._id: 649cd399dd87dd9ef76babd0
paper_authors.name: Nicholas Schiefer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:33:56.706Z
paper_authors._id: 63a246e2931d2c13ad842a59
paper_authors.avatarUrl: /avatars/9ac4f3d01c293ce97507be402dfa688c.svg
paper_authors.fullname: Nicholas Schiefer
paper_authors.type: user
paper_authors.user: nschiefer
paper_authors._id: 649cd399dd87dd9ef76babd1
paper_authors.name: Amanda Askell
paper_authors._id: 649cd399dd87dd9ef76babd2
paper_authors.name: Anton Bakhtin
paper_authors._id: 649cd399dd87dd9ef76babd3
paper_authors.name: Carol Chen
paper_authors._id: 649cd399dd87dd9ef76babd4
paper_authors.name: Zac Hatfield-Dodds
paper_authors._id: 649cd399dd87dd9ef76babd5
paper_authors.name: Danny Hernandez
paper_authors._id: 649cd399dd87dd9ef76babd6
paper_authors.name: Nicholas Joseph
paper_authors._id: 649cd399dd87dd9ef76babd7
paper_authors.name: Liane Lovitt
paper_authors._id: 649cd399dd87dd9ef76babd8
paper_authors.name: Sam McCandlish
paper_authors._id: 649cd399dd87dd9ef76babd9
paper_authors.name: Orowa Sikder
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:36:14.959Z
paper_authors._id: 62eab58af602390a049ae5a5
paper_authors.avatarUrl: /avatars/73328c7ac59d072756ebad7c0548789b.svg
paper_authors.fullname: Orowa Sikder
paper_authors.type: user
paper_authors.user: Orowa
paper_authors._id: 649cd399dd87dd9ef76babda
paper_authors.name: Alex Tamkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:36:54.419Z
paper_authors._id: 5fc18c42ea82dd667bb0ffaf
paper_authors.avatarUrl: /avatars/ae98f82442b560de247d2354cc364398.svg
paper_authors.fullname: Alex Tamkin
paper_authors.type: user
paper_authors.user: alextamkin
paper_authors._id: 649cd399dd87dd9ef76babdb
paper_authors.name: Janel Thamkul
paper_authors._id: 649cd399dd87dd9ef76babdc
paper_authors.name: Jared Kaplan
paper_authors._id: 649cd399dd87dd9ef76babdd
paper_authors.name: Jack Clark
paper_authors._id: 649cd399dd87dd9ef76babde
paper_authors.name: Deep Ganguli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:38:13.949Z
paper_authors._id: 63924e6b2bfa48413aa0a4ad
paper_authors.avatarUrl: /avatars/f8219c475af6995b6fb9ebe727c5be43.svg
paper_authors.fullname: Deep Ganguli
paper_authors.type: user
paper_authors.user: dganguli
paper_title: Towards Measuring the Representation of Subjective Global Opinions in
    Language Models
paper_summary: Large language models (LLMs) may not equitably represent diverse global
  perspectives on societal issues. In this paper, we develop a quantitative
  framework to evaluate whose opinions model-generated responses are more similar
  to. We first build a dataset, GlobalOpinionQA, comprised of questions and
  answers from cross-national surveys designed to capture diverse opinions on
  global issues across different countries. Next, we define a metric that
  quantifies the similarity between LLM-generated survey responses and human
  responses, conditioned on country. With our framework, we run three experiments
  on an LLM trained to be helpful, honest, and harmless with Constitutional AI.
  By default, LLM responses tend to be more similar to the opinions of certain
  populations, such as those from the USA, and some European and South American
  countries, highlighting the potential for biases. When we prompt the model to
  consider a particular country's perspective, responses shift to be more similar
  to the opinions of the prompted populations, but can reflect harmful cultural
  stereotypes. When we translate GlobalOpinionQA questions to a target language,
  the model's responses do not necessarily become the most similar to the
  opinions of speakers of those languages. We release our dataset for others to
  use and build on. Our data is at
  https://huggingface.co/datasets/Anthropic/llm_global_opinions. We also provide
  an interactive visualization at https://llmglobalvalues.anthropic.com.
paper_ai_summary: A quantitative framework evaluates the similarities between large language model responses and human perspectives on global issues, highlighting biases and limitations.
paper_ai_keywords: large language models GlobalOpinionQA cross-national surveys metric Constitutional AI country perspectives cultural stereotypes interactive visualization