title: SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage
    Leveraging Generative Models
summary: Stereotype benchmark datasets are crucial to detect and mitigate social
  stereotypes about groups of people in NLP models. However, existing datasets
  are limited in size and coverage, and are largely restricted to stereotypes
  prevalent in the Western society. This is especially problematic as language
  technologies gain hold across the globe. To address this gap, we present
  SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative
  capabilities of large language models such as PaLM, and GPT-3, and leveraging a
  globally diverse rater pool to validate the prevalence of those stereotypes in
  society. SeeGULL is in English, and contains stereotypes about identity groups
  spanning 178 countries across 8 different geo-political regions across 6
  continents, as well as state-level identities within the US and India. We also
  include fine-grained offensiveness scores for different stereotypes and
  demonstrate their global disparities. Furthermore, we include comparative
  annotations about the same groups by annotators living in the region vs. those
  that are based in North America, and demonstrate that within-region stereotypes
  about groups differ from those prevalent in North America. CONTENT WARNING:
  This paper contains stereotype examples that may be offensive.
paper_authors._id: 646aeaf3a5dd10c9a4a2ff86
paper_authors.name: Akshita Jha
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-05-24T17:27:04.949Z
paper_authors._id: 646e48d80867c99c2d3dfabb
paper_authors.avatarUrl: /avatars/9f34ada512447823a8718cfd4d5bfd16.svg
paper_authors.fullname: Akshita Jha
paper_authors.type: user
paper_authors.user: akshitajha
paper_authors._id: 646aeaf3a5dd10c9a4a2ff87
paper_authors.name: Aida Davani
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-19T18:48:09.297Z
paper_authors._id: 682b7cce8f72754097083a48
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/yEKYS5pT2HxXHHERCsBu7.png
paper_authors.fullname: Aida Davani
paper_authors.type: user
paper_authors.user: aidamd
paper_authors._id: 646aeaf3a5dd10c9a4a2ff88
paper_authors.name: Chandan K. Reddy
paper_authors._id: 646aeaf3a5dd10c9a4a2ff89
paper_authors.name: Shachi Dave
paper_authors._id: 646aeaf3a5dd10c9a4a2ff8a
paper_authors.name: Vinodkumar Prabhakaran
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-05-22T04:09:24.147Z
paper_authors._id: 60c2b4cf22a9d3ec6810297b
paper_authors.avatarUrl: /avatars/52d8fbda4233da69e406814c278dca02.svg
paper_authors.fullname: Vinodkumar Prabhakaran
paper_authors.type: user
paper_authors.user: vinodkpg
paper_authors._id: 646aeaf3a5dd10c9a4a2ff8b
paper_authors.name: Sunipa Dev
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-05-22T04:09:24.147Z
paper_authors._id: 62e82ec840bd989bb71853ec
paper_authors.avatarUrl: /avatars/c194747a796a2d61e6e5e793f9fefb2d.svg
paper_authors.fullname: Sunipa Dev
paper_authors.type: user
paper_authors.user: sunipadev
paper_title: SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage
    Leveraging Generative Models
paper_summary: Stereotype benchmark datasets are crucial to detect and mitigate social
  stereotypes about groups of people in NLP models. However, existing datasets
  are limited in size and coverage, and are largely restricted to stereotypes
  prevalent in the Western society. This is especially problematic as language
  technologies gain hold across the globe. To address this gap, we present
  SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative
  capabilities of large language models such as PaLM, and GPT-3, and leveraging a
  globally diverse rater pool to validate the prevalence of those stereotypes in
  society. SeeGULL is in English, and contains stereotypes about identity groups
  spanning 178 countries across 8 different geo-political regions across 6
  continents, as well as state-level identities within the US and India. We also
  include fine-grained offensiveness scores for different stereotypes and
  demonstrate their global disparities. Furthermore, we include comparative
  annotations about the same groups by annotators living in the region vs. those
  that are based in North America, and demonstrate that within-region stereotypes
  about groups differ from those prevalent in North America. CONTENT WARNING:
  This paper contains stereotype examples that may be offensive.
paper_ai_summary: SeeGULL is a globally diverse stereotype dataset built using large language models, spanning 178 countries and including offensiveness scores and regional comparisons.
paper_ai_keywords: generative capabilities large language models stereotype dataset globally diverse offensiveness scores regional comparisons