title: Benchmarking LLMs for Political Science: A United Nations Perspective
summary: Large Language Models (LLMs) have achieved significant advances in natural
  language processing, yet their potential for high-stake political
  decision-making remains largely unexplored. This paper addresses the gap by
  focusing on the application of LLMs to the United Nations (UN) decision-making
  process, where the stakes are particularly high and political decisions can
  have far-reaching consequences. We introduce a novel dataset comprising
  publicly available UN Security Council (UNSC) records from 1994 to 2024,
  including draft resolutions, voting records, and diplomatic speeches. Using
  this dataset, we propose the United Nations Benchmark (UNBench), the first
  comprehensive benchmark designed to evaluate LLMs across four interconnected
  political science tasks: co-penholder judgment, representative voting
  simulation, draft adoption prediction, and representative statement generation.
  These tasks span the three stages of the UN decision-making process--drafting,
  voting, and discussing--and aim to assess LLMs' ability to understand and
  simulate political dynamics. Our experimental analysis demonstrates the
  potential and challenges of applying LLMs in this domain, providing insights
  into their strengths and limitations in political science. This work
  contributes to the growing intersection of AI and political science, opening
  new avenues for research and practical applications in global governance. The
  UNBench Repository can be accessed at:
  https://github.com/yueqingliang1/UNBench.
paper_authors._id: 67bbecd2039a172a715f7b50
paper_authors.name: Yueqing Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:07:13.108Z
paper_authors._id: 650d3ed2e063bef6d9c46da7
paper_authors.avatarUrl: /avatars/d16b3d0d403c1953780f6532e9cf27ad.svg
paper_authors.fullname: Yueqing Liang
paper_authors.type: user
paper_authors.user: yueqingliang
paper_authors._id: 67bbecd2039a172a715f7b51
paper_authors.name: Liangwei Yang
paper_authors._id: 67bbecd2039a172a715f7b52
paper_authors.name: Chen Wang
paper_authors._id: 67bbecd2039a172a715f7b53
paper_authors.name: Congying Xia
paper_authors._id: 67bbecd2039a172a715f7b54
paper_authors.name: Rui Meng
paper_authors._id: 67bbecd2039a172a715f7b55
paper_authors.name: Xiongxiao Xu
paper_authors._id: 67bbecd2039a172a715f7b56
paper_authors.name: Haoran Wang
paper_authors._id: 67bbecd2039a172a715f7b57
paper_authors.name: Ali Payani
paper_authors._id: 67bbecd2039a172a715f7b58
paper_authors.name: Kai Shu
paper_title: Benchmarking LLMs for Political Science: A United Nations Perspective
paper_summary: Large Language Models (LLMs) have achieved significant advances in natural
  language processing, yet their potential for high-stake political
  decision-making remains largely unexplored. This paper addresses the gap by
  focusing on the application of LLMs to the United Nations (UN) decision-making
  process, where the stakes are particularly high and political decisions can
  have far-reaching consequences. We introduce a novel dataset comprising
  publicly available UN Security Council (UNSC) records from 1994 to 2024,
  including draft resolutions, voting records, and diplomatic speeches. Using
  this dataset, we propose the United Nations Benchmark (UNBench), the first
  comprehensive benchmark designed to evaluate LLMs across four interconnected
  political science tasks: co-penholder judgment, representative voting
  simulation, draft adoption prediction, and representative statement generation.
  These tasks span the three stages of the UN decision-making process--drafting,
  voting, and discussing--and aim to assess LLMs' ability to understand and
  simulate political dynamics. Our experimental analysis demonstrates the
  potential and challenges of applying LLMs in this domain, providing insights
  into their strengths and limitations in political science. This work
  contributes to the growing intersection of AI and political science, opening
  new avenues for research and practical applications in global governance. The
  UNBench Repository can be accessed at:
  https://github.com/yueqingliang1/UNBench.
paper_ai_keywords: Large Language Models (LLMs) natural language processing political decision-making United Nations (UN) United Nations Security Council (UNSC) publicly available UN Security Council records draft resolutions voting records diplomatic speeches United Nations Benchmark (UNBench) co-penholder judgment representative voting simulation draft adoption prediction representative statement generation UN decision-making process political dynamics