title: Language Model Council: Benchmarking Foundation Models on Highly
    Subjective Tasks by Consensus
summary: The rapid advancement of Large Language Models (LLMs) necessitates robust and
  challenging benchmarks. Leaderboards like Chatbot Arena rank LLMs based on how
  well their responses align with human preferences. However, many tasks such as
  those related to emotional intelligence, creative writing, or persuasiveness,
  are highly subjective and often lack majoritarian human agreement. Judges may
  have irreconcilable disagreements about what constitutes a better response. To
  address the challenge of ranking LLMs on highly subjective tasks, we propose a
  novel benchmarking framework, the Language Model Council (LMC). The LMC
  operates through a democratic process to: 1) formulate a test set through equal
  participation, 2) administer the test among council members, and 3) evaluate
  responses as a collective jury. We deploy a council of 20 newest LLMs on an
  open-ended emotional intelligence task: responding to interpersonal dilemmas.
  Our results show that the LMC produces rankings that are more separable,
  robust, and less biased than those from any individual LLM judge, and is more
  consistent with a human-established leaderboard compared to other benchmarks.
paper_authors._id: 666bcb7b3fc0651814cbfb83
paper_authors.name: Justin Zhao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-14T05:07:00.962Z
paper_authors._id: 6462ac71514ee1645bd1f7f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6462ac71514ee1645bd1f7f7/kw2gwI8pyp6uyT4neqV_H.png
paper_authors.fullname: Justin Zhao
paper_authors.type: user
paper_authors.user: justinxzhao
paper_authors._id: 666bcb7b3fc0651814cbfb84
paper_authors.name: Flor Miriam Plaza-del-Arco
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:28:12.473Z
paper_authors._id: 60f7ec6ad51f9e10a4f422af
paper_authors.avatarUrl: /avatars/12ac5037657cfba4e2b48b08c688a629.svg
paper_authors.fullname: Flor Miriam Plaza-del-Arco
paper_authors.type: user
paper_authors.user: fmplaza
paper_authors._id: 666bcb7b3fc0651814cbfb85
paper_authors.name: Amanda Cercas Curry
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:28:21.671Z
paper_authors._id: 620bacbbc7e4ae30abee681c
paper_authors.avatarUrl: /avatars/60710c4627e8924719f3a1c034a6c644.svg
paper_authors.fullname: Amanda Curry
paper_authors.type: user
paper_authors.user: amandacurry
paper_title: Language Model Council: Benchmarking Foundation Models on Highly
    Subjective Tasks by Consensus
paper_summary: The rapid advancement of Large Language Models (LLMs) necessitates robust and
  challenging benchmarks. Leaderboards like Chatbot Arena rank LLMs based on how
  well their responses align with human preferences. However, many tasks such as
  those related to emotional intelligence, creative writing, or persuasiveness,
  are highly subjective and often lack majoritarian human agreement. Judges may
  have irreconcilable disagreements about what constitutes a better response. To
  address the challenge of ranking LLMs on highly subjective tasks, we propose a
  novel benchmarking framework, the Language Model Council (LMC). The LMC
  operates through a democratic process to: 1) formulate a test set through equal
  participation, 2) administer the test among council members, and 3) evaluate
  responses as a collective jury. We deploy a council of 20 newest LLMs on an
  open-ended emotional intelligence task: responding to interpersonal dilemmas.
  Our results show that the LMC produces rankings that are more separable,
  robust, and less biased than those from any individual LLM judge, and is more
  consistent with a human-established leaderboard compared to other benchmarks.
paper_ai_summary: A democratic council framework improves benchmarking of Large Language Models, particularly for subjective tasks like emotional intelligence, by reducing bias and increasing robustness.
paper_ai_keywords: Large Language Models Chatbot Arena emotional intelligence creative writing persuasiveness democratic process test set council members collective jury