title: GLEE: A Unified Framework and Benchmark for Language-based Economic
    Environments
summary: Large Language Models (LLMs) show significant potential in economic and
  strategic interactions, where communication via natural language is often
  prevalent. This raises key questions: Do LLMs behave rationally? Can they mimic
  human behavior? Do they tend to reach an efficient and fair outcome? What is
  the role of natural language in the strategic interaction? How do
  characteristics of the economic environment influence these dynamics? These
  questions become crucial concerning the economic and societal implications of
  integrating LLM-based agents into real-world data-driven systems, such as
  online retail platforms and recommender systems. While the ML community has
  been exploring the potential of LLMs in such multi-agent setups, varying
  assumptions, design choices and evaluation criteria across studies make it
  difficult to draw robust and meaningful conclusions. To address this, we
  introduce a benchmark for standardizing research on two-player, sequential,
  language-based games. Inspired by the economic literature, we define three base
  families of games with consistent parameterization, degrees of freedom and
  economic measures to evaluate agents' performance (self-gain), as well as the
  game outcome (efficiency and fairness). We develop an open-source framework for
  interaction simulation and analysis, and utilize it to collect a dataset of LLM
  vs. LLM interactions across numerous game configurations and an additional
  dataset of human vs. LLM interactions. Through extensive experimentation, we
  demonstrate how our framework and dataset can be used to: (i) compare the
  behavior of LLM-based agents to human players in various economic contexts;
  (ii) evaluate agents in both individual and collective performance measures;
  and (iii) quantify the effect of the economic characteristics of the
  environments on the behavior of agents.
paper_authors._id: 670686055dc3f2a798ad4a5f
paper_authors.name: Eilam Shapira
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T13:46:45.517Z
paper_authors._id: 64802fb6c57f629056c59966
paper_authors.avatarUrl: /avatars/d5ecabaceeba759969855acf512b6649.svg
paper_authors.fullname: Eilam Shapira
paper_authors.type: user
paper_authors.user: EilamSha
paper_authors._id: 670686055dc3f2a798ad4a60
paper_authors.name: Omer Madmon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T14:08:16.799Z
paper_authors._id: 6707a02ae1e3c1a7b6f457b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ZdwhsoEXaMPT1M29lz3uU.png
paper_authors.fullname: Omer Madmon
paper_authors.type: user
paper_authors.user: omermadmon
paper_authors._id: 670686055dc3f2a798ad4a61
paper_authors.name: Itamar Reinman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:22:28.588Z
paper_authors._id: 64e67f74fdb84138027ae2b1
paper_authors.avatarUrl: /avatars/f0bca499a30cfcfa058ac4bb31a08ee3.svg
paper_authors.fullname: Itamar Reinman
paper_authors.type: user
paper_authors.user: ireinman
paper_authors._id: 670686055dc3f2a798ad4a62
paper_authors.name: Samuel Joseph Amouyal
paper_authors._id: 670686055dc3f2a798ad4a63
paper_authors.name: Roi Reichart
paper_authors._id: 670686055dc3f2a798ad4a64
paper_authors.name: Moshe Tennenholtz
paper_title: GLEE: A Unified Framework and Benchmark for Language-based Economic
    Environments
paper_summary: Large Language Models (LLMs) show significant potential in economic and
  strategic interactions, where communication via natural language is often
  prevalent. This raises key questions: Do LLMs behave rationally? Can they mimic
  human behavior? Do they tend to reach an efficient and fair outcome? What is
  the role of natural language in the strategic interaction? How do
  characteristics of the economic environment influence these dynamics? These
  questions become crucial concerning the economic and societal implications of
  integrating LLM-based agents into real-world data-driven systems, such as
  online retail platforms and recommender systems. While the ML community has
  been exploring the potential of LLMs in such multi-agent setups, varying
  assumptions, design choices and evaluation criteria across studies make it
  difficult to draw robust and meaningful conclusions. To address this, we
  introduce a benchmark for standardizing research on two-player, sequential,
  language-based games. Inspired by the economic literature, we define three base
  families of games with consistent parameterization, degrees of freedom and
  economic measures to evaluate agents' performance (self-gain), as well as the
  game outcome (efficiency and fairness). We develop an open-source framework for
  interaction simulation and analysis, and utilize it to collect a dataset of LLM
  vs. LLM interactions across numerous game configurations and an additional
  dataset of human vs. LLM interactions. Through extensive experimentation, we
  demonstrate how our framework and dataset can be used to: (i) compare the
  behavior of LLM-based agents to human players in various economic contexts;
  (ii) evaluate agents in both individual and collective performance measures;
  and (iii) quantify the effect of the economic characteristics of the
  environments on the behavior of agents.
paper_ai_keywords: Large Language Models (LLMs) Economic and strategic complexities Natural language communication Self-rational behavior Human behavior mimicry Efficient and fair behavior Latent diffusion models (LDMs) (mentioned in the example but not in the actual abstract, so not included) Economic environment characteristics Multi-agent setups Two-player, sequential, language-based games Interaction simulation Consistent parameterization Degrees of freedom Economic measures Open-source framework Collective performance measures Individual performance measures Behavior of LLM-based agents Human players interaction Economic characteristics of environments