title: PokerBench: Training Large Language Models to become Professional Poker
    Players
summary: We introduce PokerBench - a benchmark for evaluating the poker-playing
  abilities of large language models (LLMs). As LLMs excel in traditional NLP
  tasks, their application to complex, strategic games like poker poses a new
  challenge. Poker, an incomplete information game, demands a multitude of skills
  such as mathematics, reasoning, planning, strategy, and a deep understanding of
  game theory and human psychology. This makes Poker the ideal next frontier for
  large language models. PokerBench consists of a comprehensive compilation of
  11,000 most important scenarios, split between pre-flop and post-flop play,
  developed in collaboration with trained poker players. We evaluate prominent
  models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models,
  finding that all state-of-the-art LLMs underperform in playing optimal poker.
  However, after fine-tuning, these models show marked improvements. We validate
  PokerBench by having models with different scores compete with each other,
  demonstrating that higher scores on PokerBench lead to higher win rates in
  actual poker games. Through gameplay between our fine-tuned model and GPT-4, we
  also identify limitations of simple supervised fine-tuning for learning optimal
  playing strategy, suggesting the need for more advanced methodologies for
  effectively training language models to excel in games. PokerBench thus
  presents a unique benchmark for a quick and reliable evaluation of the
  poker-playing ability of LLMs as well as a comprehensive benchmark to study the
  progress of LLMs in complex game-playing scenarios. The dataset and code will
  be made available at: https://github.com/pokerllm/pokerbench.
paper_authors._id: 6787367e42691fc7dc545292
paper_authors.name: Richard Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:49:44.178Z
paper_authors._id: 6596f6b992afb150dde380cb
paper_authors.avatarUrl: /avatars/f069bd55f288e1f008a476e7b6e3f150.svg
paper_authors.fullname: Richard Zhuang
paper_authors.type: user
paper_authors.user: RZ412
paper_authors._id: 6787367e42691fc7dc545293
paper_authors.name: Akshat Gupta
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-15T04:19:21.989Z
paper_authors._id: 64e8f4a24f3f7b0b84834315
paper_authors.avatarUrl: /avatars/242bb68c7ccffe5061c2d1c229ea3b0b.svg
paper_authors.fullname: Akshat Gupta
paper_authors.type: user
paper_authors.user: akshat57
paper_authors._id: 6787367e42691fc7dc545294
paper_authors.name: Richard Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:49:50.389Z
paper_authors._id: 63ff98b4b09f82a81a1f07c5
paper_authors.avatarUrl: /avatars/691c6a5e6946c225d2f3ac6d97a8dd52.svg
paper_authors.fullname: Yang
paper_authors.type: user
paper_authors.user: RichardYang
paper_authors._id: 6787367e42691fc7dc545295
paper_authors.name: Aniket Rahane
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:50:05.690Z
paper_authors._id: 65971d98260bcd5fb5225889
paper_authors.avatarUrl: /avatars/494c10ebcc084d1bdfb894e160c6883a.svg
paper_authors.fullname: Aniket Rahane
paper_authors.type: user
paper_authors.user: aniketarahane
paper_authors._id: 6787367e42691fc7dc545296
paper_authors.name: Zhengyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:50:11.928Z
paper_authors._id: 64807bfbf5be39206aef41b5
paper_authors.avatarUrl: /avatars/9a8d037f117223326520c5b522180487.svg
paper_authors.fullname: Zhengyu Li
paper_authors.type: user
paper_authors.user: LZY729
paper_authors._id: 6787367e42691fc7dc545297
paper_authors.name: Gopala Anumanchipalli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T16:50:19.727Z
paper_authors._id: 60523e4aa7226b25aaeea2b8
paper_authors.avatarUrl: /avatars/316ca348da91ebced86991f36150c959.svg
paper_authors.fullname: Gopala Anumanchipalli
paper_authors.type: user
paper_authors.user: gopalakr
paper_title: PokerBench: Training Large Language Models to become Professional Poker
    Players
paper_summary: We introduce PokerBench - a benchmark for evaluating the poker-playing
  abilities of large language models (LLMs). As LLMs excel in traditional NLP
  tasks, their application to complex, strategic games like poker poses a new
  challenge. Poker, an incomplete information game, demands a multitude of skills
  such as mathematics, reasoning, planning, strategy, and a deep understanding of
  game theory and human psychology. This makes Poker the ideal next frontier for
  large language models. PokerBench consists of a comprehensive compilation of
  11,000 most important scenarios, split between pre-flop and post-flop play,
  developed in collaboration with trained poker players. We evaluate prominent
  models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models,
  finding that all state-of-the-art LLMs underperform in playing optimal poker.
  However, after fine-tuning, these models show marked improvements. We validate
  PokerBench by having models with different scores compete with each other,
  demonstrating that higher scores on PokerBench lead to higher win rates in
  actual poker games. Through gameplay between our fine-tuned model and GPT-4, we
  also identify limitations of simple supervised fine-tuning for learning optimal
  playing strategy, suggesting the need for more advanced methodologies for
  effectively training language models to excel in games. PokerBench thus
  presents a unique benchmark for a quick and reliable evaluation of the
  poker-playing ability of LLMs as well as a comprehensive benchmark to study the
  progress of LLMs in complex game-playing scenarios. The dataset and code will
  be made available at: https://github.com/pokerllm/pokerbench.
paper_ai_summary: PokerBench evaluates the poker-playing abilities of large language models, demonstrating that fine-tuning is necessary for these models to perform optimally in such complex games.
paper_ai_keywords: large language models LLMs poker incomplete information game game theory human psychology PokerBench GPT-4 ChatGPT 3.5 Llama Gemma pre-flop post-flop fine-tuning gameplay win rates