PokerBench: Training Large Language Models to become Professional Poker
  Players
We introduce PokerBench - a benchmark for evaluating the poker-playing
abilities of large language models (LLMs). As LLMs excel in traditional NLP
tasks, their application to complex, strategic games like poker poses a new
challenge. Poker, an incomplete information game, demands a multitude of skills
such as mathematics, reasoning, planning, strategy, and a deep understanding of
game theory and human psychology. This makes Poker the ideal next frontier for
large language models. PokerBench consists of a comprehensive compilation of
11,000 most important scenarios, split between pre-flop and post-flop play,
developed in collaboration with trained poker players. We evaluate prominent
models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models,
finding that all state-of-the-art LLMs underperform in playing optimal poker.
However, after fine-tuning, these models show marked improvements. We validate
PokerBench by having models with different scores compete with each other,
demonstrating that higher scores on PokerBench lead to higher win rates in
actual poker games. Through gameplay between our fine-tuned model and GPT-4, we
also identify limitations of simple supervised fine-tuning for learning optimal
playing strategy, suggesting the need for more advanced methodologies for
effectively training language models to excel in games. PokerBench thus
presents a unique benchmark for a quick and reliable evaluation of the
poker-playing ability of LLMs as well as a comprehensive benchmark to study the
progress of LLMs in complex game-playing scenarios. The dataset and code will
be made available at: https://github.com/pokerllm/pokerbench.
6787367e42691fc7dc545292
Richard Zhuang
admin_assigned
2025-01-15T16:49:44.178Z
6596f6b992afb150dde380cb
/avatars/f069bd55f288e1f008a476e7b6e3f150.svg
Richard Zhuang
user
RZ412
6787367e42691fc7dc545293
Akshat Gupta
extracted_confirmed
2025-01-15T04:19:21.989Z
64e8f4a24f3f7b0b84834315
/avatars/242bb68c7ccffe5061c2d1c229ea3b0b.svg
Akshat Gupta
user
akshat57
6787367e42691fc7dc545294
Richard Yang
admin_assigned
2025-01-15T16:49:50.389Z
63ff98b4b09f82a81a1f07c5
/avatars/691c6a5e6946c225d2f3ac6d97a8dd52.svg
Yang
user
RichardYang
6787367e42691fc7dc545295
Aniket Rahane
admin_assigned
2025-01-15T16:50:05.690Z
65971d98260bcd5fb5225889
/avatars/494c10ebcc084d1bdfb894e160c6883a.svg
Aniket Rahane
user
aniketarahane
6787367e42691fc7dc545296
Zhengyu Li
admin_assigned
2025-01-15T16:50:11.928Z
64807bfbf5be39206aef41b5
/avatars/9a8d037f117223326520c5b522180487.svg
Zhengyu Li
user
LZY729
6787367e42691fc7dc545297
Gopala Anumanchipalli
admin_assigned
2025-01-15T16:50:19.727Z
60523e4aa7226b25aaeea2b8
/avatars/316ca348da91ebced86991f36150c959.svg
Gopala Anumanchipalli
user
gopalakr
PokerBench: Training Large Language Models to become Professional Poker
  Players
We introduce PokerBench - a benchmark for evaluating the poker-playing
abilities of large language models (LLMs). As LLMs excel in traditional NLP
tasks, their application to complex, strategic games like poker poses a new
challenge. Poker, an incomplete information game, demands a multitude of skills
such as mathematics, reasoning, planning, strategy, and a deep understanding of
game theory and human psychology. This makes Poker the ideal next frontier for
large language models. PokerBench consists of a comprehensive compilation of
11,000 most important scenarios, split between pre-flop and post-flop play,
developed in collaboration with trained poker players. We evaluate prominent
models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models,
finding that all state-of-the-art LLMs underperform in playing optimal poker.
However, after fine-tuning, these models show marked improvements. We validate
PokerBench by having models with different scores compete with each other,
demonstrating that higher scores on PokerBench lead to higher win rates in
actual poker games. Through gameplay between our fine-tuned model and GPT-4, we
also identify limitations of simple supervised fine-tuning for learning optimal
playing strategy, suggesting the need for more advanced methodologies for
effectively training language models to excel in games. PokerBench thus
presents a unique benchmark for a quick and reliable evaluation of the
poker-playing ability of LLMs as well as a comprehensive benchmark to study the
progress of LLMs in complex game-playing scenarios. The dataset and code will
be made available at: https://github.com/pokerllm/pokerbench.
PokerBench evaluates the poker-playing abilities of large language models, demonstrating that fine-tuning is necessary for these models to perform optimally in such complex games.
large language models LLMs poker incomplete information game game theory human psychology PokerBench GPT-4 ChatGPT 3.5 Llama Gemma pre-flop post-flop fine-tuning gameplay win rates