title: Can large language models explore in-context?
summary: We investigate the extent to which contemporary Large Language Models (LLMs)
  can engage in exploration, a core capability in reinforcement learning and
  decision making. We focus on native performance of existing LLMs, without
  training interventions. We deploy LLMs as agents in simple multi-armed bandit
  environments, specifying the environment description and interaction history
  entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5,
  GPT-4, and Llama2, using a variety of prompt designs, and find that the models
  do not robustly engage in exploration without substantial interventions: i)
  Across all of our experiments, only one configuration resulted in satisfactory
  exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally
  summarized interaction history, presented as sufficient statistics; ii) All
  other configurations did not result in robust exploratory behavior, including
  those with chain-of-thought reasoning but unsummarized history. Although these
  findings can be interpreted positively, they suggest that external
  summarization -- which may not be possible in more complex settings -- is
  important for obtaining desirable behavior from LLM agents. We conclude that
  non-trivial algorithmic interventions, such as fine-tuning or dataset curation,
  may be required to empower LLM-based decision making agents in complex
  settings.
paper_authors._id: 6600d47a8270811586f9fe3a
paper_authors.name: Akshay Krishnamurthy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:11:50.638Z
paper_authors._id: 64b71a5efa7eabaae5f35442
paper_authors.avatarUrl: /avatars/347f58f15820630e96908a54dd25ed81.svg
paper_authors.fullname: Akshay Krishnamurthy
paper_authors.type: user
paper_authors.user: akshaykrish
paper_authors._id: 6600d47a8270811586f9fe3b
paper_authors.name: Keegan Harris
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-25T11:42:48.432Z
paper_authors._id: 64b85a44cf14c2fabe8e929f
paper_authors.avatarUrl: /avatars/21e106c247bf6f027c0384dec8910426.svg
paper_authors.fullname: Keegan Harris
paper_authors.type: user
paper_authors.user: keeganharris
paper_authors._id: 6600d47a8270811586f9fe3c
paper_authors.name: Dylan J. Foster
paper_authors._id: 6600d47a8270811586f9fe3d
paper_authors.name: Cyril Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:12:55.961Z
paper_authors._id: 63a6a3a93d8cc9b1183c6593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a6a3a93d8cc9b1183c6593/UF-9ostqBHiAGDOuEtIw_.png
paper_authors.fullname: Cyril Zhang
paper_authors.type: user
paper_authors.user: cyrilzhang
paper_authors._id: 6600d47a8270811586f9fe3e
paper_authors.name: Aleksandrs Slivkins
paper_title: Can large language models explore in-context?
paper_summary: We investigate the extent to which contemporary Large Language Models (LLMs)
  can engage in exploration, a core capability in reinforcement learning and
  decision making. We focus on native performance of existing LLMs, without
  training interventions. We deploy LLMs as agents in simple multi-armed bandit
  environments, specifying the environment description and interaction history
  entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5,
  GPT-4, and Llama2, using a variety of prompt designs, and find that the models
  do not robustly engage in exploration without substantial interventions: i)
  Across all of our experiments, only one configuration resulted in satisfactory
  exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally
  summarized interaction history, presented as sufficient statistics; ii) All
  other configurations did not result in robust exploratory behavior, including
  those with chain-of-thought reasoning but unsummarized history. Although these
  findings can be interpreted positively, they suggest that external
  summarization -- which may not be possible in more complex settings -- is
  important for obtaining desirable behavior from LLM agents. We conclude that
  non-trivial algorithmic interventions, such as fine-tuning or dataset curation,
  may be required to empower LLM-based decision making agents in complex
  settings.
paper_ai_summary: Experimentation with LLMs in multi-armed bandit environments reveals that robust exploration requires specific prompts, external summarization of history, or algorithmic interventions.
paper_ai_keywords: Large Language Models LLMs reinforcement learning decision making multi-armed bandit environments chain-of-thought reasoning externally summarized interaction history fine-tuning dataset curation