title: Zero-shot Model-based Reinforcement Learning using Large Language Models
summary: The emerging zero-shot capabilities of Large Language Models (LLMs) have led
  to their applications in areas extending well beyond natural language
  processing tasks. In reinforcement learning, while LLMs have been extensively
  used in text-based environments, their integration with continuous state spaces
  remains understudied. In this paper, we investigate how pre-trained LLMs can be
  leveraged to predict in context the dynamics of continuous Markov decision
  processes. We identify handling multivariate data and incorporating the control
  signal as key challenges that limit the potential of LLMs' deployment in this
  setup and propose Disentangled In-Context Learning (DICL) to address them. We
  present proof-of-concept applications in two reinforcement learning settings:
  model-based policy evaluation and data-augmented off-policy reinforcement
  learning, supported by theoretical analysis of the proposed methods. Our
  experiments further demonstrate that our approach produces well-calibrated
  uncertainty estimates. We release the code at
  https://github.com/abenechehab/dicl.
paper_authors._id: 67176ecfd73dbe05349195a0
paper_authors.name: Abdelhakim Benechehab
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-12T13:22:27.661Z
paper_authors._id: 621d59ebd3df05d67132e8d9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621d59ebd3df05d67132e8d9/0gPfPTRKKnz5kq0InTqm5.jpeg
paper_authors.fullname: Abdelhakim Benechehab
paper_authors.type: user
paper_authors.user: abenechehab
paper_authors._id: 67176ecfd73dbe05349195a1
paper_authors.name: Youssef Attia El Hili
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:36:43.902Z
paper_authors._id: 6389b24e51b7825b34286e8b
paper_authors.avatarUrl: /avatars/3d10ef7ea849348526ef95c04d21ed7a.svg
paper_authors.fullname: Youssef Attia El Hili
paper_authors.type: user
paper_authors.user: Yattia
paper_authors._id: 67176ecfd73dbe05349195a2
paper_authors.name: Ambroise Odonnat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T11:31:52.056Z
paper_authors._id: 6661f79ac36ae4c83f3213e4
paper_authors.avatarUrl: /avatars/2e48f052fdf37b5b06d101a6a3232eea.svg
paper_authors.fullname: Ambroise Odonnat 
paper_authors.type: user
paper_authors.user: ambroiseodt
paper_authors._id: 67176ecfd73dbe05349195a3
paper_authors.name: Oussama Zekri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T09:32:57.062Z
paper_authors._id: 651e90f4b395a05dff63fe9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651e90f4b395a05dff63fe9e/51g5erCgvJ43szg26EQ_M.png
paper_authors.fullname: Oussama Zekri
paper_authors.type: user
paper_authors.user: Xssama
paper_authors._id: 67176ecfd73dbe05349195a4
paper_authors.name: Albert Thomas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T11:31:34.241Z
paper_authors._id: 639789c20f1ac9c2f34a59f7
paper_authors.avatarUrl: /avatars/fd73c93d50264d14f532ff52bc0d48f7.svg
paper_authors.fullname: Albert Thomas
paper_authors.type: user
paper_authors.user: albert9000
paper_authors._id: 67176ecfd73dbe05349195a5
paper_authors.name: Giuseppe Paolo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T11:31:27.734Z
paper_authors._id: 65e98cd8e19214e9d151f29e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65e98cd8e19214e9d151f29e/XjQzoVgKVzv8AZBWFQnHz.jpeg
paper_authors.fullname: Giuseppe Paolo
paper_authors.type: user
paper_authors.user: GPaolo
paper_authors._id: 67176ecfd73dbe05349195a6
paper_authors.name: Maurizio Filippone
paper_authors._id: 67176ecfd73dbe05349195a7
paper_authors.name: Ievgen Redko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T11:35:17.552Z
paper_authors._id: 643d3c9c043254ce1092d195
paper_authors.avatarUrl: /avatars/d44f504d62888e1e8b58f5723147ae9b.svg
paper_authors.fullname: Ievgen R
paper_authors.type: user
paper_authors.user: ievred
paper_authors._id: 67176ecfd73dbe05349195a8
paper_authors.name: Balázs Kégl
paper_title: Zero-shot Model-based Reinforcement Learning using Large Language Models
paper_summary: The emerging zero-shot capabilities of Large Language Models (LLMs) have led
  to their applications in areas extending well beyond natural language
  processing tasks. In reinforcement learning, while LLMs have been extensively
  used in text-based environments, their integration with continuous state spaces
  remains understudied. In this paper, we investigate how pre-trained LLMs can be
  leveraged to predict in context the dynamics of continuous Markov decision
  processes. We identify handling multivariate data and incorporating the control
  signal as key challenges that limit the potential of LLMs' deployment in this
  setup and propose Disentangled In-Context Learning (DICL) to address them. We
  present proof-of-concept applications in two reinforcement learning settings:
  model-based policy evaluation and data-augmented off-policy reinforcement
  learning, supported by theoretical analysis of the proposed methods. Our
  experiments further demonstrate that our approach produces well-calibrated
  uncertainty estimates. We release the code at
  https://github.com/abenechehab/dicl.
paper_ai_summary: LLMs are applied to predict dynamics in continuous Markov decision processes using Disentangled In-Context Learning to handle multivariate data and control signals, showing well-calibrated uncertainty estimates.
paper_ai_keywords: Large Language Models zero-shot capabilities reinforcement learning continuous state spaces pre-trained LLMs in context dynamics prediction continuous Markov decision processes Disentangled In-Context Learning model-based policy evaluation data-augmented off-policy reinforcement learning uncertainty estimates