Zero-shot Model-based Reinforcement Learning using Large Language Models
The emerging zero-shot capabilities of Large Language Models (LLMs) have led
to their applications in areas extending well beyond natural language
processing tasks. In reinforcement learning, while LLMs have been extensively
used in text-based environments, their integration with continuous state spaces
remains understudied. In this paper, we investigate how pre-trained LLMs can be
leveraged to predict in context the dynamics of continuous Markov decision
processes. We identify handling multivariate data and incorporating the control
signal as key challenges that limit the potential of LLMs' deployment in this
setup and propose Disentangled In-Context Learning (DICL) to address them. We
present proof-of-concept applications in two reinforcement learning settings:
model-based policy evaluation and data-augmented off-policy reinforcement
learning, supported by theoretical analysis of the proposed methods. Our
experiments further demonstrate that our approach produces well-calibrated
uncertainty estimates. We release the code at
https://github.com/abenechehab/dicl.
67176ecfd73dbe05349195a0
Abdelhakim Benechehab
extracted_confirmed
2024-11-12T13:22:27.661Z
621d59ebd3df05d67132e8d9
https://cdn-avatars.huggingface.co/v1/production/uploads/621d59ebd3df05d67132e8d9/0gPfPTRKKnz5kq0InTqm5.jpeg
Abdelhakim Benechehab
user
abenechehab
67176ecfd73dbe05349195a1
Youssef Attia El Hili
claimed_verified
2025-02-18T09:36:43.902Z
6389b24e51b7825b34286e8b
/avatars/3d10ef7ea849348526ef95c04d21ed7a.svg
Youssef Attia El Hili
user
Yattia
67176ecfd73dbe05349195a2
Ambroise Odonnat
admin_assigned
2024-10-22T11:31:52.056Z
6661f79ac36ae4c83f3213e4
/avatars/2e48f052fdf37b5b06d101a6a3232eea.svg
Ambroise Odonnat 
user
ambroiseodt
67176ecfd73dbe05349195a3
Oussama Zekri
claimed_verified
2024-10-22T09:32:57.062Z
651e90f4b395a05dff63fe9e
https://cdn-avatars.huggingface.co/v1/production/uploads/651e90f4b395a05dff63fe9e/51g5erCgvJ43szg26EQ_M.png
Oussama Zekri
user
Xssama
67176ecfd73dbe05349195a4
Albert Thomas
admin_assigned
2024-10-22T11:31:34.241Z
639789c20f1ac9c2f34a59f7
/avatars/fd73c93d50264d14f532ff52bc0d48f7.svg
Albert Thomas
user
albert9000
67176ecfd73dbe05349195a5
Giuseppe Paolo
admin_assigned
2024-10-22T11:31:27.734Z
65e98cd8e19214e9d151f29e
https://cdn-avatars.huggingface.co/v1/production/uploads/65e98cd8e19214e9d151f29e/XjQzoVgKVzv8AZBWFQnHz.jpeg
Giuseppe Paolo
user
GPaolo
67176ecfd73dbe05349195a6
Maurizio Filippone
67176ecfd73dbe05349195a7
Ievgen Redko
admin_assigned
2024-10-22T11:35:17.552Z
643d3c9c043254ce1092d195
/avatars/d44f504d62888e1e8b58f5723147ae9b.svg
Ievgen R
user
ievred
67176ecfd73dbe05349195a8
Balázs Kégl
Zero-shot Model-based Reinforcement Learning using Large Language Models
The emerging zero-shot capabilities of Large Language Models (LLMs) have led
to their applications in areas extending well beyond natural language
processing tasks. In reinforcement learning, while LLMs have been extensively
used in text-based environments, their integration with continuous state spaces
remains understudied. In this paper, we investigate how pre-trained LLMs can be
leveraged to predict in context the dynamics of continuous Markov decision
processes. We identify handling multivariate data and incorporating the control
signal as key challenges that limit the potential of LLMs' deployment in this
setup and propose Disentangled In-Context Learning (DICL) to address them. We
present proof-of-concept applications in two reinforcement learning settings:
model-based policy evaluation and data-augmented off-policy reinforcement
learning, supported by theoretical analysis of the proposed methods. Our
experiments further demonstrate that our approach produces well-calibrated
uncertainty estimates. We release the code at
https://github.com/abenechehab/dicl.
LLMs are applied to predict dynamics in continuous Markov decision processes using Disentangled In-Context Learning to handle multivariate data and control signals, showing well-calibrated uncertainty estimates.
Large Language Models zero-shot capabilities reinforcement learning continuous state spaces pre-trained LLMs in context dynamics prediction continuous Markov decision processes Disentangled In-Context Learning model-based policy evaluation data-augmented off-policy reinforcement learning uncertainty estimates