Supervised Pretraining Can Learn In-Context Reinforcement Learning
Large transformer models trained on diverse datasets have shown a remarkable
ability to learn in-context, achieving high few-shot performance on tasks they
were not explicitly trained to solve. In this paper, we study the in-context
learning capabilities of transformers in decision-making problems, i.e.,
reinforcement learning (RL) for bandits and Markov decision processes. To do
so, we introduce and study Decision-Pretrained Transformer (DPT), a supervised
pretraining method where the transformer predicts an optimal action given a
query state and an in-context dataset of interactions, across a diverse set of
tasks. This procedure, while simple, produces a model with several surprising
capabilities. We find that the pretrained transformer can be used to solve a
range of RL problems in-context, exhibiting both exploration online and
conservatism offline, despite not being explicitly trained to do so. The model
also generalizes beyond the pretraining distribution to new tasks and
automatically adapts its decision-making strategies to unknown structure.
Theoretically, we show DPT can be viewed as an efficient implementation of
Bayesian posterior sampling, a provably sample-efficient RL algorithm. We
further leverage this connection to provide guarantees on the regret of the
in-context algorithm yielded by DPT, and prove that it can learn faster than
algorithms used to generate the pretraining data. These results suggest a
promising yet simple path towards instilling strong in-context decision-making
abilities in transformers.
649a84b0a37f2e731852670e
Jonathan N. Lee
claimed_verified
2023-06-29T20:55:56.608Z
649de792e685e40238a07f65
/avatars/96c3ac0b410b85719bffa8a1a9b035b1.svg
Jonathan Lee
user
jnlee
649a84b0a37f2e731852670f
Annie Xie
claimed_verified
2023-06-30T10:23:10.030Z
649e23bb3e5e75047640b3b5
/avatars/019d008a5bd7057b8665e1ef856f13a7.svg
Annie Xie
user
anniexie
649a84b0a37f2e7318526710
Aldo Pacchiano
649a84b0a37f2e7318526711
Yash Chandak
649a84b0a37f2e7318526712
Chelsea Finn
claimed_verified
2023-07-11T07:56:18.636Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
649a84b0a37f2e7318526713
Ofir Nachum
649a84b0a37f2e7318526714
Emma Brunskill
Supervised Pretraining Can Learn In-Context Reinforcement Learning
Large transformer models trained on diverse datasets have shown a remarkable
ability to learn in-context, achieving high few-shot performance on tasks they
were not explicitly trained to solve. In this paper, we study the in-context
learning capabilities of transformers in decision-making problems, i.e.,
reinforcement learning (RL) for bandits and Markov decision processes. To do
so, we introduce and study Decision-Pretrained Transformer (DPT), a supervised
pretraining method where the transformer predicts an optimal action given a
query state and an in-context dataset of interactions, across a diverse set of
tasks. This procedure, while simple, produces a model with several surprising
capabilities. We find that the pretrained transformer can be used to solve a
range of RL problems in-context, exhibiting both exploration online and
conservatism offline, despite not being explicitly trained to do so. The model
also generalizes beyond the pretraining distribution to new tasks and
automatically adapts its decision-making strategies to unknown structure.
Theoretically, we show DPT can be viewed as an efficient implementation of
Bayesian posterior sampling, a provably sample-efficient RL algorithm. We
further leverage this connection to provide guarantees on the regret of the
in-context algorithm yielded by DPT, and prove that it can learn faster than
algorithms used to generate the pretraining data. These results suggest a
promising yet simple path towards instilling strong in-context decision-making
abilities in transformers.
Transformers pretrained with Decision-Pretrained Transformer (DPT) exhibit strong in-context learning capabilities in reinforcement learning, achieving exploration, generalization, and faster learning compared to algorithms used for pretraining.
transformers reinforcement learning bandits Markov decision processes Decision-Pretrained Transformer in-context learning Bayesian posterior sampling regret