Unleashing the Power of Pre-trained Language Models for Offline
  Reinforcement Learning
Offline reinforcement learning (RL) aims to find a near-optimal policy using
pre-collected datasets. In real-world scenarios, data collection could be
costly and risky; therefore, offline RL becomes particularly challenging when
the in-domain data is limited. Given recent advances in Large Language Models
(LLMs) and their few-shot learning prowess, this paper introduces
Language Models for Motion Control (LaMo), a
general framework based on Decision Transformers to effectively use pre-trained
Language Models (LMs) for offline RL. Our framework highlights four crucial
components: (1) Initializing Decision Transformers with sequentially
pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to
full-weight fine-tuning, to combine the pre-trained knowledge from LMs and
in-domain knowledge effectively, (3) using the non-linear MLP transformation
instead of linear projections, to generate embeddings, and (4) integrating an
auxiliary language prediction loss during fine-tuning to stabilize the LMs and
retain their original abilities on languages. Empirical results indicate
LaMo achieves state-of-the-art performance in sparse-reward tasks
and closes the gap between value-based offline RL methods and decision
transformers in dense-reward tasks. In particular, our method demonstrates
superior performance in scenarios with limited data samples. Our project
website is https://lamo2023.github.io
6541cb8ac446a9a48bbe3bcf
Ruizhe Shi
6541cb8ac446a9a48bbe3bd0
Yuyao Liu
admin_assigned
2023-11-02T15:43:30.529Z
6427a3bad039ee7fa4e354f4
/avatars/48640c908ab4c7a35c349a467db6e336.svg
Yuyao Liu
user
nature21
6541cb8ac446a9a48bbe3bd1
Yanjie Ze
admin_assigned
2023-11-02T15:43:37.065Z
63509bc859bfa9a85d4220aa
/avatars/ca2cc9b87f5ca5cd51606b2f9edf89d0.svg
Yanjie Ze
user
yjze
6541cb8ac446a9a48bbe3bd2
Simon S. Du
6541cb8ac446a9a48bbe3bd3
Huazhe Xu
admin_assigned
2023-11-02T16:15:09.356Z
649c130f92525b3ac9fbcffe
/avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
Huazhe Xu
user
Huazhe-Xu
Unleashing the Power of Pre-trained Language Models for Offline
  Reinforcement Learning
Offline reinforcement learning (RL) aims to find a near-optimal policy using
pre-collected datasets. In real-world scenarios, data collection could be
costly and risky; therefore, offline RL becomes particularly challenging when
the in-domain data is limited. Given recent advances in Large Language Models
(LLMs) and their few-shot learning prowess, this paper introduces
Language Models for Motion Control (LaMo), a
general framework based on Decision Transformers to effectively use pre-trained
Language Models (LMs) for offline RL. Our framework highlights four crucial
components: (1) Initializing Decision Transformers with sequentially
pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to
full-weight fine-tuning, to combine the pre-trained knowledge from LMs and
in-domain knowledge effectively, (3) using the non-linear MLP transformation
instead of linear projections, to generate embeddings, and (4) integrating an
auxiliary language prediction loss during fine-tuning to stabilize the LMs and
retain their original abilities on languages. Empirical results indicate
LaMo achieves state-of-the-art performance in sparse-reward tasks
and closes the gap between value-based offline RL methods and decision
transformers in dense-reward tasks. In particular, our method demonstrates
superior performance in scenarios with limited data samples. Our project
website is https://lamo2023.github.io
The LaMo framework uses Decision Transformers initialized with pre-trained Language Models to improve offline RL, particularly in sparse-reward tasks and limited data scenarios.
offline reinforcement learning Decision Transformers pre-trained Language Models LoRA fine-tuning non-linear MLP transformation auxiliary language prediction loss sparse-reward tasks dense-reward tasks parameter-efficient fine-tuning