title: Unleashing the Power of Pre-trained Language Models for Offline
    Reinforcement Learning
summary: Offline reinforcement learning (RL) aims to find a near-optimal policy using
  pre-collected datasets. In real-world scenarios, data collection could be
  costly and risky; therefore, offline RL becomes particularly challenging when
  the in-domain data is limited. Given recent advances in Large Language Models
  (LLMs) and their few-shot learning prowess, this paper introduces
  Language Models for Motion Control (LaMo), a
  general framework based on Decision Transformers to effectively use pre-trained
  Language Models (LMs) for offline RL. Our framework highlights four crucial
  components: (1) Initializing Decision Transformers with sequentially
  pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to
  full-weight fine-tuning, to combine the pre-trained knowledge from LMs and
  in-domain knowledge effectively, (3) using the non-linear MLP transformation
  instead of linear projections, to generate embeddings, and (4) integrating an
  auxiliary language prediction loss during fine-tuning to stabilize the LMs and
  retain their original abilities on languages. Empirical results indicate
  LaMo achieves state-of-the-art performance in sparse-reward tasks
  and closes the gap between value-based offline RL methods and decision
  transformers in dense-reward tasks. In particular, our method demonstrates
  superior performance in scenarios with limited data samples. Our project
  website is https://lamo2023.github.io
paper_authors._id: 6541cb8ac446a9a48bbe3bcf
paper_authors.name: Ruizhe Shi
paper_authors._id: 6541cb8ac446a9a48bbe3bd0
paper_authors.name: Yuyao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:43:30.529Z
paper_authors._id: 6427a3bad039ee7fa4e354f4
paper_authors.avatarUrl: /avatars/48640c908ab4c7a35c349a467db6e336.svg
paper_authors.fullname: Yuyao Liu
paper_authors.type: user
paper_authors.user: nature21
paper_authors._id: 6541cb8ac446a9a48bbe3bd1
paper_authors.name: Yanjie Ze
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:43:37.065Z
paper_authors._id: 63509bc859bfa9a85d4220aa
paper_authors.avatarUrl: /avatars/ca2cc9b87f5ca5cd51606b2f9edf89d0.svg
paper_authors.fullname: Yanjie Ze
paper_authors.type: user
paper_authors.user: yjze
paper_authors._id: 6541cb8ac446a9a48bbe3bd2
paper_authors.name: Simon S. Du
paper_authors._id: 6541cb8ac446a9a48bbe3bd3
paper_authors.name: Huazhe Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T16:15:09.356Z
paper_authors._id: 649c130f92525b3ac9fbcffe
paper_authors.avatarUrl: /avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
paper_authors.fullname: Huazhe Xu
paper_authors.type: user
paper_authors.user: Huazhe-Xu
paper_title: Unleashing the Power of Pre-trained Language Models for Offline
    Reinforcement Learning
paper_summary: Offline reinforcement learning (RL) aims to find a near-optimal policy using
  pre-collected datasets. In real-world scenarios, data collection could be
  costly and risky; therefore, offline RL becomes particularly challenging when
  the in-domain data is limited. Given recent advances in Large Language Models
  (LLMs) and their few-shot learning prowess, this paper introduces
  Language Models for Motion Control (LaMo), a
  general framework based on Decision Transformers to effectively use pre-trained
  Language Models (LMs) for offline RL. Our framework highlights four crucial
  components: (1) Initializing Decision Transformers with sequentially
  pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to
  full-weight fine-tuning, to combine the pre-trained knowledge from LMs and
  in-domain knowledge effectively, (3) using the non-linear MLP transformation
  instead of linear projections, to generate embeddings, and (4) integrating an
  auxiliary language prediction loss during fine-tuning to stabilize the LMs and
  retain their original abilities on languages. Empirical results indicate
  LaMo achieves state-of-the-art performance in sparse-reward tasks
  and closes the gap between value-based offline RL methods and decision
  transformers in dense-reward tasks. In particular, our method demonstrates
  superior performance in scenarios with limited data samples. Our project
  website is https://lamo2023.github.io
paper_ai_summary: The LaMo framework uses Decision Transformers initialized with pre-trained Language Models to improve offline RL, particularly in sparse-reward tasks and limited data scenarios.
paper_ai_keywords: offline reinforcement learning Decision Transformers pre-trained Language Models LoRA fine-tuning non-linear MLP transformation auxiliary language prediction loss sparse-reward tasks dense-reward tasks parameter-efficient fine-tuning