title: PASTA: Pretrained Action-State Transformer Agents
summary: Self-supervised learning has brought about a revolutionary paradigm shift in
  various computing domains, including NLP, vision, and biology. Recent
  approaches involve pre-training transformer models on vast amounts of unlabeled
  data, serving as a starting point for efficiently solving downstream tasks. In
  the realm of reinforcement learning, researchers have recently adapted these
  approaches by developing models pre-trained on expert trajectories, enabling
  them to address a wide range of tasks, from robotics to recommendation systems.
  However, existing methods mostly rely on intricate pre-training objectives
  tailored to specific downstream applications. This paper presents a
  comprehensive investigation of models we refer to as Pretrained Action-State
  Transformer Agents (PASTA). Our study uses a unified methodology and covers an
  extensive set of general downstream tasks including behavioral cloning, offline
  RL, sensor failure robustness, and dynamics change adaptation. Our goal is to
  systematically compare various design choices and provide valuable insights to
  practitioners for building robust models. Key highlights of our study include
  tokenization at the action and state component level, using fundamental
  pre-training objectives like next token prediction, training models across
  diverse domains simultaneously, and using parameter efficient fine-tuning
  (PEFT). The developed models in our study contain fewer than 10 million
  parameters and the application of PEFT enables fine-tuning of fewer than 10,000
  parameters during downstream adaptation, allowing a broad community to use
  these models and reproduce our experiments. We hope that this study will
  encourage further research into the use of transformers with first-principles
  design choices to represent RL trajectories and contribute to robust policy
  learning.
paper_authors._id: 64ba0b7962c0bfc573974118
paper_authors.name: Raphael Boige
paper_authors._id: 64ba0b7962c0bfc573974119
paper_authors.name: Yannis Flet-Berliac
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:30:09.111Z
paper_authors._id: 64a5623d3e255e5c57dd5efb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0ZtKZi2UevQ1AJlw73hq4.jpeg
paper_authors.fullname: Yannis Flet-Berliac
paper_authors.type: user
paper_authors.user: yfletberliac
paper_authors._id: 64ba0b7962c0bfc57397411a
paper_authors.name: Arthur Flajolet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:30:18.936Z
paper_authors._id: 64267de5f8868e1ac351095e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64267de5f8868e1ac351095e/auPKussoQAlvJtZOtysZD.jpeg
paper_authors.fullname: Arthur Flajolet
paper_authors.type: user
paper_authors.user: aflajolet
paper_authors._id: 64ba0b7962c0bfc57397411b
paper_authors.name: Guillaume Richard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:31:07.624Z
paper_authors._id: 6426b768bc4f1d51f546e8c5
paper_authors.avatarUrl: /avatars/0cce972b561d050d9fb5569a9d8f3e48.svg
paper_authors.fullname: Guillaume Richard
paper_authors.type: user
paper_authors.user: GRichard513
paper_authors._id: 64ba0b7962c0bfc57397411c
paper_authors.name: Thomas Pierrot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:31:21.535Z
paper_authors._id: 63d048c28cf6c8e4d208b072
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d048c28cf6c8e4d208b072/sGrp6YITRr1QUtgLWaMKY.jpeg
paper_authors.fullname: Thomas Pierrot
paper_authors.type: user
paper_authors.user: tpierrot
paper_title: PASTA: Pretrained Action-State Transformer Agents
paper_summary: Self-supervised learning has brought about a revolutionary paradigm shift in
  various computing domains, including NLP, vision, and biology. Recent
  approaches involve pre-training transformer models on vast amounts of unlabeled
  data, serving as a starting point for efficiently solving downstream tasks. In
  the realm of reinforcement learning, researchers have recently adapted these
  approaches by developing models pre-trained on expert trajectories, enabling
  them to address a wide range of tasks, from robotics to recommendation systems.
  However, existing methods mostly rely on intricate pre-training objectives
  tailored to specific downstream applications. This paper presents a
  comprehensive investigation of models we refer to as Pretrained Action-State
  Transformer Agents (PASTA). Our study uses a unified methodology and covers an
  extensive set of general downstream tasks including behavioral cloning, offline
  RL, sensor failure robustness, and dynamics change adaptation. Our goal is to
  systematically compare various design choices and provide valuable insights to
  practitioners for building robust models. Key highlights of our study include
  tokenization at the action and state component level, using fundamental
  pre-training objectives like next token prediction, training models across
  diverse domains simultaneously, and using parameter efficient fine-tuning
  (PEFT). The developed models in our study contain fewer than 10 million
  parameters and the application of PEFT enables fine-tuning of fewer than 10,000
  parameters during downstream adaptation, allowing a broad community to use
  these models and reproduce our experiments. We hope that this study will
  encourage further research into the use of transformers with first-principles
  design choices to represent RL trajectories and contribute to robust policy
  learning.
paper_ai_summary: Pretrained Action-State Transformer Agents (PASTA) are investigated for a range of reinforcement learning tasks using unified methodologies and parameter-efficient fine-tuning.
paper_ai_keywords: self-supervised learning transformer models reinforcement learning expert trajectories behavioral cloning offline RL sensor failure robustness dynamics change adaptation tokenization next token prediction parameter-efficient fine-tuning PEFT