title: Q-Transformer: Scalable Offline Reinforcement Learning via
    Autoregressive Q-Functions
summary: In this work, we present a scalable reinforcement learning method for
  training multi-task policies from large offline datasets that can leverage both
  human demonstrations and autonomously collected data. Our method uses a
  Transformer to provide a scalable representation for Q-functions trained via
  offline temporal difference backups. We therefore refer to the method as
  Q-Transformer. By discretizing each action dimension and representing the
  Q-value of each action dimension as separate tokens, we can apply effective
  high-capacity sequence modeling techniques for Q-learning. We present several
  design decisions that enable good performance with offline RL training, and
  show that Q-Transformer outperforms prior offline RL algorithms and imitation
  learning techniques on a large diverse real-world robotic manipulation task
  suite. The project's website and videos can be found at
  https://q-transformer.github.io
paper_authors._id: 650a54f8948ce5dce8cf9155
paper_authors.name: Yevgen Chebotar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:00:24.657Z
paper_authors._id: 64c93f9c9471c54a9438162c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
paper_authors.fullname: Yevgen Chebotar
paper_authors.type: user
paper_authors.user: ychebotar
paper_authors._id: 650a54f8948ce5dce8cf9156
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:00:44.397Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: kabut1211
paper_authors._id: 650a54f8948ce5dce8cf9157
paper_authors.name: Alex Irpan
paper_authors._id: 650a54f8948ce5dce8cf9158
paper_authors.name: Karol Hausman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:01:51.203Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 650a54f8948ce5dce8cf9159
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:10:22.426Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 650a54f8948ce5dce8cf915a
paper_authors.name: Yao Lu
paper_authors._id: 650a54f8948ce5dce8cf915b
paper_authors.name: Aviral Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:03:35.665Z
paper_authors._id: 64879130cc0e54f272b5dbcc
paper_authors.avatarUrl: /avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralkumar
paper_authors._id: 650a54f8948ce5dce8cf915c
paper_authors.name: Tianhe Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:03:43.791Z
paper_authors._id: 64c95527d33e6a7f82b8b911
paper_authors.avatarUrl: /avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
paper_authors.fullname: Tianhe Yu
paper_authors.type: user
paper_authors.user: keviny66
paper_authors._id: 650a54f8948ce5dce8cf915d
paper_authors.name: Alexander Herzog
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-25T10:52:07.590Z
paper_authors._id: 650e4bd211f3210cf79379dd
paper_authors.avatarUrl: /avatars/b2c11d747f22aa21b95063813775c59c.svg
paper_authors.fullname: Alexander Herzog
paper_authors.type: user
paper_authors.user: alexherzog
paper_authors._id: 650a54f8948ce5dce8cf915e
paper_authors.name: Karl Pertsch
paper_authors._id: 650a54f8948ce5dce8cf915f
paper_authors.name: Keerthana Gopalakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:04:30.680Z
paper_authors._id: 632f7dee34c562cb1d380fdc
paper_authors.avatarUrl: /avatars/d689e260535b8aa775654160028d4ec8.svg
paper_authors.fullname: Keerthana Gopalakrishnan
paper_authors.type: user
paper_authors.user: keerthanpg
paper_authors._id: 650a54f8948ce5dce8cf9160
paper_authors.name: Julian Ibarz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:04:37.469Z
paper_authors._id: 641e33302bb8b89e81ce6ad3
paper_authors.avatarUrl: /avatars/8781b559c80dab49a1b9e2bba09f67e6.svg
paper_authors.fullname: Julian Ibarz
paper_authors.type: user
paper_authors.user: jibarz
paper_authors._id: 650a54f8948ce5dce8cf9161
paper_authors.name: Ofir Nachum
paper_authors._id: 650a54f8948ce5dce8cf9162
paper_authors.name: Sumedh Sontakke
paper_authors._id: 650a54f8948ce5dce8cf9163
paper_authors.name: Grecia Salazar
paper_authors._id: 650a54f8948ce5dce8cf9164
paper_authors.name: Huong T Tran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:05:23.725Z
paper_authors._id: 642c43dca79f57aedc1c6136
paper_authors.avatarUrl: /avatars/e79a1cd0693b160a9207ab0b7f740319.svg
paper_authors.fullname: Huong Tran
paper_authors.type: user
paper_authors.user: huongk59
paper_authors._id: 650a54f8948ce5dce8cf9165
paper_authors.name: Jodilyn Peralta
paper_authors._id: 650a54f8948ce5dce8cf9166
paper_authors.name: Clayton Tan
paper_authors._id: 650a54f8948ce5dce8cf9167
paper_authors.name: Deeksha Manjunath
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:05:46.504Z
paper_authors._id: 63eaf42e91a1b8ec4fbb323d
paper_authors.avatarUrl: /avatars/ebd8f05de72bb846aca049705ebc161b.svg
paper_authors.fullname: Deeksha Manjunath
paper_authors.type: user
paper_authors.user: deekshaGT
paper_authors._id: 650a54f8948ce5dce8cf9168
paper_authors.name: Jaspiar Singht
paper_authors._id: 650a54f8948ce5dce8cf9169
paper_authors.name: Brianna Zitkovich
paper_authors._id: 650a54f8948ce5dce8cf916a
paper_authors.name: Tomas Jackson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:06:17.552Z
paper_authors._id: 63ffd800b09f82a81a220cc9
paper_authors.avatarUrl: /avatars/c9e4e577aba78e69620e8f6d02820a6d.svg
paper_authors.fullname: Tomas Jackson
paper_authors.type: user
paper_authors.user: Loss324
paper_authors._id: 650a54f8948ce5dce8cf916b
paper_authors.name: Kanishka Rao
paper_authors._id: 650a54f8948ce5dce8cf916c
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:06:33.754Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_authors._id: 650a54f8948ce5dce8cf916d
paper_authors.name: Sergey Levine
paper_title: Q-Transformer: Scalable Offline Reinforcement Learning via
    Autoregressive Q-Functions
paper_summary: In this work, we present a scalable reinforcement learning method for
  training multi-task policies from large offline datasets that can leverage both
  human demonstrations and autonomously collected data. Our method uses a
  Transformer to provide a scalable representation for Q-functions trained via
  offline temporal difference backups. We therefore refer to the method as
  Q-Transformer. By discretizing each action dimension and representing the
  Q-value of each action dimension as separate tokens, we can apply effective
  high-capacity sequence modeling techniques for Q-learning. We present several
  design decisions that enable good performance with offline RL training, and
  show that Q-Transformer outperforms prior offline RL algorithms and imitation
  learning techniques on a large diverse real-world robotic manipulation task
  suite. The project's website and videos can be found at
  https://q-transformer.github.io
paper_ai_keywords: Transformer Q-functions offline temporal difference backups Q-Transformer action dimension tokens sequence modeling techniques Q-learning offline RL training imitation learning techniques real-world robotic manipulation task suite