Q-Transformer: Scalable Offline Reinforcement Learning via
  Autoregressive Q-Functions
In this work, we present a scalable reinforcement learning method for
training multi-task policies from large offline datasets that can leverage both
human demonstrations and autonomously collected data. Our method uses a
Transformer to provide a scalable representation for Q-functions trained via
offline temporal difference backups. We therefore refer to the method as
Q-Transformer. By discretizing each action dimension and representing the
Q-value of each action dimension as separate tokens, we can apply effective
high-capacity sequence modeling techniques for Q-learning. We present several
design decisions that enable good performance with offline RL training, and
show that Q-Transformer outperforms prior offline RL algorithms and imitation
learning techniques on a large diverse real-world robotic manipulation task
suite. The project's website and videos can be found at
https://q-transformer.github.io
650a54f8948ce5dce8cf9155
Yevgen Chebotar
admin_assigned
2023-09-20T12:00:24.657Z
64c93f9c9471c54a9438162c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
Yevgen Chebotar
user
ychebotar
650a54f8948ce5dce8cf9156
Quan Vuong
admin_assigned
2023-09-20T12:00:44.397Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
650a54f8948ce5dce8cf9157
Alex Irpan
650a54f8948ce5dce8cf9158
Karol Hausman
admin_assigned
2023-09-20T12:01:51.203Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
650a54f8948ce5dce8cf9159
Fei Xia
claimed_verified
2024-01-23T08:10:22.426Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
650a54f8948ce5dce8cf915a
Yao Lu
650a54f8948ce5dce8cf915b
Aviral Kumar
admin_assigned
2023-09-20T12:03:35.665Z
64879130cc0e54f272b5dbcc
/avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
Aviral Kumar
user
aviralkumar
650a54f8948ce5dce8cf915c
Tianhe Yu
admin_assigned
2023-09-20T12:03:43.791Z
64c95527d33e6a7f82b8b911
/avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
Tianhe Yu
user
keviny66
650a54f8948ce5dce8cf915d
Alexander Herzog
claimed_verified
2023-09-25T10:52:07.590Z
650e4bd211f3210cf79379dd
/avatars/b2c11d747f22aa21b95063813775c59c.svg
Alexander Herzog
user
alexherzog
650a54f8948ce5dce8cf915e
Karl Pertsch
650a54f8948ce5dce8cf915f
Keerthana Gopalakrishnan
admin_assigned
2023-09-20T12:04:30.680Z
632f7dee34c562cb1d380fdc
/avatars/d689e260535b8aa775654160028d4ec8.svg
Keerthana Gopalakrishnan
user
keerthanpg
650a54f8948ce5dce8cf9160
Julian Ibarz
admin_assigned
2023-09-20T12:04:37.469Z
641e33302bb8b89e81ce6ad3
/avatars/8781b559c80dab49a1b9e2bba09f67e6.svg
Julian Ibarz
user
jibarz
650a54f8948ce5dce8cf9161
Ofir Nachum
650a54f8948ce5dce8cf9162
Sumedh Sontakke
650a54f8948ce5dce8cf9163
Grecia Salazar
650a54f8948ce5dce8cf9164
Huong T Tran
admin_assigned
2023-09-20T12:05:23.725Z
642c43dca79f57aedc1c6136
/avatars/e79a1cd0693b160a9207ab0b7f740319.svg
Huong Tran
user
huongk59
650a54f8948ce5dce8cf9165
Jodilyn Peralta
650a54f8948ce5dce8cf9166
Clayton Tan
650a54f8948ce5dce8cf9167
Deeksha Manjunath
admin_assigned
2023-09-20T12:05:46.504Z
63eaf42e91a1b8ec4fbb323d
/avatars/ebd8f05de72bb846aca049705ebc161b.svg
Deeksha Manjunath
user
deekshaGT
650a54f8948ce5dce8cf9168
Jaspiar Singht
650a54f8948ce5dce8cf9169
Brianna Zitkovich
650a54f8948ce5dce8cf916a
Tomas Jackson
admin_assigned
2023-09-20T12:06:17.552Z
63ffd800b09f82a81a220cc9
/avatars/c9e4e577aba78e69620e8f6d02820a6d.svg
Tomas Jackson
user
Loss324
650a54f8948ce5dce8cf916b
Kanishka Rao
650a54f8948ce5dce8cf916c
Chelsea Finn
admin_assigned
2023-09-20T12:06:33.754Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
650a54f8948ce5dce8cf916d
Sergey Levine
Q-Transformer: Scalable Offline Reinforcement Learning via
  Autoregressive Q-Functions
In this work, we present a scalable reinforcement learning method for
training multi-task policies from large offline datasets that can leverage both
human demonstrations and autonomously collected data. Our method uses a
Transformer to provide a scalable representation for Q-functions trained via
offline temporal difference backups. We therefore refer to the method as
Q-Transformer. By discretizing each action dimension and representing the
Q-value of each action dimension as separate tokens, we can apply effective
high-capacity sequence modeling techniques for Q-learning. We present several
design decisions that enable good performance with offline RL training, and
show that Q-Transformer outperforms prior offline RL algorithms and imitation
learning techniques on a large diverse real-world robotic manipulation task
suite. The project's website and videos can be found at
https://q-transformer.github.io
Q-Transformer, a novel reinforcement learning method leveraging a Transformer for scalable Q-function representation, excels in offline multi-task robotic manipulation by efficiently modeling complex action sequences.
Transformer Q-functions offline temporal difference backups Q-Transformer high-capacity sequence modeling offline RL training imitation learning