Robotic Offline RL from Internet Videos via Value-Function Pre-Training
Pre-training on Internet data has proven to be a key ingredient for broad
generalization in many modern ML systems. What would it take to enable such
capabilities in robotic reinforcement learning (RL)? Offline RL methods, which
learn from datasets of robot experience, offer one way to leverage prior data
into the robotic learning pipeline. However, these methods have a "type
mismatch" with video data (such as Ego4D), the largest prior datasets available
for robotics, since video offers observation-only experience without the action
or reward annotations needed for RL methods. In this paper, we develop a system
for leveraging large-scale human video datasets in robotic offline RL, based
entirely on learning value functions via temporal-difference learning. We show
that value learning on video datasets learns representations that are more
conducive to downstream robotic offline RL than other approaches for learning
from video data. Our system, called V-PTR, combines the benefits of
pre-training on video data with robotic offline RL approaches that train on
diverse robot data, resulting in value functions and policies for manipulation
tasks that perform better, act robustly, and generalize broadly. On several
manipulation tasks on a real WidowX robot, our framework produces policies that
greatly improve over prior methods. Our video and additional details can be
found at https://dibyaghosh.com/vptr/
65111d209d93eec4a774169f
Chethan Bhateja
admin_assigned
2023-09-25T11:20:07.671Z
6407c14d4dc5f2846c9c1708
/avatars/823b8c3acdbee263abcf4c7d8fc11d07.svg
Chethan Bhateja
user
chetb
65111d209d93eec4a77416a0
Derek Guo
extracted_pending
2023-09-25T05:39:48.536Z
63e9482a46574e63a2c6ccd5
/avatars/8c4b7ebce6a524f8e7352e751a476053.svg
Dreek G
user
Knife-kun
65111d209d93eec4a77416a1
Dibya Ghosh
65111d209d93eec4a77416a2
Anikait Singh
claimed_verified
2023-09-26T09:32:05.926Z
6511ee845b7e52b0251fdee9
https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
Anikait Singh
user
Asap7772
65111d209d93eec4a77416a3
Manan Tomar
65111d209d93eec4a77416a4
Quan Vuong
admin_assigned
2023-09-25T11:20:40.613Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
65111d209d93eec4a77416a5
Yevgen Chebotar
admin_assigned
2023-09-25T11:20:47.724Z
64c93f9c9471c54a9438162c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
Yevgen Chebotar
user
ychebotar
65111d209d93eec4a77416a6
Sergey Levine
65111d209d93eec4a77416a7
Aviral Kumar
admin_assigned
2023-09-25T11:21:00.472Z
64879130cc0e54f272b5dbcc
/avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
Aviral Kumar
user
aviralkumar
Robotic Offline RL from Internet Videos via Value-Function Pre-Training
Pre-training on Internet data has proven to be a key ingredient for broad
generalization in many modern ML systems. What would it take to enable such
capabilities in robotic reinforcement learning (RL)? Offline RL methods, which
learn from datasets of robot experience, offer one way to leverage prior data
into the robotic learning pipeline. However, these methods have a "type
mismatch" with video data (such as Ego4D), the largest prior datasets available
for robotics, since video offers observation-only experience without the action
or reward annotations needed for RL methods. In this paper, we develop a system
for leveraging large-scale human video datasets in robotic offline RL, based
entirely on learning value functions via temporal-difference learning. We show
that value learning on video datasets learns representations that are more
conducive to downstream robotic offline RL than other approaches for learning
from video data. Our system, called V-PTR, combines the benefits of
pre-training on video data with robotic offline RL approaches that train on
diverse robot data, resulting in value functions and policies for manipulation
tasks that perform better, act robustly, and generalize broadly. On several
manipulation tasks on a real WidowX robot, our framework produces policies that
greatly improve over prior methods. Our video and additional details can be
found at https://dibyaghosh.com/vptr/
A system called V-PTR integrates temporal-difference learning from video data to enhance robotic offline reinforcement learning, leading to better policies for manipulation tasks.
reinforcement learning offline RL temporal-difference learning value functions policies manipulation tasks pre-training video datasets