title: Robotic Offline RL from Internet Videos via Value-Function Pre-Training
summary: Pre-training on Internet data has proven to be a key ingredient for broad
  generalization in many modern ML systems. What would it take to enable such
  capabilities in robotic reinforcement learning (RL)? Offline RL methods, which
  learn from datasets of robot experience, offer one way to leverage prior data
  into the robotic learning pipeline. However, these methods have a "type
  mismatch" with video data (such as Ego4D), the largest prior datasets available
  for robotics, since video offers observation-only experience without the action
  or reward annotations needed for RL methods. In this paper, we develop a system
  for leveraging large-scale human video datasets in robotic offline RL, based
  entirely on learning value functions via temporal-difference learning. We show
  that value learning on video datasets learns representations that are more
  conducive to downstream robotic offline RL than other approaches for learning
  from video data. Our system, called V-PTR, combines the benefits of
  pre-training on video data with robotic offline RL approaches that train on
  diverse robot data, resulting in value functions and policies for manipulation
  tasks that perform better, act robustly, and generalize broadly. On several
  manipulation tasks on a real WidowX robot, our framework produces policies that
  greatly improve over prior methods. Our video and additional details can be
  found at https://dibyaghosh.com/vptr/
paper_authors._id: 65111d209d93eec4a774169f
paper_authors.name: Chethan Bhateja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:20:07.671Z
paper_authors._id: 6407c14d4dc5f2846c9c1708
paper_authors.avatarUrl: /avatars/823b8c3acdbee263abcf4c7d8fc11d07.svg
paper_authors.fullname: Chethan Bhateja
paper_authors.type: user
paper_authors.user: chetb
paper_authors._id: 65111d209d93eec4a77416a0
paper_authors.name: Derek Guo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-09-25T05:39:48.536Z
paper_authors._id: 63e9482a46574e63a2c6ccd5
paper_authors.avatarUrl: /avatars/8c4b7ebce6a524f8e7352e751a476053.svg
paper_authors.fullname: Dreek G
paper_authors.type: user
paper_authors.user: Knife-kun
paper_authors._id: 65111d209d93eec4a77416a1
paper_authors.name: Dibya Ghosh
paper_authors._id: 65111d209d93eec4a77416a2
paper_authors.name: Anikait Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-26T09:32:05.926Z
paper_authors._id: 6511ee845b7e52b0251fdee9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
paper_authors.fullname: Anikait Singh
paper_authors.type: user
paper_authors.user: Asap7772
paper_authors._id: 65111d209d93eec4a77416a3
paper_authors.name: Manan Tomar
paper_authors._id: 65111d209d93eec4a77416a4
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:20:40.613Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: kabut1211
paper_authors._id: 65111d209d93eec4a77416a5
paper_authors.name: Yevgen Chebotar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:20:47.724Z
paper_authors._id: 64c93f9c9471c54a9438162c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c93f9c9471c54a9438162c/bYGUqFsF67VuAuXecjxjX.jpeg
paper_authors.fullname: Yevgen Chebotar
paper_authors.type: user
paper_authors.user: ychebotar
paper_authors._id: 65111d209d93eec4a77416a6
paper_authors.name: Sergey Levine
paper_authors._id: 65111d209d93eec4a77416a7
paper_authors.name: Aviral Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:21:00.472Z
paper_authors._id: 64879130cc0e54f272b5dbcc
paper_authors.avatarUrl: /avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralkumar
paper_title: Robotic Offline RL from Internet Videos via Value-Function Pre-Training
paper_summary: Pre-training on Internet data has proven to be a key ingredient for broad
  generalization in many modern ML systems. What would it take to enable such
  capabilities in robotic reinforcement learning (RL)? Offline RL methods, which
  learn from datasets of robot experience, offer one way to leverage prior data
  into the robotic learning pipeline. However, these methods have a "type
  mismatch" with video data (such as Ego4D), the largest prior datasets available
  for robotics, since video offers observation-only experience without the action
  or reward annotations needed for RL methods. In this paper, we develop a system
  for leveraging large-scale human video datasets in robotic offline RL, based
  entirely on learning value functions via temporal-difference learning. We show
  that value learning on video datasets learns representations that are more
  conducive to downstream robotic offline RL than other approaches for learning
  from video data. Our system, called V-PTR, combines the benefits of
  pre-training on video data with robotic offline RL approaches that train on
  diverse robot data, resulting in value functions and policies for manipulation
  tasks that perform better, act robustly, and generalize broadly. On several
  manipulation tasks on a real WidowX robot, our framework produces policies that
  greatly improve over prior methods. Our video and additional details can be
  found at https://dibyaghosh.com/vptr/
paper_ai_keywords: temporal-difference learning value functions pre-training offline RL robotic offline RL video datasets distance mismatch distance mismatch distance mismatch manipulation tasks policy pre-training on video data diverse robot data representation robust generalization