title: Pre-training Auto-regressive Robotic Models with 4D Representations
summary: Foundation models pre-trained on massive unlabeled datasets have
  revolutionized natural language and computer vision, exhibiting remarkable
  generalization capabilities, thus highlighting the importance of pre-training.
  Yet, efforts in robotics have struggled to achieve similar success, limited by
  either the need for costly robotic annotations or the lack of representations
  that effectively model the physical world. In this paper, we introduce ARM4R,
  an Auto-regressive Robotic Model that leverages low-level 4D Representations
  learned from human video data to yield a better pre-trained robotic model.
  Specifically, we focus on utilizing 3D point tracking representations from
  videos derived by lifting 2D representations into 3D space via monocular depth
  estimation across time. These 4D representations maintain a shared geometric
  structure between the points and robot state representations up to a linear
  transformation, enabling efficient transfer learning from human video data to
  low-level robotic control. Our experiments show that ARM4R can transfer
  efficiently from human video data to robotics and consistently improves
  performance on tasks across various robot environments and configurations.
paper_authors._id: 67b5790132be608036ee94e5
paper_authors.name: Dantong Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:12:28.457Z
paper_authors._id: 65c3fdf79d062be813813e45
paper_authors.avatarUrl: /avatars/52528a61abe5bbbef4a4a431944973cd.svg
paper_authors.fullname: Dantong Niu
paper_authors.type: user
paper_authors.user: NdtSoCool
paper_authors._id: 67b5790132be608036ee94e6
paper_authors.name: Yuvan Sharma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:12:35.531Z
paper_authors._id: 65406e82deee4716f1c29271
paper_authors.avatarUrl: /avatars/25331a773f8125f9ad1c3d6ac3375586.svg
paper_authors.fullname: Yuvan Sharma
paper_authors.type: user
paper_authors.user: yuvansharma
paper_authors._id: 67b5790132be608036ee94e7
paper_authors.name: Haoru Xue
paper_authors._id: 67b5790132be608036ee94e8
paper_authors.name: Giscard Biamby
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:12:49.219Z
paper_authors._id: 650bd36a7c99ca283e58e973
paper_authors.avatarUrl: /avatars/606d24b2dac190ebcbb4b2a2e4671380.svg
paper_authors.fullname: Giscard Biamby
paper_authors.type: user
paper_authors.user: gbiamby
paper_authors._id: 67b5790132be608036ee94e9
paper_authors.name: Junyi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:37:02.956Z
paper_authors._id: 62f0ecd2700bdc19558360de
paper_authors.avatarUrl: /avatars/5325b4b763f30c41f30e3aec0d2b59fa.svg
paper_authors.fullname: Junyi Zhang
paper_authors.type: user
paper_authors.user: Junyi42
paper_authors._id: 67b5790132be608036ee94ea
paper_authors.name: Ziteng Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:13:13.907Z
paper_authors._id: 66a09aec369dd38cf2113070
paper_authors.avatarUrl: /avatars/cc13bdd3dc1271d33b083b61e12f1a05.svg
paper_authors.fullname: Ziteng Ji
paper_authors.type: user
paper_authors.user: zitengj0618
paper_authors._id: 67b5790132be608036ee94eb
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:13:20.379Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_authors._id: 67b5790132be608036ee94ec
paper_authors.name: Roei Herzig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T09:13:26.134Z
paper_authors._id: 667c5764186b27ef806636d3
paper_authors.avatarUrl: /avatars/5c08f0109bc0e350624112c0aff544f6.svg
paper_authors.fullname: Roei Herzig
paper_authors.type: user
paper_authors.user: roeiherz
paper_title: Pre-training Auto-regressive Robotic Models with 4D Representations
paper_summary: Foundation models pre-trained on massive unlabeled datasets have
  revolutionized natural language and computer vision, exhibiting remarkable
  generalization capabilities, thus highlighting the importance of pre-training.
  Yet, efforts in robotics have struggled to achieve similar success, limited by
  either the need for costly robotic annotations or the lack of representations
  that effectively model the physical world. In this paper, we introduce ARM4R,
  an Auto-regressive Robotic Model that leverages low-level 4D Representations
  learned from human video data to yield a better pre-trained robotic model.
  Specifically, we focus on utilizing 3D point tracking representations from
  videos derived by lifting 2D representations into 3D space via monocular depth
  estimation across time. These 4D representations maintain a shared geometric
  structure between the points and robot state representations up to a linear
  transformation, enabling efficient transfer learning from human video data to
  low-level robotic control. Our experiments show that ARM4R can transfer
  efficiently from human video data to robotics and consistently improves
  performance on tasks across various robot environments and configurations.
paper_ai_summary: ARM4R, an Auto-regressive Robotic Model, uses 4D representations from human video data to enhance pre-trained robotic models and improve performance across different robotic tasks.
paper_ai_keywords: Auto-regressive Robotic Model 4D Representations 3D point tracking monocular depth estimation transfer learning robot state representations