title: TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for
    Generalist Robotic Policies
summary: Although large vision-language-action (VLA) models pretrained on extensive
  robot datasets offer promising generalist policies for robotic learning, they
  still struggle with spatial-temporal dynamics in interactive robotics, making
  them less effective in handling complex tasks, such as manipulation. In this
  work, we introduce visual trace prompting, a simple yet effective approach to
  facilitate VLA models' spatial-temporal awareness for action prediction by
  encoding state-action trajectories visually. We develop a new TraceVLA model by
  finetuning OpenVLA on our own collected dataset of 150K robot manipulation
  trajectories using visual trace prompting. Evaluations of TraceVLA across 137
  configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate
  state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and
  3.5x on real-robot tasks and exhibiting robust generalization across diverse
  embodiments and scenarios. To further validate the effectiveness and generality
  of our method, we present a compact VLA model based on 4B Phi-3-Vision,
  pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B
  OpenVLA baseline while significantly improving inference efficiency.
paper_authors._id: 6760512bda45b7d6f71baef6
paper_authors.name: Ruijie Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:41:57.819Z
paper_authors._id: 653b24ef8f8d60f204872f0a
paper_authors.avatarUrl: /avatars/45a55219e8a78be53fd32e96ba460282.svg
paper_authors.fullname: Ruijie Zheng
paper_authors.type: user
paper_authors.user: rzheng12
paper_authors._id: 6760512bda45b7d6f71baef7
paper_authors.name: Yongyuan Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:41:50.019Z
paper_authors._id: 6646d5819bb34d2b6b7455d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JFH3ZTPvlaVSg4RJJBb6L.jpeg
paper_authors.fullname: Yongyuan Liang
paper_authors.type: user
paper_authors.user: cheryyunl
paper_authors._id: 6760512bda45b7d6f71baef8
paper_authors.name: Shuaiyi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:41:42.477Z
paper_authors._id: 65cad78525a3b8e44dadaf62
paper_authors.avatarUrl: /avatars/76466c1a26195a4a1d77189b4758bf7c.svg
paper_authors.fullname: Shuaiyi Huang
paper_authors.type: user
paper_authors.user: ShirleyHuang
paper_authors._id: 6760512bda45b7d6f71baef9
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:41:35.144Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 6760512bda45b7d6f71baefa
paper_authors.name: Hal Daumé III
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:41:26.931Z
paper_authors._id: 65e0d1a52512b0269dc01b34
paper_authors.avatarUrl: /avatars/3068a8af7dc0b2d05625c282ba7a5ffb.svg
paper_authors.fullname: Hal Daumé III
paper_authors.type: user
paper_authors.user: haldaume3
paper_authors._id: 6760512bda45b7d6f71baefb
paper_authors.name: Andrey Kolobov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:41:19.350Z
paper_authors._id: 6350478dc5acdbefb8ecf722
paper_authors.avatarUrl: /avatars/75f3cdc38f6e67ae0435c30aff50b089.svg
paper_authors.fullname: Andrey Kolobov
paper_authors.type: user
paper_authors.user: akolobov
paper_authors._id: 6760512bda45b7d6f71baefc
paper_authors.name: Furong Huang
paper_authors._id: 6760512bda45b7d6f71baefd
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:40:59.234Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_title: TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for
    Generalist Robotic Policies
paper_summary: Although large vision-language-action (VLA) models pretrained on extensive
  robot datasets offer promising generalist policies for robotic learning, they
  still struggle with spatial-temporal dynamics in interactive robotics, making
  them less effective in handling complex tasks, such as manipulation. In this
  work, we introduce visual trace prompting, a simple yet effective approach to
  facilitate VLA models' spatial-temporal awareness for action prediction by
  encoding state-action trajectories visually. We develop a new TraceVLA model by
  finetuning OpenVLA on our own collected dataset of 150K robot manipulation
  trajectories using visual trace prompting. Evaluations of TraceVLA across 137
  configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate
  state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and
  3.5x on real-robot tasks and exhibiting robust generalization across diverse
  embodiments and scenarios. To further validate the effectiveness and generality
  of our method, we present a compact VLA model based on 4B Phi-3-Vision,
  pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B
  OpenVLA baseline while significantly improving inference efficiency.
paper_ai_keywords: visual trace prompting state-action trajectories TraceVLA model OpenVLA SimplerEnv WidowX robot spatial-temporal awareness action prediction robust generalization compact VLA model Phi-3-Vision Open-X-Embodiment