title: Learning Video Generation for Robotic Manipulation with Collaborative
    Trajectory Control
summary: Recent advances in video diffusion models have demonstrated strong potential
  for generating robotic decision-making data, with trajectory conditions further
  enabling fine-grained control. However, existing trajectory-based methods
  primarily focus on individual object motion and struggle to capture
  multi-object interaction crucial in complex robotic manipulation. This
  limitation arises from multi-feature entanglement in overlapping regions, which
  leads to degraded visual fidelity. To address this, we present RoboMaster, a
  novel framework that models inter-object dynamics through a collaborative
  trajectory formulation. Unlike prior methods that decompose objects, our core
  is to decompose the interaction process into three sub-stages: pre-interaction,
  interaction, and post-interaction. Each stage is modeled using the feature of
  the dominant object, specifically the robotic arm in the pre- and
  post-interaction phases and the manipulated object during interaction, thereby
  mitigating the drawback of multi-object feature fusion present during
  interaction in prior work. To further ensure subject semantic consistency
  throughout the video, we incorporate appearance- and shape-aware latent
  representations for objects. Extensive experiments on the challenging Bridge V2
  dataset, as well as in-the-wild evaluation, demonstrate that our method
  outperforms existing approaches, establishing new state-of-the-art performance
  in trajectory-controlled video generation for robotic manipulation.
paper_authors._id: 683e6b6424742a21489ec9f8
paper_authors.name: Xiao Fu
paper_authors._id: 683e6b6424742a21489ec9f9
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:36:21.590Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 683e6b6424742a21489ec9fa
paper_authors.name: Xian Liu
paper_authors._id: 683e6b6424742a21489ec9fb
paper_authors.name: Jianhong Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:36:29.349Z
paper_authors._id: 6530bf50f145530101ec03a2
paper_authors.avatarUrl: /avatars/c61c00c314cf202b64968e51e855694d.svg
paper_authors.fullname: Jianhong Bai
paper_authors.type: user
paper_authors.user: jianhongbai
paper_authors._id: 683e6b6424742a21489ec9fc
paper_authors.name: Runsen Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:36:13.754Z
paper_authors._id: 6458b1103b81018d6b93defb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6458b1103b81018d6b93defb/JFr5j-3UO0xxlEdUksikU.jpeg
paper_authors.fullname: Runsen Xu
paper_authors.type: user
paper_authors.user: RunsenXu
paper_authors._id: 683e6b6424742a21489ec9fd
paper_authors.name: Pengfei Wan
paper_authors._id: 683e6b6424742a21489ec9fe
paper_authors.name: Di Zhang
paper_authors._id: 683e6b6424742a21489ec9ff
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:35:57.663Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_title: Learning Video Generation for Robotic Manipulation with Collaborative
    Trajectory Control
paper_summary: Recent advances in video diffusion models have demonstrated strong potential
  for generating robotic decision-making data, with trajectory conditions further
  enabling fine-grained control. However, existing trajectory-based methods
  primarily focus on individual object motion and struggle to capture
  multi-object interaction crucial in complex robotic manipulation. This
  limitation arises from multi-feature entanglement in overlapping regions, which
  leads to degraded visual fidelity. To address this, we present RoboMaster, a
  novel framework that models inter-object dynamics through a collaborative
  trajectory formulation. Unlike prior methods that decompose objects, our core
  is to decompose the interaction process into three sub-stages: pre-interaction,
  interaction, and post-interaction. Each stage is modeled using the feature of
  the dominant object, specifically the robotic arm in the pre- and
  post-interaction phases and the manipulated object during interaction, thereby
  mitigating the drawback of multi-object feature fusion present during
  interaction in prior work. To further ensure subject semantic consistency
  throughout the video, we incorporate appearance- and shape-aware latent
  representations for objects. Extensive experiments on the challenging Bridge V2
  dataset, as well as in-the-wild evaluation, demonstrate that our method
  outperforms existing approaches, establishing new state-of-the-art performance
  in trajectory-controlled video generation for robotic manipulation.
paper_ai_summary: A novel framework, RoboMaster, enhances trajectory-controlled video generation for robotic manipulation by modeling inter-object dynamics through a collaborative trajectory formulation, achieving state-of-the-art performance on the Bridge V2 dataset.
paper_ai_keywords: video diffusion models trajectory conditions multi-object interaction multi-feature entanglement visual fidelity collaborative trajectory formulation pre-interaction interaction post-interaction appearance-aware latent representations shape-aware latent representations trajectory-controlled video generation robotic manipulation Bridge V2 dataset