title: Agent-to-Sim: Learning Interactive Behavior Models from Casual
    Longitudinal Videos
summary: We present Agent-to-Sim (ATS), a framework for learning interactive behavior
  models of 3D agents from casual longitudinal video collections. Different from
  prior works that rely on marker-based tracking and multiview cameras, ATS
  learns natural behaviors of animal and human agents non-invasively through
  video observations recorded over a long time-span (e.g., a month) in a single
  environment. Modeling 3D behavior of an agent requires persistent 3D tracking
  (e.g., knowing which point corresponds to which) over a long time period. To
  obtain such data, we develop a coarse-to-fine registration method that tracks
  the agent and the camera over time through a canonical 3D space, resulting in a
  complete and persistent spacetime 4D representation. We then train a generative
  model of agent behaviors using paired data of perception and motion of an agent
  queried from the 4D reconstruction. ATS enables real-to-sim transfer from video
  recordings of an agent to an interactive behavior simulator. We demonstrate
  results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos
  captured by a smartphone.
paper_authors._id: 6717d970e1288236b57e351f
paper_authors.name: Gengshan Yang
paper_authors._id: 6717d970e1288236b57e3520
paper_authors.name: Andrea Bajcsy
paper_authors._id: 6717d970e1288236b57e3521
paper_authors.name: Shunsuke Saito
paper_authors._id: 6717d970e1288236b57e3522
paper_authors.name: Angjoo Kanazawa
paper_title: Agent-to-Sim: Learning Interactive Behavior Models from Casual
    Longitudinal Videos
paper_summary: We present Agent-to-Sim (ATS), a framework for learning interactive behavior
  models of 3D agents from casual longitudinal video collections. Different from
  prior works that rely on marker-based tracking and multiview cameras, ATS
  learns natural behaviors of animal and human agents non-invasively through
  video observations recorded over a long time-span (e.g., a month) in a single
  environment. Modeling 3D behavior of an agent requires persistent 3D tracking
  (e.g., knowing which point corresponds to which) over a long time period. To
  obtain such data, we develop a coarse-to-fine registration method that tracks
  the agent and the camera over time through a canonical 3D space, resulting in a
  complete and persistent spacetime 4D representation. We then train a generative
  model of agent behaviors using paired data of perception and motion of an agent
  queried from the 4D reconstruction. ATS enables real-to-sim transfer from video
  recordings of an agent to an interactive behavior simulator. We demonstrate
  results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos
  captured by a smartphone.
paper_ai_keywords: Agent-to-Sim (ATS) longitudinal video collections marker-based tracking multiview cameras persistent 3D tracking coarse-to-fine registration canonical 3D space spacetime 4D representation generative model real-to-sim transfer monocular RGBD videos