Agent-to-Sim: Learning Interactive Behavior Models from Casual
  Longitudinal Videos
We present Agent-to-Sim (ATS), a framework for learning interactive behavior
models of 3D agents from casual longitudinal video collections. Different from
prior works that rely on marker-based tracking and multiview cameras, ATS
learns natural behaviors of animal and human agents non-invasively through
video observations recorded over a long time-span (e.g., a month) in a single
environment. Modeling 3D behavior of an agent requires persistent 3D tracking
(e.g., knowing which point corresponds to which) over a long time period. To
obtain such data, we develop a coarse-to-fine registration method that tracks
the agent and the camera over time through a canonical 3D space, resulting in a
complete and persistent spacetime 4D representation. We then train a generative
model of agent behaviors using paired data of perception and motion of an agent
queried from the 4D reconstruction. ATS enables real-to-sim transfer from video
recordings of an agent to an interactive behavior simulator. We demonstrate
results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos
captured by a smartphone.
6717d970e1288236b57e351f
Gengshan Yang
6717d970e1288236b57e3520
Andrea Bajcsy
6717d970e1288236b57e3521
Shunsuke Saito
6717d970e1288236b57e3522
Angjoo Kanazawa
Agent-to-Sim: Learning Interactive Behavior Models from Casual
  Longitudinal Videos
We present Agent-to-Sim (ATS), a framework for learning interactive behavior
models of 3D agents from casual longitudinal video collections. Different from
prior works that rely on marker-based tracking and multiview cameras, ATS
learns natural behaviors of animal and human agents non-invasively through
video observations recorded over a long time-span (e.g., a month) in a single
environment. Modeling 3D behavior of an agent requires persistent 3D tracking
(e.g., knowing which point corresponds to which) over a long time period. To
obtain such data, we develop a coarse-to-fine registration method that tracks
the agent and the camera over time through a canonical 3D space, resulting in a
complete and persistent spacetime 4D representation. We then train a generative
model of agent behaviors using paired data of perception and motion of an agent
queried from the 4D reconstruction. ATS enables real-to-sim transfer from video
recordings of an agent to an interactive behavior simulator. We demonstrate
results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos
captured by a smartphone.
Agent-to-Sim (ATS) learns interactive behavior models of 3D agents using long-term video recordings, enabling real-to-sim transfer through a coarse-to-fine registration and generative modeling approach.
Agent-to-Sim 3D agents longitudinal video collections marker-based tracking multiview cameras non-invasive video observations persistent 3D tracking coarse-to-fine registration canonical 3D space spacetime 4D representation generative model agent behaviors monocular RGBD videos interactive behavior simulator