title: MIMO: Controllable Character Video Synthesis with Spatial Decomposed
    Modeling
summary: Character video synthesis aims to produce realistic videos of animatable
  characters within lifelike scenes. As a fundamental problem in the computer
  vision and graphics community, 3D works typically require multi-view captures
  for per-case training, which severely limits their applicability of modeling
  arbitrary characters in a short time. Recent 2D methods break this limitation
  via pre-trained diffusion models, but they struggle for pose generality and
  scene interaction. To this end, we propose MIMO, a novel framework which can
  not only synthesize character videos with controllable attributes (i.e.,
  character, motion and scene) provided by simple user inputs, but also
  simultaneously achieve advanced scalability to arbitrary characters, generality
  to novel 3D motions, and applicability to interactive real-world scenes in a
  unified framework. The core idea is to encode the 2D video to compact spatial
  codes, considering the inherent 3D nature of video occurrence. Concretely, we
  lift the 2D frame pixels into 3D using monocular depth estimators, and
  decompose the video clip to three spatial components (i.e., main human,
  underlying scene, and floating occlusion) in hierarchical layers based on the
  3D depth. These components are further encoded to canonical identity code,
  structured motion code and full scene code, which are utilized as control
  signals of synthesis process. The design of spatial decomposed modeling enables
  flexible user control, complex motion expression, as well as 3D-aware synthesis
  for scene interactions. Experimental results demonstrate effectiveness and
  robustness of the proposed method.
paper_authors._id: 66f38dac487d5d062b4a438c
paper_authors.name: Yifang Men
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T10:34:16.238Z
paper_authors._id: 6582d2a7d4bce0d73a5e4df4
paper_authors.avatarUrl: /avatars/cc9d827d2b62c0276d3817c90025d163.svg
paper_authors.fullname: Yifang Men
paper_authors.type: user
paper_authors.user: menyifang
paper_authors._id: 66f38dac487d5d062b4a438d
paper_authors.name: Yuan Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T12:49:37.728Z
paper_authors._id: 645320d6c895e6437dc8919f
paper_authors.avatarUrl: /avatars/2e0749905f06f294706ef2f8dfafac64.svg
paper_authors.fullname: Yuan Yao
paper_authors.type: user
paper_authors.user: yaoyuan13
paper_authors._id: 66f38dac487d5d062b4a438e
paper_authors.name: Miaomiao Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T10:35:07.521Z
paper_authors._id: 66c313e4a15b4eed7f518f1a
paper_authors.avatarUrl: /avatars/17a05b0a92ff5cb5afc337af995e4ece.svg
paper_authors.fullname: miaomiao cui
paper_authors.type: user
paper_authors.user: cuimiao1990
paper_authors._id: 66f38dac487d5d062b4a438f
paper_authors.name: Liefeng Bo
paper_title: MIMO: Controllable Character Video Synthesis with Spatial Decomposed
    Modeling
paper_summary: Character video synthesis aims to produce realistic videos of animatable
  characters within lifelike scenes. As a fundamental problem in the computer
  vision and graphics community, 3D works typically require multi-view captures
  for per-case training, which severely limits their applicability of modeling
  arbitrary characters in a short time. Recent 2D methods break this limitation
  via pre-trained diffusion models, but they struggle for pose generality and
  scene interaction. To this end, we propose MIMO, a novel framework which can
  not only synthesize character videos with controllable attributes (i.e.,
  character, motion and scene) provided by simple user inputs, but also
  simultaneously achieve advanced scalability to arbitrary characters, generality
  to novel 3D motions, and applicability to interactive real-world scenes in a
  unified framework. The core idea is to encode the 2D video to compact spatial
  codes, considering the inherent 3D nature of video occurrence. Concretely, we
  lift the 2D frame pixels into 3D using monocular depth estimators, and
  decompose the video clip to three spatial components (i.e., main human,
  underlying scene, and floating occlusion) in hierarchical layers based on the
  3D depth. These components are further encoded to canonical identity code,
  structured motion code and full scene code, which are utilized as control
  signals of synthesis process. The design of spatial decomposed modeling enables
  flexible user control, complex motion expression, as well as 3D-aware synthesis
  for scene interactions. Experimental results demonstrate effectiveness and
  robustness of the proposed method.
paper_ai_keywords: diffusion models spatial codes monocular depth estimators spatial decomposition canonical identity code structured motion code full scene code 3D-aware synthesis