MIMO: Controllable Character Video Synthesis with Spatial Decomposed
  Modeling
Character video synthesis aims to produce realistic videos of animatable
characters within lifelike scenes. As a fundamental problem in the computer
vision and graphics community, 3D works typically require multi-view captures
for per-case training, which severely limits their applicability of modeling
arbitrary characters in a short time. Recent 2D methods break this limitation
via pre-trained diffusion models, but they struggle for pose generality and
scene interaction. To this end, we propose MIMO, a novel framework which can
not only synthesize character videos with controllable attributes (i.e.,
character, motion and scene) provided by simple user inputs, but also
simultaneously achieve advanced scalability to arbitrary characters, generality
to novel 3D motions, and applicability to interactive real-world scenes in a
unified framework. The core idea is to encode the 2D video to compact spatial
codes, considering the inherent 3D nature of video occurrence. Concretely, we
lift the 2D frame pixels into 3D using monocular depth estimators, and
decompose the video clip to three spatial components (i.e., main human,
underlying scene, and floating occlusion) in hierarchical layers based on the
3D depth. These components are further encoded to canonical identity code,
structured motion code and full scene code, which are utilized as control
signals of synthesis process. The design of spatial decomposed modeling enables
flexible user control, complex motion expression, as well as 3D-aware synthesis
for scene interactions. Experimental results demonstrate effectiveness and
robustness of the proposed method.
66f38dac487d5d062b4a438c
Yifang Men
admin_assigned
2024-09-26T10:34:16.238Z
6582d2a7d4bce0d73a5e4df4
/avatars/cc9d827d2b62c0276d3817c90025d163.svg
Yifang Men
user
menyifang
66f38dac487d5d062b4a438d
Yuan Yao
claimed_verified
2024-09-26T12:49:37.728Z
645320d6c895e6437dc8919f
/avatars/2e0749905f06f294706ef2f8dfafac64.svg
Yuan Yao
user
yaoyuan13
66f38dac487d5d062b4a438e
Miaomiao Cui
admin_assigned
2024-09-26T10:35:07.521Z
66c313e4a15b4eed7f518f1a
/avatars/17a05b0a92ff5cb5afc337af995e4ece.svg
miaomiao cui
user
cuimiao1990
66f38dac487d5d062b4a438f
Liefeng Bo
MIMO: Controllable Character Video Synthesis with Spatial Decomposed
  Modeling
Character video synthesis aims to produce realistic videos of animatable
characters within lifelike scenes. As a fundamental problem in the computer
vision and graphics community, 3D works typically require multi-view captures
for per-case training, which severely limits their applicability of modeling
arbitrary characters in a short time. Recent 2D methods break this limitation
via pre-trained diffusion models, but they struggle for pose generality and
scene interaction. To this end, we propose MIMO, a novel framework which can
not only synthesize character videos with controllable attributes (i.e.,
character, motion and scene) provided by simple user inputs, but also
simultaneously achieve advanced scalability to arbitrary characters, generality
to novel 3D motions, and applicability to interactive real-world scenes in a
unified framework. The core idea is to encode the 2D video to compact spatial
codes, considering the inherent 3D nature of video occurrence. Concretely, we
lift the 2D frame pixels into 3D using monocular depth estimators, and
decompose the video clip to three spatial components (i.e., main human,
underlying scene, and floating occlusion) in hierarchical layers based on the
3D depth. These components are further encoded to canonical identity code,
structured motion code and full scene code, which are utilized as control
signals of synthesis process. The design of spatial decomposed modeling enables
flexible user control, complex motion expression, as well as 3D-aware synthesis
for scene interactions. Experimental results demonstrate effectiveness and
robustness of the proposed method.
A novel framework called MIMO synthesizes character videos with controllable attributes using 2D encodings and 3D depth, achieving scalability, generality, and applicability to interactive scenes.
character video synthesis diffusion models monocular depth estimators spatial codes 3D depth hierarchical layers canonical identity code structured motion code full scene code 3D-aware synthesis