Controllable Human-centric Keyframe Interpolation with Generative Prior
Existing interpolation methods use pre-trained video diffusion priors to
generate intermediate frames between sparsely sampled keyframes. In the absence
of 3D geometric guidance, these methods struggle to produce plausible results
for complex, articulated human motions and offer limited control over the
synthesized dynamics. In this paper, we introduce PoseFuse3D Keyframe
Interpolator (PoseFuse3D-KI), a novel framework that integrates 3D human
guidance signals into the diffusion process for Controllable Human-centric
Keyframe Interpolation (CHKI). To provide rich spatial and structural cues for
interpolation, our PoseFuse3D, a 3D-informed control model, features a novel
SMPL-X encoder that transforms 3D geometry and shape into the 2D latent
conditioning space, alongside a fusion network that integrates these 3D cues
with 2D pose embeddings. For evaluation, we build CHKI-Video, a new dataset
annotated with both 2D poses and 3D SMPL-X parameters. We show that
PoseFuse3D-KI consistently outperforms state-of-the-art baselines on
CHKI-Video, achieving a 9% improvement in PSNR and a 38% reduction in LPIPS.
Comprehensive ablations demonstrate that our PoseFuse3D model improves
interpolation fidelity.
68407ac4769b3772e8fdb16a
Zujin Guo
claimed_verified
2025-06-05T07:27:36.035Z
62f4b57a9437c4d04fb5e6d6
/avatars/3681e4948fba1317c80bbbf2326a5fcd.svg
ZUJIN GUO
user
GSean
68407ac4769b3772e8fdb16b
Size Wu
68407ac4769b3772e8fdb16c
Zhongang Cai
68407ac4769b3772e8fdb16d
Wei Li
68407ac4769b3772e8fdb16e
Chen Change Loy
Controllable Human-centric Keyframe Interpolation with Generative Prior
Existing interpolation methods use pre-trained video diffusion priors to
generate intermediate frames between sparsely sampled keyframes. In the absence
of 3D geometric guidance, these methods struggle to produce plausible results
for complex, articulated human motions and offer limited control over the
synthesized dynamics. In this paper, we introduce PoseFuse3D Keyframe
Interpolator (PoseFuse3D-KI), a novel framework that integrates 3D human
guidance signals into the diffusion process for Controllable Human-centric
Keyframe Interpolation (CHKI). To provide rich spatial and structural cues for
interpolation, our PoseFuse3D, a 3D-informed control model, features a novel
SMPL-X encoder that transforms 3D geometry and shape into the 2D latent
conditioning space, alongside a fusion network that integrates these 3D cues
with 2D pose embeddings. For evaluation, we build CHKI-Video, a new dataset
annotated with both 2D poses and 3D SMPL-X parameters. We show that
PoseFuse3D-KI consistently outperforms state-of-the-art baselines on
CHKI-Video, achieving a 9% improvement in PSNR and a 38% reduction in LPIPS.
Comprehensive ablations demonstrate that our PoseFuse3D model improves
interpolation fidelity.
PoseFuse3D Keyframe Interpolator integrates 3D human guidance into diffusion models for more accurate and controllable video frame interpolation.
video diffusion priors interpolation 3D human guidance PoseFuse3D Keyframe Interpolator Controllable Human-centric Keyframe Interpolation CHKI SMPL-X encoder 3D-informed control model fusion network 2D pose embeddings CHKI-Video PSNR LPIPS