title: In-2-4D: Inbetweening from Two Single-View Images to 4D Generation
summary: We propose a new problem, In-2-4D, for generative 4D (i.e., 3D + motion)
  inbetweening from a minimalistic input setting: two single-view images
  capturing an object in two distinct motion states. Given two images
  representing the start and end states of an object in motion, our goal is to
  generate and reconstruct the motion in 4D. We utilize a video interpolation
  model to predict the motion, but large frame-to-frame motions can lead to
  ambiguous interpretations. To overcome this, we employ a hierarchical approach
  to identify keyframes that are visually close to the input states and show
  significant motion, then generate smooth fragments between them. For each
  fragment, we construct the 3D representation of the keyframe using Gaussian
  Splatting. The temporal frames within the fragment guide the motion, enabling
  their transformation into dynamic Gaussians through a deformation field. To
  improve temporal consistency and refine 3D motion, we expand the self-attention
  of multi-view diffusion across timesteps and apply rigid transformation
  regularization. Finally, we merge the independently generated 3D motion
  segments by interpolating boundary deformation fields and optimizing them to
  align with the guiding video, ensuring smooth and flicker-free transitions.
  Through extensive qualitative and quantitiave experiments as well as a user
  study, we show the effectiveness of our method and its components. The project
  page is available at https://in-2-4d.github.io/
paper_authors._id: 67fc6d8d74c3c0f0d6f24c3b
paper_authors.name: Sauradip Nag
paper_authors._id: 67fc6d8d74c3c0f0d6f24c3c
paper_authors.name: Daniel Cohen-Or
paper_authors._id: 67fc6d8d74c3c0f0d6f24c3d
paper_authors.name: Hao Zhang
paper_authors._id: 67fc6d8d74c3c0f0d6f24c3e
paper_authors.name: Ali Mahdavi-Amiri
paper_title: In-2-4D: Inbetweening from Two Single-View Images to 4D Generation
paper_summary: We propose a new problem, In-2-4D, for generative 4D (i.e., 3D + motion)
  inbetweening from a minimalistic input setting: two single-view images
  capturing an object in two distinct motion states. Given two images
  representing the start and end states of an object in motion, our goal is to
  generate and reconstruct the motion in 4D. We utilize a video interpolation
  model to predict the motion, but large frame-to-frame motions can lead to
  ambiguous interpretations. To overcome this, we employ a hierarchical approach
  to identify keyframes that are visually close to the input states and show
  significant motion, then generate smooth fragments between them. For each
  fragment, we construct the 3D representation of the keyframe using Gaussian
  Splatting. The temporal frames within the fragment guide the motion, enabling
  their transformation into dynamic Gaussians through a deformation field. To
  improve temporal consistency and refine 3D motion, we expand the self-attention
  of multi-view diffusion across timesteps and apply rigid transformation
  regularization. Finally, we merge the independently generated 3D motion
  segments by interpolating boundary deformation fields and optimizing them to
  align with the guiding video, ensuring smooth and flicker-free transitions.
  Through extensive qualitative and quantitiave experiments as well as a user
  study, we show the effectiveness of our method and its components. The project
  page is available at https://in-2-4d.github.io/
paper_ai_summary: A hierarchical 4D inbetweening approach generates smooth motion from two input images using Gaussian Splatting, multi-view diffusion, and self-attention across timesteps.
paper_ai_keywords: Gaussian Splatting multi-view diffusion self-attention deformation field rigid transformation regularization