title: DressRecon: Freeform 4D Human Reconstruction from Monocular Video
summary: We present a method to reconstruct time-consistent human body models from
  monocular videos, focusing on extremely loose clothing or handheld object
  interactions. Prior work in human reconstruction is either limited to tight
  clothing with no object interactions, or requires calibrated multi-view
  captures or personalized template scans which are costly to collect at scale.
  Our key insight for high-quality yet flexible reconstruction is the careful
  combination of generic human priors about articulated body shape (learned from
  large-scale training data) with video-specific articulated "bag-of-bones"
  deformation (fit to a single video via test-time optimization). We accomplish
  this by learning a neural implicit model that disentangles body versus clothing
  deformations as separate motion model layers. To capture subtle geometry of
  clothing, we leverage image-based priors such as human body pose, surface
  normals, and optical flow during optimization. The resulting neural fields can
  be extracted into time-consistent meshes, or further optimized as explicit 3D
  Gaussians for high-fidelity interactive rendering. On datasets with highly
  challenging clothing deformations and object interactions, DressRecon yields
  higher-fidelity 3D reconstructions than prior art. Project page:
  https://jefftan969.github.io/dressrecon/
paper_authors._id: 66fcb1ed63002e1de06055e3
paper_authors.name: Jeff Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T12:44:43.969Z
paper_authors._id: 6616db12f60442df33b7035c
paper_authors.avatarUrl: /avatars/e0737dc8cff7ddb54ebb65447da4783b.svg
paper_authors.fullname: Jeff Tan
paper_authors.type: user
paper_authors.user: jefftan969
paper_authors._id: 66fcb1ed63002e1de06055e4
paper_authors.name: Donglai Xiang
paper_authors._id: 66fcb1ed63002e1de06055e5
paper_authors.name: Shubham Tulsiani
paper_authors._id: 66fcb1ed63002e1de06055e6
paper_authors.name: Deva Ramanan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:13:35.973Z
paper_authors._id: 6337151b0267ebcf02640eb6
paper_authors.avatarUrl: /avatars/14a723cafc5587043bdfb19304fc202d.svg
paper_authors.fullname: Deva Ramanan
paper_authors.type: user
paper_authors.user: devakramanan
paper_authors._id: 66fcb1ed63002e1de06055e7
paper_authors.name: Gengshan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:13:29.334Z
paper_authors._id: 62c0597543d4858f821bc87a
paper_authors.avatarUrl: /avatars/ce9d28c08a5c9e8312a754714bae4634.svg
paper_authors.fullname: Gengshan Yang
paper_authors.type: user
paper_authors.user: gengshan-y
paper_title: DressRecon: Freeform 4D Human Reconstruction from Monocular Video
paper_summary: We present a method to reconstruct time-consistent human body models from
  monocular videos, focusing on extremely loose clothing or handheld object
  interactions. Prior work in human reconstruction is either limited to tight
  clothing with no object interactions, or requires calibrated multi-view
  captures or personalized template scans which are costly to collect at scale.
  Our key insight for high-quality yet flexible reconstruction is the careful
  combination of generic human priors about articulated body shape (learned from
  large-scale training data) with video-specific articulated "bag-of-bones"
  deformation (fit to a single video via test-time optimization). We accomplish
  this by learning a neural implicit model that disentangles body versus clothing
  deformations as separate motion model layers. To capture subtle geometry of
  clothing, we leverage image-based priors such as human body pose, surface
  normals, and optical flow during optimization. The resulting neural fields can
  be extracted into time-consistent meshes, or further optimized as explicit 3D
  Gaussians for high-fidelity interactive rendering. On datasets with highly
  challenging clothing deformations and object interactions, DressRecon yields
  higher-fidelity 3D reconstructions than prior art. Project page:
  https://jefftan969.github.io/dressrecon/
paper_ai_keywords: neural implicit model bag-of-bones deformation test-time optimization motion model layers human body pose surface normals optical flow neural fields time-consistent meshes explicit 3D Gaussians high-fidelity interactive rendering