title: Instant Multi-View Head Capture through Learnable Registration
summary: Existing methods for capturing datasets of 3D heads in dense semantic
  correspondence are slow, and commonly address the problem in two separate
  steps; multi-view stereo (MVS) reconstruction followed by non-rigid
  registration. To simplify this process, we introduce TEMPEH (Towards Estimation
  of 3D Meshes from Performances of Expressive Heads) to directly infer 3D heads
  in dense correspondence from calibrated multi-view images. Registering datasets
  of 3D scans typically requires manual parameter tuning to find the right
  balance between accurately fitting the scans surfaces and being robust to
  scanning noise and outliers. Instead, we propose to jointly register a 3D head
  dataset while training TEMPEH. Specifically, during training we minimize a
  geometric loss commonly used for surface registration, effectively leveraging
  TEMPEH as a regularizer. Our multi-view head inference builds on a volumetric
  feature representation that samples and fuses features from each view using
  camera calibration information. To account for partial occlusions and a large
  capture volume that enables head movements, we use view- and surface-aware
  feature fusion, and a spatial transformer-based head localization module,
  respectively. We use raw MVS scans as supervision during training, but, once
  trained, TEMPEH directly predicts 3D heads in dense correspondence without
  requiring scans. Predicting one head takes about 0.3 seconds with a median
  reconstruction error of 0.26 mm, 64% lower than the current state-of-the-art.
  This enables the efficient capture of large datasets containing multiple people
  and diverse facial motions. Code, model, and data are publicly available at
  https://tempeh.is.tue.mpg.de.
paper_authors._id: 6489301d50396a20780960d4
paper_authors.name: Timo Bolkart
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T07:04:58.495Z
paper_authors._id: 648b73eb1c3e38f18eb39988
paper_authors.avatarUrl: /avatars/ac7b858f504648c1f5d1c3e3034c7996.svg
paper_authors.fullname: Timo Bolkart
paper_authors.type: user
paper_authors.user: tbolkart
paper_authors._id: 6489301d50396a20780960d5
paper_authors.name: Tianye Li
paper_authors._id: 6489301d50396a20780960d6
paper_authors.name: Michael J. Black
paper_title: Instant Multi-View Head Capture through Learnable Registration
paper_summary: Existing methods for capturing datasets of 3D heads in dense semantic
  correspondence are slow, and commonly address the problem in two separate
  steps; multi-view stereo (MVS) reconstruction followed by non-rigid
  registration. To simplify this process, we introduce TEMPEH (Towards Estimation
  of 3D Meshes from Performances of Expressive Heads) to directly infer 3D heads
  in dense correspondence from calibrated multi-view images. Registering datasets
  of 3D scans typically requires manual parameter tuning to find the right
  balance between accurately fitting the scans surfaces and being robust to
  scanning noise and outliers. Instead, we propose to jointly register a 3D head
  dataset while training TEMPEH. Specifically, during training we minimize a
  geometric loss commonly used for surface registration, effectively leveraging
  TEMPEH as a regularizer. Our multi-view head inference builds on a volumetric
  feature representation that samples and fuses features from each view using
  camera calibration information. To account for partial occlusions and a large
  capture volume that enables head movements, we use view- and surface-aware
  feature fusion, and a spatial transformer-based head localization module,
  respectively. We use raw MVS scans as supervision during training, but, once
  trained, TEMPEH directly predicts 3D heads in dense correspondence without
  requiring scans. Predicting one head takes about 0.3 seconds with a median
  reconstruction error of 0.26 mm, 64% lower than the current state-of-the-art.
  This enables the efficient capture of large datasets containing multiple people
  and diverse facial motions. Code, model, and data are publicly available at
  https://tempeh.is.tue.mpg.de.
paper_ai_keywords: multi-view stereo (MVS) non-rigid registration 3D mesh estimation calibrated multi-view images geometric loss surface registration volumetric feature representation view-aware feature fusion surface-aware feature fusion spatial transformer head localization raw MVS scans dense correspondence 3D head inference