Instant Multi-View Head Capture through Learnable Registration
Existing methods for capturing datasets of 3D heads in dense semantic
correspondence are slow, and commonly address the problem in two separate
steps; multi-view stereo (MVS) reconstruction followed by non-rigid
registration. To simplify this process, we introduce TEMPEH (Towards Estimation
of 3D Meshes from Performances of Expressive Heads) to directly infer 3D heads
in dense correspondence from calibrated multi-view images. Registering datasets
of 3D scans typically requires manual parameter tuning to find the right
balance between accurately fitting the scans surfaces and being robust to
scanning noise and outliers. Instead, we propose to jointly register a 3D head
dataset while training TEMPEH. Specifically, during training we minimize a
geometric loss commonly used for surface registration, effectively leveraging
TEMPEH as a regularizer. Our multi-view head inference builds on a volumetric
feature representation that samples and fuses features from each view using
camera calibration information. To account for partial occlusions and a large
capture volume that enables head movements, we use view- and surface-aware
feature fusion, and a spatial transformer-based head localization module,
respectively. We use raw MVS scans as supervision during training, but, once
trained, TEMPEH directly predicts 3D heads in dense correspondence without
requiring scans. Predicting one head takes about 0.3 seconds with a median
reconstruction error of 0.26 mm, 64% lower than the current state-of-the-art.
This enables the efficient capture of large datasets containing multiple people
and diverse facial motions. Code, model, and data are publicly available at
https://tempeh.is.tue.mpg.de.
6489301d50396a20780960d4
Timo Bolkart
claimed_verified
2023-06-16T07:04:58.495Z
648b73eb1c3e38f18eb39988
/avatars/ac7b858f504648c1f5d1c3e3034c7996.svg
Timo Bolkart
user
tbolkart
6489301d50396a20780960d5
Tianye Li
6489301d50396a20780960d6
Michael J. Black
Instant Multi-View Head Capture through Learnable Registration
Existing methods for capturing datasets of 3D heads in dense semantic
correspondence are slow, and commonly address the problem in two separate
steps; multi-view stereo (MVS) reconstruction followed by non-rigid
registration. To simplify this process, we introduce TEMPEH (Towards Estimation
of 3D Meshes from Performances of Expressive Heads) to directly infer 3D heads
in dense correspondence from calibrated multi-view images. Registering datasets
of 3D scans typically requires manual parameter tuning to find the right
balance between accurately fitting the scans surfaces and being robust to
scanning noise and outliers. Instead, we propose to jointly register a 3D head
dataset while training TEMPEH. Specifically, during training we minimize a
geometric loss commonly used for surface registration, effectively leveraging
TEMPEH as a regularizer. Our multi-view head inference builds on a volumetric
feature representation that samples and fuses features from each view using
camera calibration information. To account for partial occlusions and a large
capture volume that enables head movements, we use view- and surface-aware
feature fusion, and a spatial transformer-based head localization module,
respectively. We use raw MVS scans as supervision during training, but, once
trained, TEMPEH directly predicts 3D heads in dense correspondence without
requiring scans. Predicting one head takes about 0.3 seconds with a median
reconstruction error of 0.26 mm, 64% lower than the current state-of-the-art.
This enables the efficient capture of large datasets containing multiple people
and diverse facial motions. Code, model, and data are publicly available at
https://tempeh.is.tue.mpg.de.
TEMPEH directly infers 3D heads in dense correspondence from calibrated multi-view images, integrating surface registration, volumetric feature representation, and spatial transformer-based localization with efficient features fusion, achieving high accuracy and reducing computational time.
multi-view stereo non-rigid registration 3D mesh estimation dense correspondence geometrical loss surface registration volumetric feature representation feature fusion partial occlusions capture volume spatial transformer raw MVS scans 3D prediction head localization