Learning the 3D Fauna of the Web
Learning 3D models of all animals on the Earth requires massively scaling up
existing solutions. With this ultimate goal in mind, we develop 3D-Fauna, an
approach that learns a pan-category deformable 3D animal model for more than
100 animal species jointly. One crucial bottleneck of modeling animals is the
limited availability of training data, which we overcome by simply learning
from 2D Internet images. We show that prior category-specific attempts fail to
generalize to rare species with limited training images. We address this
challenge by introducing the Semantic Bank of Skinned Models (SBSM), which
automatically discovers a small set of base animal shapes by combining
geometric inductive priors with semantic knowledge implicitly captured by an
off-the-shelf self-supervised feature extractor. To train such a model, we also
contribute a new large-scale dataset of diverse animal species. At inference
time, given a single image of any quadruped animal, our model reconstructs an
articulated 3D mesh in a feed-forward fashion within seconds.
65979bb1185db94370c57f40
Zizhang Li
65979bb1185db94370c57f41
Dor Litvak
65979bb1185db94370c57f42
Ruining Li
claimed_verified
2024-01-08T08:47:58.291Z
6467cc17e7a6a374fd1a41e5
/avatars/3cbd4ad5bc17067ee38b13d0cbadd80c.svg
Ruining Li
user
rayli
65979bb1185db94370c57f43
Yunzhi Zhang
claimed_verified
2024-02-22T22:28:22.133Z
63793aee2cb50dda39d6e73b
/avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
Yunzhi Zhang
user
Yunzhi
65979bb1185db94370c57f44
Tomas Jakab
65979bb1185db94370c57f45
Christian Rupprecht
65979bb1185db94370c57f46
Shangzhe Wu
65979bb1185db94370c57f47
Andrea Vedaldi
65979bb1185db94370c57f48
Jiajun Wu
Learning the 3D Fauna of the Web
Learning 3D models of all animals on the Earth requires massively scaling up
existing solutions. With this ultimate goal in mind, we develop 3D-Fauna, an
approach that learns a pan-category deformable 3D animal model for more than
100 animal species jointly. One crucial bottleneck of modeling animals is the
limited availability of training data, which we overcome by simply learning
from 2D Internet images. We show that prior category-specific attempts fail to
generalize to rare species with limited training images. We address this
challenge by introducing the Semantic Bank of Skinned Models (SBSM), which
automatically discovers a small set of base animal shapes by combining
geometric inductive priors with semantic knowledge implicitly captured by an
off-the-shelf self-supervised feature extractor. To train such a model, we also
contribute a new large-scale dataset of diverse animal species. At inference
time, given a single image of any quadruped animal, our model reconstructs an
articulated 3D mesh in a feed-forward fashion within seconds.
3D-Fauna learns a deformable 3D animal model using 2D images and the Semantic Bank of Skinned Models to handle diverse and rare animal species efficiently.
deformable 3D model Semantic Bank of Skinned Models geometric inductive priors articuated 3D mesh feed-forward fashion