  Model-based human pose estimation is currently approached through two
different paradigms. Optimization-based methods fit a parametric body model to
2D observations in an iterative manner, leading to accurate image-model
alignments, but are often slow and sensitive to the initialization. In
contrast, regression-based methods, that use a deep network to directly
estimate the model parameters from pixels, tend to provide reasonable, but not
pixel accurate, results while requiring huge amounts of supervision. In this
work, instead of investigating which approach is better, our key insight is
that the two paradigms can form a strong collaboration. A reasonable, directly
regressed estimate from the network can initialize the iterative optimization
making the fitting faster and more accurate. Similarly, a pixel accurate fit
from iterative optimization can act as strong supervision for the network. This
is the core of our proposed approach SPIN (SMPL oPtimization IN the loop). The
deep network initializes an iterative optimization routine that fits the body
model to 2D joints within the training loop, and the fitted estimate is
subsequently used to supervise the network. Our approach is self-improving by
nature, since better network estimates can lead the optimization to better
solutions, while more accurate optimization fits provide better supervision for
the network. We demonstrate the effectiveness of our approach in different
settings, where 3D ground truth is scarce, or not available, and we
consistently outperform the state-of-the-art model-based pose estimation
approaches by significant margins. The project website with videos, results,
and code can be found at https://seas.upenn.edu/~nkolot/projects/spin.

Computer Vision and Pattern Recognition