  We present XFormer, a novel human mesh and motion capture method that
achieves real-time performance on consumer CPUs given only monocular images as
input. The proposed network architecture contains two branches: a keypoint
branch that estimates 3D human mesh vertices given 2D keypoints, and an image
branch that makes predictions directly from the RGB image features. At the core
of our method is a cross-modal transformer block that allows information to
flow across these two branches by modeling the attention between 2D keypoint
coordinates and image spatial features. Our architecture is smartly designed,
which enables us to train on various types of datasets including images with
2D/3D annotations, images with 3D pseudo labels, and motion capture datasets
that do not have associated images. This effectively improves the accuracy and
generalization ability of our system. Built on a lightweight backbone
(MobileNetV3), our method runs blazing fast (over 30fps on a single CPU core)
and still yields competitive accuracy. Furthermore, with an HRNet backbone,
XFormer delivers state-of-the-art performance on Huamn3.6 and 3DPW datasets.

Computer Vision and Pattern Recognition