leased, labeled datasets exist. However, as prior published
works only train on at most a handful of them, it remains un-
known what performance could be achieved by combining
more than a decade of dataset collection efforts into a sin-
gle model. Unfortunately, this is not a trivial undertaking,
since different datasets do not use the same skeleton format
for their labels (see Fig. 1), e.g., the hip keypoints are at
different heights, some body parts are only labeled in some
datasets, some provide surface markers while others provide
keypoints inside the body, etc. Prior work has rectiﬁed such

CMU-PanopticHuman3.6M3DPW (SMPL)BML MoViBerkeley MHAD3D latent points(a) Separate skeleton prediction

Front view

Right side view

(b) With our proposed ACAE regularization
Front view

Right side view
Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats