  In this paper, we focus on unsupervised representation learning for
skeleton-based action recognition. Existing approaches usually learn action
representations by sequential prediction but they suffer from the inability to
fully learn semantic information. To address this limitation, we propose a
novel framework named Prototypical Contrast and Reverse Prediction (PCRP),
which not only creates reverse sequential prediction to learn low-level
information (e.g., body posture at every frame) and high-level pattern (e.g.,
motion order), but also devises action prototypes to implicitly encode semantic
similarity shared among sequences. In general, we regard action prototypes as
latent variables and formulate PCRP as an expectation-maximization task.
Specifically, PCRP iteratively runs (1) E-step as determining the distribution
of prototypes by clustering action encoding from the encoder, and (2) M-step as
optimizing the encoder by minimizing the proposed ProtoMAE loss, which helps
simultaneously pull the action encoding closer to its assigned prototype and
perform reverse prediction task. Extensive experiments on N-UCLA, NTU 60, and
NTU 120 dataset present that PCRP outperforms state-of-the-art unsupervised
methods and even achieves superior performance over some of supervised methods.
Codes are available at https://github.com/Mikexu007/PCRP.

Computer Vision and Pattern Recognition