  Wearable sensor-based Human Action Recognition (HAR) has made significant
strides in recent times. However, the accuracy performance of wearable
sensor-based HAR is currently still lagging behind that of visual
modalities-based systems, such as RGB video and depth data. Although diverse
input modalities can provide complementary cues and improve the accuracy
performance of HAR, wearable devices can only capture limited kinds of
non-visual time series input, such as accelerometers and gyroscopes. This
limitation hinders the deployment of multimodal simultaneously using visual and
non-visual modality data in parallel on current wearable devices. To address
this issue, we propose a novel Physical-aware Cross-modal Adversarial (PCA)
framework that utilizes only time-series accelerometer data from four inertial
sensors for the wearable sensor-based HAR problem. Specifically, we propose an
effective IMU2SKELETON network to produce corresponding synthetic skeleton
joints from accelerometer data. Subsequently, we imposed additional constraints
on the synthetic skeleton data from a physical perspective, as accelerometer
data can be regarded as the second derivative of the skeleton sequence
coordinates. After that, the original accelerometer as well as the constrained
skeleton sequence were fused together to make the final classification. In this
way, when individuals wear wearable devices, the devices can not only capture
accelerometer data, but can also generate synthetic skeleton sequences for
real-time wearable sensor-based HAR applications that need to be conducted
anytime and anywhere. To demonstrate the effectiveness of our proposed PCA
framework, we conduct extensive experiments on Berkeley-MHAD, UTD-MHAD, and
MMAct datasets. The results confirm that the proposed PCA approach has
competitive performance compared to the previous methods on the mono
sensor-based HAR classification problem.

Multimedia;Human-Computer Interaction