MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning
<|reference_start|>MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning: With the exponential growth of multimedia data, leveraging multimodal sensors presents a promising approach for improving accuracy in human activity recognition. Nevertheless, accurately identifying these activities using both video data and wearable sensor data presents challenges due to the labor-intensive data annotation, and reliance on external pretrained models or additional data. To address these challenges, we introduce Multimodal Masked Autoencoders-Based One-Shot Learning (Mu-MAE). Mu-MAE integrates a multimodal masked autoencoder with a synchronized masking strategy tailored for wearable sensors. This masking strategy compels the networks to capture more meaningful spatiotemporal features, which enables effective self-supervised pretraining without the need for external data. Furthermore, Mu-MAE leverages the representation extracted from multimodal masked autoencoders as prior information input to a cross-attention multimodal fusion layer. This fusion layer emphasizes spatiotemporal features requiring attention across different modalities while highlighting differences from other classes, aiding in the classification of various classes in metric-based one-shot learning. Comprehensive evaluations on MMAct one-shot classification show that Mu-MAE outperforms all the evaluated approaches, achieving up to an 80.17% accuracy for five-way one-shot multimodal classification, without the use of additional data.<|reference_end|>
@article{liu2024mu-mae:,
    title={MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning},
    author={Rex Liu, Xin Liu},
    journal={arXiv preprint arXiv:2408.04243},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.04243},
    primaryClass={cs.CV cs.MM}
}