Fusing uncalibrated IMUs and handheld smartphone video to reconstruct knee kinematics
<|reference_start|>Fusing uncalibrated IMUs and handheld smartphone video to reconstruct knee kinematics: Video and wearable sensor data provide complementary information about human movement. Video provides a holistic understanding of the entire body in the world while wearable sensors provide high-resolution measurements of specific body segments. A robust method to fuse these modalities and obtain biomechanically accurate kinematics would have substantial utility for clinical assessment and monitoring. While multiple video-sensor fusion methods exist, most assume that a time-intensive, and often brittle, sensor-body calibration process has already been performed. In this work, we present a method to combine handheld smartphone video and uncalibrated wearable sensor data at their full temporal resolution. Our monocular, video-only, biomechanical reconstruction already performs well, with only several degrees of error at the knee during walking compared to markerless motion capture. Reconstructing from a fusion of video and wearable sensor data further reduces this error. We validate this in a mixture of people with no gait impairments, lower limb prosthesis users, and individuals with a history of stroke. We also show that sensor data allows tracking through periods of visual occlusion.<|reference_end|>
@article{peiffer2024fusing,
    title={Fusing uncalibrated IMUs and handheld smartphone video to reconstruct
  knee kinematics},
    author={J. D. Peiffer, Kunal Shah, Shawana Anarwala, Kayan Abdou and R. James
  Cotton},
    journal={arXiv preprint arXiv:2405.17368},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17368},
    primaryClass={cs.CV}
}