Multi-modal perception for soft robotic interactions using generative models
<|reference_start|>Multi-modal perception for soft robotic interactions using generative models: Perception is essential for the active interaction of physical agents with the external environment. The integration of multiple sensory modalities, such as touch and vision, enhances this perceptual process, creating a more comprehensive and robust understanding of the world. Such fusion is particularly useful for highly deformable bodies such as soft robots. Developing a compact, yet comprehensive state representation from multi-sensory inputs can pave the way for the development of complex control strategies. This paper introduces a perception model that harmonizes data from diverse modalities to build a holistic state representation and assimilate essential information. The model relies on the causality between sensory input and robotic actions, employing a generative model to efficiently compress fused information and predict the next observation. We present, for the first time, a study on how touch can be predicted from vision and proprioception on soft robots, the importance of the cross-modal generation and why this is essential for soft robotic interactions in unstructured environments.<|reference_end|>
@article{donato2024multi-modal,
    title={Multi-modal perception for soft robotic interactions using generative
  models},
    author={Enrico Donato and Egidio Falotico and Thomas George Thuruthel},
    journal={arXiv preprint arXiv:2404.04220},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.04220},
    primaryClass={cs.RO cs.AI cs.LG}
}