  Objective: For lower arm amputees, robotic prosthetic hands promise to regain
the capability to perform daily living activities. Current control methods
based on physiological signals such as electromyography (EMG) are prone to
yielding poor inference outcomes due to motion artifacts, muscle fatigue, and
many more. Vision sensors are a major source of information about the
environment state and can play a vital role in inferring feasible and intended
gestures. However, visual evidence is also susceptible to its own artifacts,
most often due to object occlusion, lighting changes, etc. Multimodal evidence
fusion using physiological and vision sensor measurements is a natural approach
due to the complementary strengths of these modalities. Methods: In this paper,
we present a Bayesian evidence fusion framework for grasp intent inference
using eye-view video, eye-gaze, and EMG from the forearm processed by neural
network models. We analyze individual and fused performance as a function of
time as the hand approaches the object to grasp it. For this purpose, we have
also developed novel data processing and augmentation techniques to train
neural network components. Results: Our results indicate that, on average,
fusion improves the instantaneous upcoming grasp type classification accuracy
while in the reaching phase by 13.66% and 14.8%, relative to EMG and visual
evidence individually, resulting in an overall fusion accuracy of 95.3%.
Conclusion: Our experimental data analyses demonstrate that EMG and visual
evidence show complementary strengths, and as a consequence, fusion of
multimodal evidence can outperform each individual evidence modality at any
given time.

Robotics;Artificial Intelligence;Computer Vision and Pattern Recognition;Human-Computer Interaction;Signal Processing