A natural human-computer interaction system based on multi-sensing data fusion, comprising an MEMS arm tracking device, a visual tracking device, a force feedback device, and a PC terminal. The MEMS arm tracking device consists of three groups of independent MEMS sensors for collecting arm joint angle information and measuring an arm movement track. The visual tracking device consists of a binocular camera for collecting image information and measuring a finger movement track. The force feedback device is mounted on the palm of an operator for providing force feedback to the finger. The PC terminal comprises a data display module, an arm movement calculating module, an image processing module, a mechanical calculating module, and a virtual scene rendering module. The system tracks the arm movement track of the operator by taking advantage of wide measurement range of the MEMS sensor and tracks the finger movement track of the operator by taking advantage of high precision of visual interaction, and provides force feedback interaction to the finger of the operator by means of the force feedback device, so that the sense of immersion of the operator can be improved.
NATURAL HUMAN-COMPUTER INTERACTION SYSTEM BASED ON MULTI-SENSING DATA FUSION