The disclosure describes an overall system and method for designing and building multi-sense systems using a generalized synchrony based fusion technique. By treating “time” as a common thread that runs across all the modes, the invention uses trigger inputs to form several groups of inputs across all modes. The best group is selected as the one that has the maximize synchrony as determined by a combination of a weight for the group and a timing correlation of inputs within that group. The features from each mode within the best group are then used individually or jointly for pattern recognition or understanding. The result is a robust practically implementable generalized theory of multi-sense fusion. The proposed invention has immense potential to leap frog single input systems and revolutionize human-machine interactions.
METHOD FOR MULTI-SENSE FUSION USING SYNCHRONY