A system and method are described for adaptive language understanding multimodal language acquisition in human-computer interaction. Words, phrases, sentences, production rules (syntactic information) as well as their corresponding meanings (semantic information) are stored. New words, phrases, sentences, production rules and their corresponding meanings can be acquired through interaction with users, using different input modalities, such as, speech, taping, pointing, drawing and image capturing. This system therefore acquires language through a natural language and multimodal interaction with users. New language knowledge is acquired in two ways. First, by acquiring new linguistic units, i.e. words or phrases and their corresponding semantics, and second by aquiring new sentences or language rules and their corresponding computer actions. The system represents an adaptive spoken interface capable of interpeting the user's spoken commands and sensory inputs and of learning new linguistic concepts and production rules. Such a system and the underlying method can not only be used to build adptive interactive compputer interfaces and operating systems, expert systems and computer games.
SYSTEM AND METHOD FOR ADAPTIVE LANGUAGE UNDERSTANDING BY COMPUTERS