A method and system designed to augment, rather than replace, the work habits of its users. These work habits include practices such as drawing on Post-it(TM) notes using a symbolic language. The system observes and understands the language used on the Post-it notes and the system assigns meaning simultaneously to objects in both the physical and virtual worlds. Since the data is preserved in physical form, the physical documents serve as a back-up in the case of electronic system failure. With the present invention users can rollout a primary paper document such as a map, register it, and place secondary physical documents on the map. The secondary physical documents can be moved from one place to another on the primary document. Once an object is augmented, users can modify the meaning represented by it, ask questions about that representation, view it in virtual reality, or give directions to it, all with speech and gestures.
Augmenting and not replacing paper based work practice via multi-modal interaction