A system for building computational models of a goal-driven task from demonstration is disclosed. A task recording subsystem receives a recorded video file or recorded sensor data representative of an expert demonstration for a task. An instructor authoring tool generates one or more sub-activity proposals; enables an instructor to specify one or more sub-activity labels upon modification of the one or more sub-activity proposals into one or more sub-tasks. A task learning subsystem learns the one or more sub-tasks represented in the demonstration of the task; builds an activity model to predict and locate the task being performed in the recorded video file. A task evaluation subsystem evaluates a live video representative of the task; generates at least one performance description statistics; identifies a type of activity step executed by the one or more actors; provides an activity guidance feedback in real-time to the one or more actors.
System and method for building computational models of a goal-driven task from demonstration