Computer system user interfaces often utilize a virtual pointer to manipulate a graphical representation on a display screen. The virtual pointer can either be visible (as in the case when a peripheral device such as a mouse is used) or implied (as in the case when a touch-sensitive display screen is used). Each of these types of virtual pointers can support two distinct styles of user input: direct manipulation and gesturing.
In the case of direct manipulation, a user can either directly manipulate the graphical representation (e.g., by panning a map via dragging of the map) or directly manipulate a peripheral control to manipulate the graphical representation (e.g., by dragging a slider to adjust the zoom level at which a map is displayed).
In the case of gesturing, however, the motion of the pointer need not bear any relation to the underlying representation. That is, these gestures can be interpreted arbitrarily (e.g., drawing spirals anywhere may cause scrolling within a linear document).
However, each of these approaches has shortcomings. Direct manipulation requires that the pointer be used as a tool, which leads to the potentially problematic need to switch between tools. For instance, the screen must accommodate the icons or menu items representing the possible tool choices, thus consuming the valuable resource of display space. Furthermore, in the case of direct manipulation of the graphical representation, the user may accidentally select incorrect tools or may misremember the current tool selection. In both cases, since the selected tool rather than the nature of the action determines the effect of the action, actions with the tool lead to unexpected effects.
Although direct manipulation of controls does not suffer from this latter shortcoming (since the effects of actions such as clicking and dragging are interpreted differently in different areas of the screen according to the function of the underlying control), this type of direct manipulation also presents unique challenges. First, these controls again consume display space and require the user to divide the user's attention between the manipulation of the peripheral control and its effect on the graphical representation.
Finally, although the third interaction alternative of gesturing allows actions to be performed directly on top of the graphical representation, the graphical representation itself does not provide clues as to what gestures the underlying computer system will interpret nor how the system will interpret such gestures.