1. The Field of the Invention
The present invention relates to systems and methods for generating an annotation guide. More particularly, the present invention relates to systems and methods for interactively generating an annotation guide from speech data.
2. Introduction
Dialog applications are applications that are often used to automate the process of receiving and responding to customer inquiries. Dialog applications use a combination of voice recognition modules, language understanding modules, and text-to-speech systems to appropriately respond to speech input received from a user or a customer. Billing inquiries, information queries, customer complaints, and general questions are examples of the speech input that is received by dialog applications. The response of the dialog application to a particular speech input depends on the logic of the dialog application.
The development of a successful dialog application, however, is a time-consuming process and requires a significant amount of manual labor because of the nature of the tasks being performed. One of the tasks performed in the development of a dialog application is the generation of an annotation guide that is used to annotate or label raw speech data. The annotation guide is generally created by a user experience person (or other user) that is familiar with the purposes and goals of the dialog application. Becoming familiar with the purposes and goals of the dialog application is also a labor-intensive process.
Currently, the generation of an annotation guide requires the user experience person to examine the raw speech data and create the categories, call types, and examples that are usually included in the annotation guide. The annotation guide aids the development of a dialog application because the annotation guide is used by labelers to classify the raw speech data with the call types defined by the user experience person in the annotation guide. The annotation guide helps insure that different labelers classify similar utterances with the same call type.
The ability to properly classify the raw speech data ultimately has a significant impact on whether the dialog application can respond to speech input appropriately. An annotation guide that does not sufficiently define the call types introduces potential errors into the dialog application because the speech data may be labeled incorrectly by the labelers. If incorrectly labeled or annotated speech data is used to train portions of the dialog application such as the natural language understanding modules, the dialog application will clearly not function properly and will frustrate customers. Thus, the development of a quality annotation guide is important to the development of a quality dialog application.
A labeler uses the annotation guide to determine whether a particular utterance should be labeled with a particular call type. For example, an annotation guide may describe a Pay_Bill call type used to label utterances that suggest the customer wants to pay his or her bill. The following utterances from raw speech data, for example, should be labeled with the Pay_Bill call type:                I want to pay a bill; and        I got mail and I have my credit card ready.        
A poorly developed annotation guide may not provide sufficient guidance to a labeler to properly annotate the second example of “I got mail and I have my credit card ready.” Without the guidance of the annotation guide, different labelers may assign different call types to this example. For these reasons, it is desirable to have a well developed and thought out annotation guide to assist in the annotation of the speech data.
As previously mentioned, however, the user experience person does not currently have any way to develop the annotation guide other than by directly examining the raw speech data. Not surprisingly, the development of the annotation guide can be a lengthy process. In other words, the development of an annotation guide is a slow and tedious process and the time required to develop the annotation guide introduces additional cost into the dialog application. Systems and methods are needed to reduce the time required to develop an annotation guide.