Evaluating Discourse Processing Algorithms
<|reference_start|>Evaluating Discourse Processing Algorithms: In order to take steps towards establishing a methodology for evaluating Natural Language systems, we conducted a case study. We attempt to evaluate two different approaches to anaphoric processing in discourse by comparing the accuracy and coverage of two published algorithms for finding the co-specifiers of pronouns in naturally occurring texts and dialogues. We present the quantitative results of hand-simulating these algorithms, but this analysis naturally gives rise to both a qualitative evaluation and recommendations for performing such evaluations in general. We illustrate the general difficulties encountered with quantitative evaluation. These are problems with: (a) allowing for underlying assumptions, (b) determining how to handle underspecifications, and (c) evaluating the contribution of false positives and error chaining.<|reference_end|>
@article{walker1994evaluating,
    title={Evaluating Discourse Processing Algorithms},
    author={Marilyn A. Walker},
    journal={Association of Computational Linguistics, 1989, p. 251-262},
    year={1994},
    archivePrefix={arXiv},
    eprint={cmp-lg/9410006},
    primaryClass={cmp-lg cs.CL}
}