Automatic summarization of reader comments in on-line news is an extremely challenging task and a capability for which there is a#R##N#clear need. Work to date has focussed on producing extractive summaries using well-known techniques imported from other areas of#R##N#language processing. But are extractive summaries of comments what users really want? Do they support users in performing the sorts#R##N#of tasks they are likely to want to perform with reader comments? In this paper we address these questions by doing three things. First,#R##N#we offer a specification of one possible summary type for reader comment, based on an analysis of reader comment in terms of issues#R##N#and viewpoints. Second, we define a task-based evaluation framework for reader comment summarization that allows summarization#R##N#systems to be assessed in terms of how well they support users in a time-limited task of identifying issues and characterising opinion on#R##N#issues in comments. Third, we describe a pilot evaluation in which we used the task-based evaluation framework to evaluate a prototype#R##N#reader comment clustering and summarization system, demonstrating the viability of the evaluation framework and illustrating the sorts#R##N#of insight such an evaluation affords.
['Emma Barker', 'Monica Lestari Paramita', 'Adam Funk', 'Emina Kurtic', 'Ahmet Aker', 'Jonathan Foster', 'Mark Hepple', 'Robert J. Gaizauskas']
What's the issue here?: Task-based evaluation of reader comment summarization systems