In Section 5 , we discuss the difficulties associated with such user studies , and describe a human-based evaluation we conducted for a small subset of the responses generated by our system ( Marom and Zukerman 2007b ) .
evaluation of a largescale email response system
Y Marom I Zukerman
Quality is a subjective measure , which is best judged by the users of the system ( i.e. , the help-desk customers or operators ) . In Section 5 , we discuss the difficulties associated with such user studies , and describe a human-based evaluation we conducted for a small subset of the responses generated by our system ( Marom and Zukerman 2007b ) . However , our more comprehensive evaluation is an automatic one that treats the responses generated by the help-desk operators as model responses , and performs text-based comparisons between the model responses and the automatically generated ones .
In Section 5 , we discuss the difficulties associated with such user studies , and describe a human-based evaluation we conducted for a small subset of the responses generated by our system ( @@CITATIONb ) .