Multi-dimensional Evaluation of Empathetic Dialog Responses
<|reference_start|>Multi-dimensional Evaluation of Empathetic Dialog Responses: Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents -- that is, the way empathy is expressed. Yet, these works ignore the fact that conversation is also a collaboration involving both speakers and listeners. In contrast, we propose a multi-dimensional empathy evaluation framework to measure both expressed intents from the speaker's perspective and perceived empathy from the listener's perspective. We apply our proposed framework to analyze our internal customer-service dialogue. We find the two dimensions (expressed intent types and perceived empathy) are inter-connected, and perceived empathy has a high correlation with dialogue satisfaction levels.   To reduce the annotation cost, we explore different options to automatically measure conversational empathy: prompting LLMs and training language model-based classifiers. Our experiments show that prompting methods with even popular models like GPT-4 and Flan family models perform relatively poorly on both public and our internal datasets. In contrast, instruction-finetuned classifiers based on Flan-T5 family models outperform prior works and competitive baselines. We conduct a detailed ablation study to give more insights into instruction finetuning method's strong performance.<|reference_end|>
@article{xu2024multi-dimensional,
    title={Multi-dimensional Evaluation of Empathetic Dialog Responses},
    author={Zhichao Xu, Jiepu Jiang},
    journal={arXiv preprint arXiv:2402.11409},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.11409},
    primaryClass={cs.CL}
}