Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors
<|reference_start|>Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors: The conversational recommendation system (CRS) has been criticized regarding its user experience in real-world scenarios, despite recent significant progress achieved in academia. Existing evaluation protocols for CRS may prioritize system-centric factors such as effectiveness and fluency in conversation while neglecting user-centric aspects. Thus, we propose a new and inclusive evaluation protocol, Concept, which integrates both system- and user-centric factors. We conceptualise three key characteristics in representing such factors and further divide them into six primary abilities. To implement Concept, we adopt a LLM-based user simulator and evaluator with scoring rubrics that are tailored for each primary ability. Our protocol, Concept, serves a dual purpose. First, it provides an overview of the pros and cons in current CRS models. Second, it pinpoints the problem of low usability in the "omnipotent" ChatGPT and offers a comprehensive reference guide for evaluating CRS, thereby setting the foundation for CRS improvement.<|reference_end|>
@article{huang2024concept,
    title={Concept -- An Evaluation Protocol on Conversational Recommender Systems
  with System-centric and User-centric Factors},
    author={Chen Huang, Peixin Qin, Yang Deng, Wenqiang Lei, Jiancheng Lv,
  Tat-Seng Chua},
    journal={arXiv preprint arXiv:2404.03304},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.03304},
    primaryClass={cs.CL cs.AI}
}