UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches
<|reference_start|>UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches: Large language models (LLMs) have shown remarkable capabilities in generating user summaries from a long list of raw user activity data. These summaries capture essential user information such as preferences and interests, and therefore are invaluable for LLM-based personalization applications, such as explainable recommender systems. However, the development of new summarization techniques is hindered by the lack of ground-truth labels, the inherent subjectivity of user summaries, and human evaluation which is often costly and time-consuming. To address these challenges, we introduce \UserSumBench, a benchmark framework designed to facilitate iterative development of LLM-based summarization approaches. This framework offers two key components: (1) A reference-free summary quality metric. We show that this metric is effective and aligned with human preferences across three diverse datasets (MovieLens, Yelp and Amazon Review). (2) A novel robust summarization method that leverages time-hierarchical summarizer and self-critique verifier to produce high-quality summaries while eliminating hallucination. This method serves as a strong baseline for further innovation in summarization techniques.<|reference_end|>
@article{wang2024usersumbench:,
    title={UserSumBench: A Benchmark Framework for Evaluating User Summarization
  Approaches},
    author={Chao Wang, Neo Wu, Lin Ning, Jiaxing Wu, Luyang Liu, Jun Xie, Shawn
  O'Banion, Bradley Green},
    journal={arXiv preprint arXiv:2408.16966},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.16966},
    primaryClass={cs.LG cs.AI cs.CL}
}