PersonaFeedback: A Large-scale Human-annotated Benchmark For
  Personalization
With the rapid improvement in the general capabilities of LLMs, LLM
personalization, i.e., how to build LLM systems that can generate personalized
responses or services that are tailored to distinct user personas, has become
an increasingly important research and engineering problem. However, unlike
many new challenging benchmarks being released for evaluating the
general/reasoning capabilities, the lack of high-quality benchmarks for
evaluating LLM personalization greatly hinders progress in this field. To
address this, we introduce PersonaFeedback, a new benchmark that directly
evaluates LLMs' ability to provide personalized responses given pre-defined
user personas and queries. Unlike existing benchmarks that require models to
infer implicit user personas from historical interactions, PersonaFeedback
decouples persona inference from personalization, focusing on evaluating the
model's ability to generate responses tailored to explicit personas.
PersonaFeedback consists of 8298 human-annotated test cases, which are
categorized into easy, medium, and hard tiers based on the contextual
complexity of the user personas and the difficulty in distinguishing subtle
differences between two personalized responses. We conduct comprehensive
evaluations across a wide range of models. The empirical results reveal that
even state-of-the-art LLMs that can solve complex real-world reasoning tasks
could fall short on the hard tier of PersonaFeedback where even human
evaluators may find the distinctions challenging. Furthermore, we conduct an
in-depth analysis of failure modes across various types of systems,
demonstrating that the current retrieval-augmented framework should not be seen
as a de facto solution for personalization tasks. All benchmark data,
annotation protocols, and the evaluation pipeline will be publicly available to
facilitate future research on LLM personalization.
6850da255e07650ecce890d3
Meiling Tao
6850da255e07650ecce890d4
Chenghao Zhu
6850da255e07650ecce890d5
Dongyi Ding
6850da255e07650ecce890d6
Tiannan Wang
claimed_verified
2025-06-18T16:15:50.486Z
632bfaebea6e62428ab0e9c2
/avatars/344aaf371bbba9aea091b12741c451e5.svg
Tiannan Wang
user
WTNswaggy
6850da255e07650ecce890d7
Yuchen Eleanor Jiang
6850da255e07650ecce890d8
Wangchunshu Zhou
claimed_verified
2025-06-18T12:16:50.221Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
PersonaFeedback: A Large-scale Human-annotated Benchmark For
  Personalization
With the rapid improvement in the general capabilities of LLMs, LLM
personalization, i.e., how to build LLM systems that can generate personalized
responses or services that are tailored to distinct user personas, has become
an increasingly important research and engineering problem. However, unlike
many new challenging benchmarks being released for evaluating the
general/reasoning capabilities, the lack of high-quality benchmarks for
evaluating LLM personalization greatly hinders progress in this field. To
address this, we introduce PersonaFeedback, a new benchmark that directly
evaluates LLMs' ability to provide personalized responses given pre-defined
user personas and queries. Unlike existing benchmarks that require models to
infer implicit user personas from historical interactions, PersonaFeedback
decouples persona inference from personalization, focusing on evaluating the
model's ability to generate responses tailored to explicit personas.
PersonaFeedback consists of 8298 human-annotated test cases, which are
categorized into easy, medium, and hard tiers based on the contextual
complexity of the user personas and the difficulty in distinguishing subtle
differences between two personalized responses. We conduct comprehensive
evaluations across a wide range of models. The empirical results reveal that
even state-of-the-art LLMs that can solve complex real-world reasoning tasks
could fall short on the hard tier of PersonaFeedback where even human
evaluators may find the distinctions challenging. Furthermore, we conduct an
in-depth analysis of failure modes across various types of systems,
demonstrating that the current retrieval-augmented framework should not be seen
as a de facto solution for personalization tasks. All benchmark data,
annotation protocols, and the evaluation pipeline will be publicly available to
facilitate future research on LLM personalization.
A new benchmark, PersonaFeedback, evaluates Large Language Models' ability to generate personalized responses given explicit user personas, revealing limitations in current systems.
Large Language Models (LLMs) LLM personalization PersonaFeedback user personas personalized responses contextual complexity human-annotated test cases retrieval-augmented framework