Are Large Language Models In-Context Personalized Summarizers? Get an iCOPERNICUS Test Done!
<|reference_start|>Are Large Language Models In-Context Personalized Summarizers? Get an iCOPERNICUS Test Done!: Large Language Models (LLMs) have succeeded considerably in In-Context-Learning (ICL) based summarization. However, saliency is subject to the users' specific preference histories. Hence, we need reliable In-Context Personalization Learning (ICPL) capabilities within such LLMs. For any arbitrary LLM to exhibit ICPL, it needs to have the ability to discern contrast in user profiles. A recent study proposed a measure for degree-of-personalization called EGISES for the first time. EGISES measures a model's responsiveness to user profile differences. However, it cannot test if a model utilizes all three types of cues provided in ICPL prompts: (i) example summaries, (ii) user's reading histories, and (iii) contrast in user profiles. To address this, we propose the iCOPERNICUS framework, a novel In-COntext PERsonalization learNIng sCrUtiny of Summarization capability in LLMs that uses EGISES as a comparative measure. As a case-study, we evaluate 17 state-of-the-art LLMs based on their reported ICL performances and observe that 15 models' ICPL degrades (min: 1.6%; max: 3.6%) when probed with richer prompts, thereby showing lack of true ICPL.<|reference_end|>
@article{patel2024are,
    title={Are Large Language Models In-Context Personalized Summarizers? Get an
  iCOPERNICUS Test Done!},
    author={Divya Patel, Pathik Patel, Ankush Chander, Sourish Dasgupta, Tanmoy
  Chakraborty},
    journal={arXiv preprint arXiv:2410.00149},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00149},
    primaryClass={cs.CL cs.LG cs.NE}
}