TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants
<|reference_start|>TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants: Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agents (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSA to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations. The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.<|reference_end|>
@article{aliannejadi2024trec,
    title={TREC iKAT 2023: A Test Collection for Evaluating Conversational and
  Interactive Knowledge Assistants},
    author={Mohammad Aliannejadi and Zahra Abbasiantaeb and Shubham Chatterjee and
  Jeffery Dalton and Leif Azzopardi},
    journal={arXiv preprint arXiv:2405.02637},
    year={2024},
    doi={10.1145/3626772.3657860},
    archivePrefix={arXiv},
    eprint={2405.02637},
    primaryClass={cs.IR cs.AI cs.CL}
}