Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation
<|reference_start|>Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation: Evaluating production-level retrieval systems at scale is a crucial yet challenging task due to the limited availability of a large pool of well-trained human annotators. Large Language Models (LLMs) have the potential to address this scaling issue and offer a viable alternative to humans for the bulk of annotation tasks. In this paper, we propose a framework for assessing the product search engines in a large-scale e-commerce setting, leveraging Multimodal LLMs for (i) generating tailored annotation guidelines for individual queries, and (ii) conducting the subsequent annotation task. Our method, validated through deployment on a large e-commerce platform, demonstrates comparable quality to human annotations, significantly reduces time and cost, facilitates rapid problem discovery, and provides an effective solution for production-level quality control at scale.<|reference_end|>
@article{hosseini2024retrieve,,
    title={Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for
  Large-Scale Product Retrieval Evaluation},
    author={Kasra Hosseini, Thomas Kober, Josip Krapac, Roland Vollgraf, Weiwei
  Cheng, Ana Peleteiro Ramallo},
    journal={arXiv preprint arXiv:2409.11860},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.11860},
    primaryClass={cs.IR cs.AI cs.CL cs.ET cs.HC}
}