Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large Language Model-based Recommendation
<|reference_start|>Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large Language Model-based Recommendation: Large language models (LLMs) have gained much attention in the recommendation community; some studies have observed that LLMs, fine-tuned by the cross-entropy loss with a full softmax, could achieve state-of-the-art performance already. However, these claims are drawn from unobjective and unfair comparisons. In view of the substantial quantity of items in reality, conventional recommenders typically adopt a pointwise/pairwise loss function instead for training. This substitute however causes severe performance degradation, leading to under-estimation of conventional methods and over-confidence in the ranking capability of LLMs.   In this work, we theoretically justify the superiority of cross-entropy, and showcase that it can be adequately replaced by some elementary approximations with certain necessary modifications. The remarkable results across three public datasets corroborate that even in a practical sense, existing LLM-based methods are not as effective as claimed for next-item recommendation. We hope that these theoretical understandings in conjunction with the empirical results will facilitate an objective evaluation of LLM-based recommendation in the future.<|reference_end|>
@article{xu2024understanding,
    title={Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large
  Language Model-based Recommendation},
    author={Cong Xu, Zhangchi Zhu, Jun Wang, Jianyong Wang, Wei Zhang},
    journal={arXiv preprint arXiv:2402.06216},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.06216},
    primaryClass={cs.IR}
}