Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs
<|reference_start|>Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs: Scalability issue plays a crucial role in productionizing modern recommender systems. Even lightweight architectures may suffer from high computational overload due to intermediate calculations, limiting their practicality in real-world applications. Specifically, applying full Cross-Entropy (CE) loss often yields state-of-the-art performance in terms of recommendations quality. Still, it suffers from excessive GPU memory utilization when dealing with large item catalogs. This paper introduces a novel Scalable Cross-Entropy (SCE) loss function in the sequential learning setup. It approximates the CE loss for datasets with large-size catalogs, enhancing both time efficiency and memory usage without compromising recommendations quality. Unlike traditional negative sampling methods, our approach utilizes a selective GPU-efficient computation strategy, focusing on the most informative elements of the catalog, particularly those most likely to be false positives. This is achieved by approximating the softmax distribution over a subset of the model outputs through the maximum inner product search. Experimental results on multiple datasets demonstrate the effectiveness of SCE in reducing peak memory usage by a factor of up to 100 compared to the alternatives, retaining or even exceeding their metrics values. The proposed approach also opens new perspectives for large-scale developments in different domains, such as large language models.<|reference_end|>
@article{mezentsev2024scalable,
    title={Scalable Cross-Entropy Loss for Sequential Recommendations with Large
  Item Catalogs},
    author={Gleb Mezentsev, Danil Gusak, Ivan Oseledets, Evgeny Frolov},
    journal={arXiv preprint arXiv:2409.18721},
    year={2024},
    doi={10.1145/3640457.3688140},
    archivePrefix={arXiv},
    eprint={2409.18721},
    primaryClass={cs.IR cs.LG}
}