CROSS-JEM: Accurate and Efficient Cross-encoders for Short-text Ranking Tasks
<|reference_start|>CROSS-JEM: Accurate and Efficient Cross-encoders for Short-text Ranking Tasks: Ranking a set of items based on their relevance to a given query is a core problem in search and recommendation. Transformer-based ranking models are the state-of-the-art approaches for such tasks, but they score each query-item independently, ignoring the joint context of other relevant items. This leads to sub-optimal ranking accuracy and high computational costs. In response, we propose Cross-encoders with Joint Efficient Modeling (CROSS-JEM), a novel ranking approach that enables transformer-based models to jointly score multiple items for a query, maximizing parameter utilization. CROSS-JEM leverages (a) redundancies and token overlaps to jointly score multiple items, that are typically short-text phrases arising in search and recommendations, and (b) a novel training objective that models ranking probabilities. CROSS-JEM achieves state-of-the-art accuracy and over 4x lower ranking latency over standard cross-encoders. Our contributions are threefold: (i) we highlight the gap between the ranking application's need for scoring thousands of items per query and the limited capabilities of current cross-encoders; (ii) we introduce CROSS-JEM for joint efficient scoring of multiple items per query; and (iii) we demonstrate state-of-the-art accuracy on standard public datasets and a proprietary dataset. CROSS-JEM opens up new directions for designing tailored early-attention-based ranking models that incorporate strict production constraints such as item multiplicity and latency.<|reference_end|>
@article{paliwal2024cross-jem:,
    title={CROSS-JEM: Accurate and Efficient Cross-encoders for Short-text Ranking
  Tasks},
    author={Bhawna Paliwal, Deepak Saini, Mudit Dhawan, Siddarth Asokan, Nagarajan
  Natarajan, Surbhi Aggarwal, Pankaj Malhotra, Jian Jiao, Manik Varma},
    journal={arXiv preprint arXiv:2409.09795},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09795},
    primaryClass={cs.IR}
}