Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers
Large Language Models (LLMs) have recently been applied to reranking tasks in
information retrieval, achieving strong performance. However, their high
computational demands often hinder practical deployment. Existing studies
evaluate the efficiency of LLM-based rerankers using proxy metrics such as
latency, the number of forward passes, input tokens, and output tokens.
However, these metrics depend on hardware and running-time choices (\eg
parallel or not, batch size, etc), and often fail to account for model size,
making it difficult to interpret and obscuring the evaluation of the
efficiency-effectiveness tradeoff. To address this issue, we propose
E2R-FLOPs, for LLM-based rerankers: ranking metrics per
PetaFLOP (RPP) for relevance per compute and queries per PetaFLOP (QPP) for
hardware-agnostic throughput. Companied with the new metrics, an interpretable
FLOPs estimator is built to estimate the FLOPs of an LLM-based reranker even
without running any experiments. Based on the proposed metrics, we conduct
comprehensive experiments to evaluate a wide range of LLM-based rerankers with
different architecture, studying the efficiency-effectiveness trade-off and
bringing this issue to the attention of the research community.
686dc72ccb5725779c60b356
Zhiyuan Peng
686dc72ccb5725779c60b357
Ting-ruen Wei
686dc72ccb5725779c60b358
Tingyu Song
claimed_verified
2025-07-09T08:50:31.763Z
64dc29d9b5d625e0e9a6ecb9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg
Tingyu Song
user
songtingyu
686dc72ccb5725779c60b359
Yilun Zhao
claimed_verified
2025-07-09T08:50:33.878Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
686dc72ccb5725779c60b35a
Yi Fang
Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers
Large Language Models (LLMs) have recently been applied to reranking tasks in
information retrieval, achieving strong performance. However, their high
computational demands often hinder practical deployment. Existing studies
evaluate the efficiency of LLM-based rerankers using proxy metrics such as
latency, the number of forward passes, input tokens, and output tokens.
However, these metrics depend on hardware and running-time choices (\eg
parallel or not, batch size, etc), and often fail to account for model size,
making it difficult to interpret and obscuring the evaluation of the
efficiency-effectiveness tradeoff. To address this issue, we propose
E2R-FLOPs, for LLM-based rerankers: ranking metrics per
PetaFLOP (RPP) for relevance per compute and queries per PetaFLOP (QPP) for
hardware-agnostic throughput. Companied with the new metrics, an interpretable
FLOPs estimator is built to estimate the FLOPs of an LLM-based reranker even
without running any experiments. Based on the proposed metrics, we conduct
comprehensive experiments to evaluate a wide range of LLM-based rerankers with
different architecture, studying the efficiency-effectiveness trade-off and
bringing this issue to the attention of the research community.
E\textsuperscript{2}R-FLOPs metrics, including RPP and QPP, provide a hardware-agnostic evaluation of LLM-based rerankers' efficiency and effectiveness.
Large Language Models reranking tasks information retrieval computational demands proxy metrics latency forward passes input tokens output tokens hardware choices model size efficiency-effectiveness tradeoff E\textsuperscript{2}R-FLOPs ranking metrics per PetaFLOP queries per PetaFLOP FLOPs estimator