downside is that most negatives are produced by an older version of model parameters. In Table 8,
in-batch negatives still perform favorably. Empirically, we ﬁnd that MoCo is more sensitive to certain
hyperparameters such as temperature, better results are possible with more tuning.
BM25 vs Dense Retrieval With the rapid development of dense retrieval models, can we replace
the long-standing BM25 algorithm from now on? The answer is likely “not yet”. BM25 still holds
obvious advantages in terms of simplicity, efﬁciency, and interpretability. For long-tail domains such
as Trec-Covid [55] and retrieval tasks that involve long documents (Touche-2020) [4] or rely heavily
on exact lexical match (Fever) [54], further research efforts are still necessary to improve current
dense retrievers.
E5