summaries: This paper investigate the gap problem between cross-attention (CA) and dual-encoder (DE) models for document reranking. In general, cross-attention models perform much better than dual-encoder models while consuming more computational cost. The authors first leverage Mercer's theorem to prove that dual-encoders models are sufficiently expressive to model a broad class of scores using countably infinite dimension, which mainly the gap between CA and DE models should not exist from theoretical analysis. They further identified the gap is caused by the generalization ability of the DE models. To reduce the overfitting on the training data, they proposed to use distillation algorithms. Specially, they design the loss function which matches the margins between the teacher (CA) and student (DE) models. They also extend the margin to probability matching by using softmax cross-entropy loss. Through the experiments, they demonstrated that their proposed distillation approaches can further reduce the gap between DE and CA models.  The main contributions of this paper is that they show there should not be a gap between DE and CA models from theoretical analysis. They found that the gap is caused by the generalization ability of CA model and proposed distillation approach to reduce the gap.

summaries: In information retrieval problems, there are two kinds of models: dual-encoder models and cross-attention models. Cross-attention models generally outperform dual encoder models by a large margin. Due to the latency requirement of real search systems, cross-attention models might not meet the requirements. So one ideal solution is to transfer the knowledge from cross-attention models to dual encoder models. This paper follows this line of research ideas by reducing the gap between the two models.   Firstly, the paper theoretically analyzes the effects with sufficiently large embedding dimensions. Secondly, the paper empirically verifies the theoretical results. Thirdly, the paper proposes a new method that further bridges the gap between the two methods. The writing is clear and well-structured. I like the idea and logic of the paper. I have some questions about the experiment part.

summaries: This paper aims to narrow the performance gap between cross-attention BERT and dual-encoder BERT for re-ranking task. The authors empirically and theoretically analyze the underlying reasons of the performance gap. The gap could be mitigated by the proposed knowledge distillation method, where the original cross-attention BERT model acts as the “teacher” and the more efficient dual-encoder BERT model is used as student. Comprehensive experiments confirmed the effectiveness of the proposed KD method for various re-ranking tasks.

summaries: The paper contrasts cross-encoder vs. bi-encoder architectures for re-ranking in information retrieval.  The authors argue that bi-encoders under-perform cross-encoders, not for lack of capacity, but due to poor generalization, and back this claim with some theory and empirical results.  The authors then propose distillation of the cross-encoder model into a bi-encoder, which is shown to improve bi-encoder results.  While this has been proposed before, the distillation loss function the authors propose is slightly different than the standard cross-entropy (which the authors show is closely related).

abstractText: Transformer-based models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query. There are two broad flavours of such models: cross-attention (CA) models, which learn a joint embedding for the query and document, and dual-encoder (DE) models, which learn separate embeddings for the query and document. Empirically, CA models are often more accurate, which has motivated a series of works seeking to bridge this gap. However, a fundamental question remains less explored: does this performance gap reflect a limitation in the capacity of DE models, or in the training of such models? In this paper, we study this question, with three contributions. First, we establish theoretically that with a sufficiently large encoder size, DE models can capture a broad class of scores without cross-attention. Second, we show empirically that on real-world problems, the gap between CA and DE models may be due to the latter overfitting to the training set. To mitigate this behaviour, we propose a distillation strategy that focuses on preserving the ordering amongst documents, and confirm its efficacy on benchmark neural re-ranking datasets.

authors:
Jimmy Ba
Rich Caruana

title: Do deep nets really need to be deep

venue: Advances in Neural Information Processing Systems,

authors:
Jane Bromley
Isabelle Guyon
Yann LeCun
Eduard Säckinger
Roopak Shah

title: Signature verification using a “Siamese” time delay neural network

venue: In Proceedings of the 6th International Conference on Neural Information Processing Systems,

authors:
Wei-Cheng Chang
Felix X. Yu
Yin-Wen Chang
Yiming Yang
Sanjiv Kumar

title: Pre-training tasks for embedding-based large-scale retrieval

authors:
J.H. Cho
B. Hariharan

title: On the efficacy of knowledge distillation

venue: IEEE/CVF International Conference on Computer Vision (ICCV),

authors:
Nick Craswell
Bhaskar Mitra
Emine Yilmaz
Daniel Campos

title: Overview of the TREC 2020 deep learning track

venue: Proceedings of the Twenty-Ninth Text REtrieval Conference,

authors:
Zhuyun Dai
Jamie Callan

title: Deeper text understanding for IR with contextual neural language modeling

venue: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: pre-training of deep bidirectional transformers for language understanding

venue: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,

authors:
Tommaso Furlanello
Zachary Chase Lipton
Michael Tschannen
Laurent Itti
andAnimaAnandkumar

title: Born-again neural networks

venue: In Proceedings of the 35th International Conference on Machine Learning,

authors:
Luyu Gao
Zhuyun Dai
Jamie Callan

title: Understanding BERT rankers under distillation

venue: In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval,

authors:
Daniel Gillick
Sayali Kulkarni
Larry Lansing
Alessandro Presta
Jason Baldridge
Eugene Ie
Diego Garcia-Olano

title: Learning dense representations for entity retrieval, 2019

authors:
Ruiqi Guo
Philip Sun
Erik Lindgren
Quan Geng
David Simcha
Felix Chern
Sanjiv Kumar

title: Accelerating large-scale inference with anisotropic vector quantization

venue: In Proceedings of the 37th International Conference on Machine Learning,

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the knowledge in a neural network

authors:
Sebastian Hofstätter
Sophia Althammer
Michael Schröder
Mete Sertkan
Allan Hanbury

title: Improving efficient neural ranking models with cross-architecture knowledge distillation

authors:
Sebastian Hofstätter
Markus Zlabinger
Allan Hanbury

title: Interpretable & time-budget-constrained contextualization for re-ranking

venue: ECAI 2020 - 24th European Conference on Artificial Intelligence,

authors:
Sebastian Hofstätter
Bhaskar Mitra
Hamed Zamani
Nick Craswell
Allan Hanbury

title: IntraDocument Cascading: Learning to Select Passages for Neural Document Ranking

venue: In Proc. of SIGIR,

authors:
Gautier Izacard
Edouard Grave

title: Distilling knowledge from reader to retriever for question answering, 2020

authors:
Xiaoqi Jiao
Yichun Yin
Lifeng Shang
Xin Jiang
Xiao Chen
Linlin Li
Fang Wang
Qun Liu

title: TinyBERT: Distilling BERT for natural language understanding

venue: In Findings of the Association for Computational Linguistics: EMNLP

authors:
Jeff Johnson
Matthijs Douze
Hervé Jégou

title: Billion-scale similarity search with gpus

venue: IEEE Transactions on Big Data,

authors:
Vladimir Karpukhin
Barlas Oguz
SewonMin
Patrick Lewis
Ledell Wu
Sergey Edunov
Danqi Chen
Wen-tau Yih

title: Dense passage retrieval for open-domain question answering

venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781,

title: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, pp. 39–48

venue: Association for Computing Machinery,

authors:
Tom Kwiatkowski
Jennimaria Palomaki
Olivia Redfield
Michael Collins
Ankur Parikh
Chris Alberti
Danielle Epstein
Illia Polosukhin
Matthew Kelcey
Jacob Devlin
Kenton Lee
Kristina N. Toutanova
Llion Jones
Ming-Wei Chang
Andrew Dai
Jakob Uszkoreit
Quoc Le
Slav Petrov

title: Natural questions: a benchmark for question answering research

venue: Transactions of the Association of Computational Linguistics,

authors:
Kenton Lee
Ming-Wei Chang
Kristina Toutanova

title: Latent retrieval for weakly supervised open domain question answering

venue: Proceedings of the 57th Conference of the Association for Computational Linguistics,

authors:
Tsung-Yi Lin
Priya Goyal
Ross B. Girshick
Kaiming He
Piotr Dollár

title: Focal loss for dense object detection

venue: In IEEE International Conference on Computer Vision, ICCV 2017,

authors:
Wenhao Lu
Jian Jiao
Ruofei Zhang

title: Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval

venue: In Proceedings of the 29th ACM International Conference on Information & Knowledge Management,

authors:
Yi Luan
Jacob Eisenstein
Kristina Toutanova
Michael Collins

title: Sparse, Dense, and Attentional Representations for Text Retrieval

venue: Transactions of the Association for Computational Linguistics, 9:329–345,

authors:
Michal Lukasik
Srinadh Bhojanapalli
Aditya Krishna Menon
Sanjiv Kumar

title: Does label smoothing mitigate label noise

authors:
Ulrike von Luxburg
Olivier Bousquet

title: Distance–based classification with lipschitz functions

venue: J. Mach. Learn. Res.,

authors:
Xueguang Ma
Kai Sun
Ronak Pradeep
Jimmy Lin

title: A replication study of dense passage retriever

authors:
Sean MacAvaney
Andrew Yates
Arman Cohan
Nazli Goharian

title: Cedr: Contextualized embeddings for document ranking

venue: In SIGIR,

authors:
Sean MacAvaney
Franco Maria Nardini
Raffaele Perego
Nicola Tonellotto
Nazli Goharian
Ophir Frieder

title: Efficient Document Re-Ranking for Transformers by Precomputing Term Representations, pp. 49–58

authors:
Irina Matveeva
Chris Burges
Timo Burkard
Andy Laucius
Leon Wong

title: High accuracy retrieval with multiple nested ranker

venue: In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,

authors:
Charles A. Micchelli
Yuesheng Xu
Haizhang Zhang

title: Universal kernels

venue: Journal of Machine Learning Research,

authors:
Antoine Miech
Jean-Baptiste Alayrac
Ivan Laptev
Josef Sivic
Andrew Zisserman

title: Thinking fast and slow: Efficient text-to-visual retrieval with transformers

venue: In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021,

authors:
Bhaskar Mitra
Nick Craswell

title: An introduction to neural information retrieval

venue: Foundations and Trends® in Information Retrieval,

authors:
Rafael Müller
Simon Kornblith
Geoffrey E. Hinton

title: When does label smoothing help? In Advances in Neural Information Processing Systems

venue: Annual Conference on Neural Information Processing Systems

authors:
Tri Nguyen
Mir Rosenberg
Xia Song
Jianfeng Gao
Saurabh Tiwary
RanganMajumder
Li Deng

title: MS MARCO: A human generated machine reading comprehension dataset

venue: Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016,

authors:
Rodrigo Nogueira
Kyunghyun Cho

title: Passage re-ranking with BERT

authors:
Yingqi Qu
Yuchen Ding
Jing Liu
Kai Liu
Ruiyang Ren
Wayne Xin Zhao
Daxiang Dong
Hua Wu
Haifeng Wang

title: RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering

venue: Proceedings of the 2021 Conference of the North American Chapter of the Association

authors:
Dragomir R. Radev
Hong Qi
Harris Wu
Weiguo Fan

title: Evaluating web-based question answering systems

venue: In Proceedings of the Third International Conference on Language Resources and Evaluation

authors:
Sashank Reddi
Rama Kumar Pasumarthi
Aditya Menon
Ankit Singh Rawat
Felix Yu
Seungyeon Kim
Andreas Veit
Sanjiv Kumar

title: RankDistil: Knowledge distillation for ranking

venue: Proceedings of The 24th International Conference on Artificial Intelligence and Statistics,

authors:
Nils Reimers
Iryna Gurevych

title: Sentence-bert: Sentence embeddings using Siamese BERTnetworks

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics,

authors:
Stephen Robertson
Hugo Zaragoza

title: The probabilistic relevance framework: Bm25 and beyond

venue: Found. Trends Inf. Retr.,

authors:
Victor Sanh
Lysandre Debut
Julien Chaumond
Thomas Wolf

title: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

venue: In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing

authors:
Bharath K. Sriperumbudur
Kenji Fukumizu
Gert R.G. Lanckriet

title: Universality, characteristic kernels and RKHS embedding of measures

authors:
Ingo Steinwart
Andreas Christmann

title: Support Vector Machines. Information Science and Statistics

authors:
Siqi Sun
Yu Cheng
Zhe Gan
Jingjing Liu

title: Patient knowledge distillation for BERT model compression

venue: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,

authors:
Zhiqing Sun
Hongkun Yu
Xiaodan Song
Renjie Liu
Yiming Yang
Denny Zhou

title: MobileBERT: a compact task-agnostic BERT for resource-limited devices

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,

authors:
Christian Szegedy
Vincent Vanhoucke
Sergey Ioffe
Jonathon Shlens
Zbigniew Wojna

title: Rethinking the inception architecture for computer vision

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Iulia Turc
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: Well-read students learn better: The impact of student initialization on knowledge distillation

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N. Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,

authors:
Wenhui Wang
Furu Wei
Li Dong
Hangbo Bao
Nan Yang
Ming Zhou

title: Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers

authors:
Lee Xiong
Chenyan Xiong
Ye Li
Kwok-Fung Tang
Jialin Liu
Paul N. Bennett
Junaid Ahmed
Arnold Overwijk

title: Approximate nearest neighbor negative contrastive learning for dense text retrieval

venue: In International Conference on Learning Representations,

authors:
Canwen Xu
Wangchunshu Zhou
Tao Ge
Furu Wei
Ming Zhou

title: BERT-of-theseus: Compressing BERT by progressive module replacing

authors:
Sohee Yang
Minjoon Seo

title: Is retriever merely an approximator of reader

authors:
Zeynep Akkalyoncu Yilmaz
Wei Yang
Haotian Zhang
Jimmy Lin

title: Cross-domain modeling of sentence-level evidence for document retrieval

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

authors:
Chulhee Yun
Srinadh Bhojanapalli
Ankit Singh Rawat
Sashank Reddi
Sanjiv Kumar

title: Are transformers universal approximators of sequence-to-sequence functions

authors:
Jingtao Zhan
Jiaxin Mao
Yiqun Liu
Min Zhang
Shaoping Ma

title: RepBERT: Contextualized text embeddings for first-stage retrieval

authors:
Jingtao Zhan
Jiaxin Mao
Yiqun Liu
Jiafeng Guo
Min Zhang
Shaoping Ma

title: Optimizing dense retrieval model training with hard negatives

title: Jointly optimizing query encoder and product quantization to improve retrieval performance

venue: In CIKM,

venue: Following Karpukhin et al

title: Billy Gray was born on January 13, 1938 in Los Angeles, California, USA as Will

heading: 1 Introduction: transformer-based neural ranking

text:
Information retrieval (Mitra &Craswell, 2018) is the classic problem of identifying relevant documents for a given query. Typically, such retrieval is performed in a two-step manner (Matveeva et al., 2006): one uses an initial model to efficiently retrieve a candidate set of documents for a query, and then uses a second model to re-rank these candidates. The retrieval stage is generally implemented by a model with low inference cost, for which candidate document generation is tractable; if this candidate set is reasonably small, it is feasible to use more complex models in the re-ranking stage.
Transformer-based models such as BERT (Devlin et al., 2019) have proven successful in both the retrieval and re-ranking stages. Such neural ranking models have two flavours: dual-encoder (DE) models (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Xiong et al., 2021; Luan et al., 2021), which learn separate (factorised) embeddings for the query and document; and cross-attention (CA) models (Nogueira & Cho, 2019; Dai & Callan, 2019; Yilmaz et al., 2019; MacAvaney et al., 2019; Gao et al., 2020), which learn a joint embedding for the query and document. Only DE models are applicable for retrieval, as they admit efficient nearest neighbour search (Guo et al., 2020; Johnson et al., 2021) to identify candidate documents. Both CA and DE models are applicable for re-ranking; however, empirically, CA models have proven more accurate (Hofstätter et al., 2020a).
Does the re-ranking performance gap between CA and DE models reflect a limitation in the inherent capacity of the latter’s factorised representation, or in the training of DE models? These questions are of conceptual interest, but are also practically relevant: DE models allow for more efficient inference owing to the ability to pre-compute document embeddings, which enables fast similarity search when presented with a candidate query (Khattab & Zaharia, 2020; Hofstätter et al., 2020b). Several works have explored means of improving DE models, including by changing1 the scoring layer (Khattab & Zaharia, 2020; MacAvaney et al., 2020; Hofstätter et al., 2020b), and by distilling predictions from a CA model (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021). However, the root cause of the gap between CA and DE models remains elusive.
In this paper, we study these questions, with the following contributions:
1Unless otherwise stated, we use “DE” to mean a dual-encoder with dot-product scoring per Equation 2.
(i) we establish theoretically that with a sufficiently large embedding dimension (and under mild assumptions), any continuous ground-truth scores can be modelled by generic DE models (Proposition 1), and in particular by sufficiently deep transformer-based DEmodels (Proposition 2). Thus, in principle, there is no fundamental restriction in using DE versus CAmodels for re-ranking.
(ii) we show empirically (Figure 1) that on real-world problems, CA and DE models can achieve similar training performance, but DE models may do worse on test data due to overfitting (§3.2). Thus, the DE models may suffer due to poorer generalisation ability rather than capacity.
(iii) to mitigate the above, we propose a distillation strategy focussing on mimicking the teacher’s ordering amongst documents (§4). This generalises the recent margin MSE loss (Hofstätter et al., 2020a) (§4.1), and justifies the utility of softmax cross-entropy based distillation for re-ranking (Proposition 3, §4.2). We confirm the efficacy of this strategy on the re-reranking benchmarks MSMARCO-Passage (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019).
Overall, our results aim to shed light on the fundamental differences between CA and DE models, and give a simple yet effective distillation strategy to boost the performance of the latter.

heading: 2 Background and notation

text:
Retrieval problems. Suppose we have query space Q and document space D. For a given query q ∈ Q, the goal of information retrieval is to identify a set of relevant documents Drel(q) ⊆ D (Mitra & Craswell, 2018). This is typically achieved by learning a scorer s : Q × D → R that predicts the relevance of a query and document, and simply selecting the top-k highest scoring documents for a given query. Typically, this scorer is itself implemented in two phases (Matveeva et al., 2006; Nogueira & Cho, 2019; Chang et al., 2020; Miech et al., 2021; Hofstätter et al., 2021). In the retrieval phase, one identifies an initial candidate set of documents via an initial scorer sret. In the re-ranking phase, one re-scores only these candidate documents via a distinct scorer srrk to obtain Drel(q). We may thus regard s : Q×D→ R as a suitable composition of both sret and srrk. To learn a scorer, suppose we have supervision {(qn,dn,yn)}Nn=1, where for each query qn we have a list of K associated documents dn . = (dn1, . . . , dnK) ∈ DK with ground-truth relevance labels yn . = (yn1, . . . , ynK) ∈ {0, 1}K . Typically, dn comprises a small subset of D (i.e., K |D|), since it is rarely feasible to collect relevance labels for all documents. Consider a parameterised class of scorers S .= {s(·, ·; θ) : Q × D → R | θ ∈ Θ}. For any given θ ∈ Θ, let sn(θ) . = (s(qn, dn1; θ), . . . , s(qn, dnK ; θ)) ∈ RK denote the vector of model scores on the provided
documents. One may then learn a scorer s via minimising R(θ) .= 1N ∑ n∈[N ] `(yn, sn(θ)) for loss
` : RK × RK → R+, such as the mean square error `mse(yn, sn(θ)) . = ∑ k∈[K](ynk − snk(θ))2;
and the softmax cross-entropy (assuming for simplicity 1>y = 1): for temperature τ > 0,
`ce(yn, sn(θ)) . = − ∑ k∈[K] ynk · log pnk(θ) where pnk(θ) . = exp(τ−1 · snk(θ))∑ k′∈[K] exp(τ −1 · snk′(θ)) . (1)
To instantiate either objective, one missing ingredient is the precise parameterisation of S. We now describe one possible parameterisation, given by a transformer model.
Cross-attention and dual-encoders. Transformers (Vaswani et al., 2017) are sequence-to-sequence models with good empirical performance on language tasks. Suppose we have a sequence (x1, . . . , xL) of L tokens (e.g., words in a language), with X .= (x1,x2, . . . ,xL) ∈ RD×L being a corresponding sequence of token embeddings (e.g., word embeddings). A transformer encoder maps this sequence to Z .= (z1, z2, . . . , zL) ∈ RD×L , via a composition of attention and feedforward layers. The former is a mapping attn : RD×L → RD×L, which in its simplest (“single head”) form is attn(X) = X + WVXA, where A . = σ ( X>WQWKX/ √ D ) for softmax σ(·) and weights WQ,WK,WV.
Intuitively, this embeds a token via a weighted sum of all tokens’ embeddings. The weighting is given by the attention matrix A ∈ RL×L, which is a learned similarity measure between pairs of tokens. BERT (Devlin et al., 2019) is a canonical instantiation of the transform encoder, coupled with effective pre-training tasks shown to improve downstream performance in small-sample regimes.
Transformers can readily score (query, document) relevance: given a pair (q, d) ∈ Q×D of tokenised queries and documents, one may pass their concatenation through a transformer encoder T . The resulting embedding may be pooled (e.g., using the [CLS] token (Devlin et al., 2019)) to yield
s(q, d) = w>pool(T (concat(q, d)))
for weights w ∈ RD. Such cross-attention (CA) models apply attention layers on the queries and documents jointly, which intuitively allow for rich interactions.
A distinct strategy is to separately embed queries and documents (e.g., via BERT encoders), and score them with their dot-product (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Zhan et al., 2020; Ma et al., 2021; Xiong et al., 2021; Luan et al., 2021; Qu et al., 2021; Zhan et al., 2021a):
s(q, d) = pool(T (q))>pool(T (d)). (2)
Such dual-encoder (DE) models have their roots in Siamese networks for image processing (Bromley et al., 1993), and have received recent interest in a range of language problems (Reimers & Gurevych, 2019; Gillick et al., 2019; Karpukhin et al., 2020). A salient feature of Equation 2 is that one can index the query and document embeddings. This implies that DE models can have significantly lower the inference cost compared to CA models, which makes them appealing to use for both the retrieval and re-ranking stages. Indeed, DE models are widely used for retrieval, owing to their amenability to fast approximate nearest neighbour search (Guo et al., 2020; Johnson et al., 2021). Unfortunately, for re-ranking, one typically observes a large gap between DE and CA performance (see §3.2).
Cross-attention to dual-encoder distillation. One strategy to bridge the performance gap between CA and DE models is distillation (Ba & Caruana, 2014; Hinton et al., 2015). Suppose we have supervision {(qn,dn, tn,yn)}Nn=1, where tn ∈ RK are the scores from a “teacher” (e.g., CA) model. Then, we may minimise Rdist(θ) . = 1N ∑ n∈[N ] `(tn, sn(θ)), noting that we use the teacher supervision tn rather than “one-hot” labels yn; one may also combine the teacher and one-hot labels. Two canonical instantiations are logit matching (Ba & Caruana, 2014), which uses the the mean square error `mse(tn, sn(θ)) between the teacher and student scores, and probability matching (Hinton et al., 2015), which uses the softmax cross-entropy between the teacher and student softmax probabilities, i.e., `(tn, sn) = `ce(p teach n , sn) where pteachnk . = exp(τ −1·tnk)∑ k′∈[K] exp(τ −1·tnk′ ) for temperature τ > 0. Empirically, distillation has been shown to at least partially bridge the chasm between CA and DE performance (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021).

heading: 3 The capacity of dual-encoder models

text: We show that while DE models have sufficient capacity to represent complex score functions, in practice they may overfit and thus perform poorly on test data.

heading: 3.1 How expressive are dual-encoders in theory?

text:
To study the expressivity of DE models, we ask: how well can such models approximate a given score function s∗ : Q×D→ R? This concerns the theoretical capacity of representing s∗, and side-steps the issue of estimating it from samples; we discuss the latter subsequently. For our purposes, the DE family comprises any model of the form s(q, d) = z(q)>w(d), where z : Q→ RD andw : D→ RD are (arbitrarily powerful) query- and document embedding functions. Under mild assumptions on D, we may show that DE models with sufficiently large embeddings can model a broad class of s∗. Proposition 1. Suppose D is a compact metric space, and s∗(q, ·) : D→ R is continuous for each q ∈ Q. Then, for any > 0, ∃ vectors zq,wd of at most countably infinite dimension such that
(∀q ∈ Q, d ∈ D) |s∗(q, d)− z>q wd| ≤ .
The proof relies on the classic Mercer’s theorem for kernel methods, and is provided in the Appendix. We make a few remarks on the above. First, note that by symmetry, we could equally swap the roles of Q and D above. Second, the assumption of D being a compact metric space is mild: we may do this, e.g., by identifying each document with a normalised Euclidean embedding. Third, while the above employs a kernel-based argument, we do not require that s∗ is symmetric; rather, s∗(q, ·) is simply a function to be approximated by an element in a reproducing kernel Hilbert space.
Fourth, one can gain intuition by assuming Q and D are finite spaces. Here, we may associate with any s∗ : Q ×D → R a matrix S∗ ∈ R|Q|×|D|, where S∗q,d . = s∗(q, d). By the singular value decomposition, S∗ = ZW> for matricesZ ∈ R|Q|×r,W ∈ R|D|×r, where r .= rank(S∗). Thus, we may always write s∗(q, d) = S∗q,d = z(q)>w(d) by simply defining z(q) = Zq: and w(d) = W:d.
Finally, the above does not restrict the function class used to represent either z orw. Restricting these to transformers, we may appeal to their universal approximation power (Yun et al., 2020): a sufficiently deep transformer encoder can approximate any continuous mapping T : RD×L → RD×L to arbitrary precision. Now suppose we represent queries Q ⊂ RD×L as embeddings of sequences of L tokenised elements, with a fixed [CLS] token at the first position. Then, any continuous z : Q→ RD can be approximated by reading the first token embedding of a suitable transformer T . Proposition 2. Fix some compact Q ⊂ RD×L. For any continuous z : Q → RD and ′ > 0, ∃ a transformer encoder T : RD×L → RD×L such that ‖z(Q)−T (Q):1‖22 ≤ ′ for almost everyQ ∈ Q.
Remark. Luan et al. (2021) similarly analysed the expressive power of CA versus DE models, but through a slightly different lens: they formalised the expressiveness of DE score functions based on random projections, showing that the resulting embedding size may need to scale with the document length. This elegant analysis sheds light on one possible failure mode of DE models, i.e., scoring the relevance of queries and overly long documents. By contrast, we do not specifically restrict the form of the score function, which can be more reflective of transformer-based DE models; further, we shall now identify a distinct, fundamental reason for the gap between DE and CA model performance.

heading: 3.2 How expressive are dual-encoders in practice?

text:
The previous section suggests that, at least in theory, DE models ought to be sufficiently expressive as to model a broad class of scores. However, this is at odds with the wealth of empirical results suggesting a non-trivial performance gap between DE and CA models (Lu et al., 2020; Izacard & Grave, 2020; Hofstätter et al., 2020a; Miech et al., 2021). What is the cause of this discrepancy? Note that the previous section considers only the capacity of DE models, i.e., the theoretical existence of a DE model capable of representing a given score function. A distinct question is whether it is feasible to learn such a model from a finite number of samples, and with a capacity restriction on the function class (e.g., a fixed embedding dimension and depth of transformer encoder).
To study this, we perform a simple experiment on the benchmarkMSMARCO-Passage dataset (Nguyen et al., 2016), which concerns query to passage retrieval; we defer a detailed discussion of this dataset and our training protocol to §5. We focus on the re-ranking setting, wherein both the CA and DE models operate on the outputs of a BM25 retrieval model (Robertson & Zaragoza, 2009). We train a series of BERT-based CA and DE models on the standard (“small”) triplets training set, employing 6-layer BERT models from Turc et al. (2019) with varying embedding size. For each such model, we compute the mean reciprocal rank (MRR)@10 (Radev et al., 2002) on the provided train and dev set.
Figure 1 compares the train and test MRR@10 for cross-attention (CA) and dual-encoder (DE) models. See Table 2 (Appendix) for a numeric summary. With increased embedding dimension, the DE model closely matches the performance of the CA on the training set. This is consistent with the previous section, which suggests suitably deep transformer-based DE models ought to model any ground truth scores; however, there is a sizable gap on the test set. This points to the DE models suffering from generalisation, rather than capacity. See Appendix C.1 for similar results with a 2-layer BERT model.

heading: 3.3 What causes the dual-encoder generalisation gap?

text:
We now attempt to further understand the cause for the DE model’s generalisation gap. We perform a more fine-grained inspection of the CA versus DE model predictions by studying the predicted scores for positive and negative pairs. More precisely, for each query q in the test set, we consider the distribution of scores for positive and negative (query, document) pairs (q, d+) and (q, d−).
Figure 2 shows that both CA and DE models possess clear modes for the positive and negative pairs. However, the CA model can be seen to not “waste” its capacity by unnecessarily modelling fine-grained distinctions amongst negative pairs; rather, it collapses most negatives to a small range of scores, and focusses intead on clearly separating the positives and negatives. By contrast, the DE model has far more diffuse negative scores. This results in a greater overlap in the positive and negative scores, implying the model has more difficulty making distinctions between these pairs. From the right plot, we also see that the DE model has smaller normalised margins (s(q, d+)− s(q, d−))/ρ between the positive and negative pairs, where ρ is the maximal score range.
The DE score distribution for negatives is plausible: when updating a DE model for a given (q, d) pair, the factorised nature of the model implies a non-trivial influence on the scores for all other pairs (q, d′) and (q′, d) sharing either the query or document. Consequently, it is intuitively harder for the model to arrive at a solution that makes fine-grained distinctions. This can be verified qualitatively; the DE model is often seen to make errors on pairs which have only superficial relevance (Appendix C.4).
We make a few remarks here. First, similar trends to the above holds on the training set; see Figure 5 (Appendix). Second, for visual clarity, in Figure 2 we apply a constant offset to each model’s scores to ensure they are mean zero. Figure 6 (Appendix) provides an uncentered plot, showing the DE model scores to have a strong baseline shift over the CA model.
3.4 How can we mitigate the dual-encoder generalisation gap?
Having identified the overfitting problem plaguing DE models, one may naturally ask how to mitigate this. One natural option is to apply standard regularisation strategies, such as dropout; however, our experiments reveal these to be largely ineffective in mitigating DE overfitting (see Appendix C.10). Alternately, the recent literature has identified two promising (complementary) strategies. The first is to modify the scoring function used to compute the final DE model score based on the query and document embeddings; i.e., we replace Equation 2 with s(q, d) = score(T (q), T (d)) for some scoring function score (see, e.g., MacAvaney et al., 2020; Khattab & Zaharia, 2020; Luan et al., 2021). Such strategies have proven effective; however, as these involve some overhead over dot-product scoring, it is natural to ask whether one can extract more gains out of the latter. The second strategy is
to distill the predictions of the CA model into the DE model. By itself, this does not require changing the dot-product scoring in Equation 2. While distillation has been widely studied for classification settings, its use for re-ranking settings is less well-developed. We now study this issue more closely.

heading: 4 Improving dual-encoders via distillation

text: We now propose a family of distillation schemes to improve the performance of DE models.

heading: 4.1 From logit to margin matching

text:
Recall that in distillation settings, each sample is the form (q,d,y, t), where for query q and documents d ∈ DK , we have ground-truth relevance labels y ∈ {0, 1}K and “teacher” model scores t ∈ RK (e.g., the outputs of a CA model). To fit scores s ∈ RK for a “student” model (e.g., a DE model), one can employ a suitable loss `(t, s). Per §2, the simplest choice is the logit matching or mean square error loss. Let us momentarily assume that for each query we have a single associated positive and negative document, with respective teacher scores t = (t1, t2) and student scores s = (s1, s2). Then, the loss simply minimises `mse(t, s) = ‖t− s‖22 = (t1 − s1)2 + (t2 − s2)2. While intuitive, this loss makes the implicit assumption that the range of scores for teacher and student are commensurate. When this is violated — as we empirially observe with the baseline shift between CA and DE model scores, per discussion in §3.2 and Figure 6 (Appendix) — the loss may be ineffective. To get around this, the margin MSE loss (Hofstätter et al., 2020a) matches margins, via
`mmse(t, s) . = ((t1 − t2)− (s1 − s2))2. (3)
Clearly, this loss is translation invariant, and thus is immune to baseline shifts in the scores. However, two issues remain. First, the loss is tied to triplet data with a single positive and negative document. Intuitively, it ought to be beneficial to leverage the teacher scores for multiple documents, with t, s ∈ RK for genericK. One naturalmeans of achieving this is `(t, s) = ∑ i∈P ∑ j∈N ((ti−tj)−(si−sj))2, where P,N denote the set of positive and negative documents, with P ∪N = [K]. Second, the above loss demands matching all documents’ margins exactly. However, this may be an ineffective use of model capacity: for re-ranking tasks, it is primarily important to ensure a clear separation between the positive and negative documents, and overcome the overlapping scores observed in Figure 2. To this end, suppose then that the negative documents are sorted in descending order of the teacher scores t = (t1, . . . , tK). We propose to extend Equation 3 via
`m3se(t, s) = ∑ i∈P ((ti − tj∗)− (si − sj∗))2 + ∑ j∈N [sj − sj∗ ]2+, (4)
where j∗ ∈ N denotes the index of the negative document with highest teacher score. This multimargin MSE (M3SE) loss only attempts to match the margins for the highest scoring negative j∗; for all other negatives j 6= j∗, we simply encourage them to have lower score than j∗. While one may design other intuitively plausible losses, an appealing property of Equation 4 is its relation to the recently proposed RankDistil framework (Reddi et al., 2021), as shall be discussed in §4.3.

heading: 4.2 From margin to probability matching

text:
The above extended the logit matching distillation loss for re-ranking settings, with care to account for differing score ranges for teacher and student. Per §2, the other classic distillation loss is the softmax cross-entropy (Equation 1), which has seen prior use for dual-encoder distillation (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021). In fact, we now show that the softmax cross-entropy generalises the margin MSE loss in a precise sense. Proposition 3. Fix any teacher and student scores t, s ∈ R2. Let `sce(t, s; τ) denote the softmax cross-entropy loss (Equation 1) with temperature τ > 0, and `mmse the margin MSE loss (Equation 3). Then, for temperature scaled sigmoid function σ(z; τ) .= (1 + exp(−τ−1 · z))−1, we have
`sce(t, s; τ) = KLbin(σ(t1 − t2; τ) ‖σ(s1 − s2; τ))
lim τ→+∞
[ τ2 · ∂
∂si `sce(t, s; τ)
] = 1
8 · ∂ ∂si `mmse(t, s) for i ∈ {1, 2}.
We make a few remarks. First, `sce swaps the square loss for a binary KL divergence KLbin(a‖b) . = −a · log ba + (1− a) · log 1−b 1−a in Equation 3. Unlike the square loss, the softmax cross-entropy is Lipschitz, which is typically favourable for generalisation (Luxburg & Bousquet, 2004). Second, the softmax and margin MSE loss derivatives with respect to the student logits converge in the high temperature limit. This gives a precise sense in which the softmax cross-entropy is a smooth approximation to the margin MSE. Intuitively, this is plausible: the former is similarly invariant to constant translations to the scores, thus allowing for differing ranges in the teacher and student.
Third, our argument is subtly different from Hinton et al. (2015), which establishes convergence of the softmax and logit matching derivatives: crucially, the latter only holds under the assumption that the teacher and student logits are centered. We reiterate that this assumption typically does not hold for CA and DE models, per §3.2 and Figure 6 (Appendix). (While it is benign to translate all teacher logits by a fixed constant, this may not guarantee the assumption as the average score 1K ∑ i∈[K] ti is typically not constant across samples.) By contrast, Proposition 3 makes no such assumption.
Finally, Proposition 3 assumes the supervision comprises a a single positive and negative document, to facilitate comparison with Equation 3. The case ofK > 2 documents with teacher and student scores t, s ∈ RK poses no conceptual issue: we may seamlessly minimise the softmax cross-entropy between the two vectors. Since typically K |D|, this does not correspond to matching the full teacher distribution over all possible “classes”. Despite this distinction to the standard use of distillation for classification problems, one may justify the loss as follows. For positive (negative) documents P (Q),
`sce(t, s) ∝ 1
τ ∑ k∈[K] eτ −1·tk∑ i∈P e τ−1·ti + ∑ j∈N e τ−1·tj · log ∑ i∈P eτ −1·(si−sk) + ∑ j∈N eτ −1·(sj−sk)  . This approaches max {maxi∈P [si − sk∗ ]+,maxj∈N [sj − sk∗ ]+} as τ → +∞, where k∗ is the highest scoring document under the teacher. The loss thus encourages the student to preserve the teacher’s highest scoring document, which is useful for re-ranking. Compared to Equation 4, this does not explicitly delineate the highest scoring negative from other negatives; however, empirically, we shall see that the departure from square loss endows the loss with comparable efficacy.

heading: 4.3 Discussion and extensions

text:
Connection to RankDistil. Equation 4 can be related to the recently proposed RankDistil framework by Reddi et al. (2021). Here, the goal is to design distillation losses to match the teacher and student ranking over labels, particularly for the top-k elements. A canonical instantiation is the binary RankDistil formulation (RankDistil-B), `rankB(t, s) = ∑ i∈P (ti − si)2 + ∑ j∈N [sj − γ0]2+. Intuitively, this seeks to match the teacher and student score on positive documents, while ensuring all negative documents fall below some threshold γ0. The relation between this loss and Equation 4 is analogous to that between logit matching and margin MSE: Equation 4 matches margins rather than raw scores, which are thus suitable for settings of differing ranges of the teacher and student scores.
Distillation for retrieval phase. Our focus thus far has been on bridging the observed gap between the CA and DE models for re-ranking. However, the losses presented above are equally applicable during the retrieval phase, where they can similarly be expected to improve DE model performance. These losses are complementary to proposals from several recent works improving the retrieval performance of dual-encoder models, including using suitable hard negative mining for training (Xiong et al., 2021; Zhan et al., 2021a), and quantisation (Zhan et al., 2021b).

heading: 5 Experimental results

text:
We now present experiments that demonstrate the viability of the proposed distillation scheme in §4. We show that this scheme yields better performance compared to several baselines, helping reduce the gap between CA and DE models by mitigating the overfitting issue identified in §3.2.
Per §4.3, we reiterate that our primary goal is to explore the feasibility of bridging the gap between CA and DE models for re-ranking. This is distinct from the more common use of DE models in the literature for retrieval (Chang et al., 2020; Xiong et al., 2021; Zhan et al., 2021a). While we do not make special effort to incorporate tricks from that literature that are beneficial for retrieval
MSMARCO re-rank MSMARCO retrieval TREC DL19 re-rank NQ re-rank Model MRR nDCG MRR nDCG MRR nDCG MRR nDCG
Baselines: one-hot BM25 (Robertson & Zaragoza, 2009) 0.194† 0.241† 0.194† 0.241† 0.689† 0.501† — — ANCE (Xiong et al., 2021) — — 0.330† — — 0.677† — — Cross-attention BERT (12-layer) 0.370 0.430 N/A N/A 0.829 0.749 0.746 0.673 Dual-encoder BERT (6-layer) 0.310 0.360 0.281 0.331 0.834 0.677 0.676 0.601
highlight the best performing DE based model. Distilling the dual-encoder with our proposed techniques significantly improves performance over one-hot training and existing distillation techniques. Results marked † are quoted from the corresponding reference, “N/A” are not applicable (e.g., the cross-attention model is not feasible to apply for retrieval), and “—” are not available from the reference. ♦ See Appendix C.11 for analysis of potential label noise influencing the results. ? See text for discussion.
(e.g., using hard negatives), we do demonstrate that our proposed distillation techniques can improve performance for this stage over simply using one-hot training labels.
Datasets. We present results on MSMARCO-Passage (Nguyen et al., 2016) and Natural Questions (NQ) (Kwiatkowski et al., 2019). For MSMARCO, we report results on the standard dev set, and the TREC DL19 test set (Craswell et al., 2020). MSMARCO comprises (query, passage) records and their relevance labels from a human rater; for each query, the associated passages are retrieved from a BM25 model. The training set is canonically represented in triplet form, comprising a query, and a single positive and negative passage. NQ comprises user questions andWikipedia passages potentially containing their answers; we use a processed version from Karpukhin et al. (2020) (cf. Appendix B).
Models. We use transformer encoders initialised with the standard pre-trained BERT model checkpoints. Following Hofstätter et al. (2020a), we use BERT-Base for the CA model, and a 6-layer BERT model (Turc et al., 2019) with embedding size 768 for all DE models. For the DE models, we tie the query and document encoder parameters. For distillation, we use the CA model as the “teacher” and the DE model as the “student”. On MSMARCO, for ease of comparison, we use the “T1” CA model annotations from Hofstätter et al. (2020a), which achieves similar re-re-ranking performance as our independently trained CA model. See Appendix B for further training details.
Metrics. We report the MRR@10 and nDCG@10 for all methods. Note that we are primarily interested in re-ranking tasks, wherein the test set comprises documents retrieved by a BM25 baseline, and our models are used to re-rank these candidates. For MSMARCO, we additionally consider full retrieval performance obtained by scoring all (8.8M) passages, but reiterate that maximising performance on this metric requires additional negative mining that we do not explore.
Baselines. We consider methods that leverage either the observed (“one-hot”) training labels on the triplet data, or the predictions from the cross-attention (“teacher”) model on the triplet data. For the former, we employ softmax cross-entropy against the using a cross-attention and dual-encoder model. For the latter, we employ the logit MSE loss (Ba & Caruana, 2014), margin MSE loss (Hofstätter et al., 2020a), and the binary version of RankDistil (RankDistil-B) (Reddi et al., 2021).
These are compared against the proposed M3SE (Equation 4), and the softmax cross-entropy (Equation 1). The latter methods handle of an arbitrary number of documentsK in the supervision (i.e., they are not restricted to triplet data). We thus aggregate the triplet data by query, and retain the top-20 passages with highest teacher model score. We study the sensitivity toK in Appendix C.7.

heading: 5.1 Results and discussion

text:
Table 1 summarises the results of all methods. We discuss a few salient findings.
Baseline performance. Comparing the performance of CA and DE models in Table 1, we make two initial observations. First, there is a sizable performance gap between the models when using one-hot labels. Second, in line with Hofstätter et al. (2020a), naïve logit matching via the MSE underperforms, due to the vastly different scales of the two models; this is however assuaged by the margin MSE loss.
Efficacy of distillation. Both the M3SE and the softmax cross-entropy losses significantly outperform existing methods on the MSMARCO and NQ re-reranking tasks, and shrinks the gap between the DE and CA models. On the TREC DL19 test set, our proposed methods appear to fall short of the margin MSE MRR@10; a closer inspection of the apparent loss cases (Appendix C.11) reveals these are unanimously the results of false negatives in the provided labels.
Figure 1 shows that M3SE distillation nearly closes the gap between CA and DE models with sufficiently large embedding dimension. Intriguingly, the test gains comes with some sacrifice of training MRR for the highest embedding size. This is consistent with findings in the classical distillation literature, where distillation can harm training accuracy but improve test accuracy (Cho & Hariharan, 2019). To further study this, Figure 4 (Appendix) shows the learning curve on the MSMARCO train and test set, which demonstrates a consistent gap in test set performance as training progresses. Distillation yields a training MRR that closely matches that of the DE model for the first 100K training steps; subsequently, however, the training MRR under distillation begins to saturate, while that of the DE model with one-hot labels keeps increasing. Intuitively, this suggests that the DE model starts exploring a poorly generalising part of the parameter space beyond this point.
As a final illustration, Figure 2 (right) shows that softmax cross-entropy distillation makes the DE scores better behaved: it shifts the margins of the DE model to be closer to the the CA model.
Beyond the dot-product: generic scoring functions. We have thus far focussed on results using the standard dot-product based scoring for the DE models, per Equation 2. It is of interest as to what impact more complex scoring functions, per §3.4, have on final model performance. To study this, we consider the ColBERT model (Khattab & Zaharia, 2020), which computes the average of the maximum query-document token similarities. Consistent with Khattab & Zaharia (2020), using this model with one-hot labels by itself improves performance significantly over the dot-product.
When further combined with distillation, the performance exceeds that of the CA teacher, reaching an MRR@10 of 0.376 for the MSMARCO re-ranking task under the softmax cross-entropy loss. (Full results in Appendix, Table 7.) This further highlights that factorised representations need not imply a loss in performance, provided they are suitably trained. We defer to future work a similar analysis of the conceptually similar multi-vector scoring layer of Luan et al. (2021).
Full-retrieval performance. Thus far, we have focussed on the re-ranking performance of all models, wherein the predictions from a BM25 retriever are provided as input. This allows for the comparison of DE and CA models on an equal footing. One may however naturally wonder how the proposed methods fare in the retrieval phase, wherein the models score all possible passages. Table 1 also summarises the results in this setting on MSMARCO. We again see that the proposed distillation techniques offer strong gains over standard one-hot training of the DE model, as well as existing distillation techniques. We remark also that the poor retrieval performance of the MSE loss is a result of the model strongly overfitting to the teacher scores on the provided documents.
Additional results. In the Appendix, we present several additional results and ablations, including the sensitivity of performance to the number of documentsK in the distilled supervision (Table 6); the effectiveness of label smoothing as opposed to full distillation (Appendix C.9); and the impact of other regularisation strategies aimed at minimising overfitting (Appendix C.10).

heading: 6 Conclusion and future work

text:
Several avenues for future work remain open. For example, in the literature on distilling between cross-attention models, a common strategy is to match more than simply output logits (Sun et al., 2019; Turc et al., 2019; Sanh et al., 2019; Xu et al., 2020; Sun et al., 2020; Jiao et al., 2020; Wang et al., 2020). Are such strategies useful when distilling to a dual encoder? Further, our distillation strategy assumes access to scores from a cross-attention model. Can one do away with this altogether, e.g., using self-distillation and related ideas (Furlanello et al., 2018)? Finally, the efficacy of the proposed losses when combined with recent advances for retrieval (Xiong et al., 2021) is of interest.
Ethics Statement
This work concerns the mathematical and empirical analysis of techniques for neural re-ranking, a particular problem encountered in information retrieval. We do not foresee the former having undue societal effects. The latter does not explicitly consider issues of fairness in retrieval, which is an important yet under-studied dimension. We do not foresee our distillation techniques as unduly amplifying biases in existing algorithms; nonetheless, we consider a more careful study of the topic of fairness in retrieval an important avenue for future work.

heading: A Proofs

text:
Proof of Proposition 1. For any q ∈ Q, let us write sq . = s(q, ·) : D→ R. Pick a reproducing kernel Hilbert space H over D with associated inner-product 〈·, ·〉H. Suppose further H has associated kernel k : D×D→ R that is continuous, and universal (Micchelli et al., 2006). Then,H is dense in the space of continuous functions (Sriperumbudur et al., 2011). Recall that by assumption sq is continuous. Consequently, each sq can be approximated to arbitrary precision (with respect to the `∞ metric) by some s∗q ∈ H. Since s∗q ∈ H, by the reproducing property of an RKHS,
(∀d ∈ D) s∗(q, d) = 〈s∗q , kd〉H,
where in an abuse of notation kd . = k(d, ·) ∈ H. Observe that we may interpret s∗q as an “embedding” for the query q, and kd a “weight vector” for the document d.
It may appear at this stage that we are done; however, each of s∗q and kd may be uncountably infinite dimensional objects, and the inner product in H may not be the standard dot-product. Nonetheless, under the stated assumptions (in particular, compactness of D), we may appeal to a Mercer representation of the kernel, following Steinwart & Christmann (2008, Theorem 4.51). In particular, we may construct a Mercer feature map Φ: D→ `2 forH, such that
(∀h ∈ H) (∃(an)∞n=1 ∈ `2)h(d) = ∞∑ n=1 an · Φn(d).
Further, the inner product between any h1, h2 ∈ H with “Mercer coefficients” (an)∞n=1, (bn)∞n=1 is expressible as
〈h1, h2〉H = ∞∑ n=1 an · bn,
i.e., a familiar dot-product. In our context, write the Mercer coefficients for s∗q as (zqn)∞n=1, and for kd as (wdn)∞n=1. Thus, we may write
〈s∗q , kd〉H = z>q wd,
where the vectors zq,wd have at most countably infinite dimension.
Proof of Proposition 2. Any continuous z : RD×L → RD can be trivially associated with z̃ : RD×L → RD×L, where each column z̃:i = z. Such a mapping preserves continuity. By Yun et al. (2020, Theorem 2), we may thus find a transformer encoder T : RD×L → RD×L such that ∫ Q ‖z̃(Q) − T(Q)‖22 dQ ≤ ′. Since the integrad is nonnegative, we must also have ‖z̃(Q)−T(Q)‖22 dQ ≤ ′ for any givenQ. Consequently, we must also have ‖z̃(Q):1−T(Q):1‖22 ≤ ′ almost surely, which by definition implies ‖z(Q)−T(Q):1‖22 ≤ ′,
Proof of Proposition 3. For temperature τ > 0, simple algebra reveals Equation 1 to be
`sce(t, s) = − ∑
y∈{1,2}
exp(τ−1 · ty) exp(τ−1 · t1) + exp(τ−1 · t2) · log exp(τ −1 · sy) exp(τ−1 · s1) + exp(τ−1 · s2)
= ∑
z∈{±1}
1
1 + ez·τ−1·(t2−t1) · log(1 + ez·τ −1·(s2−s1))
= − ∑
z∈{±1}
σ(z · (t1 − t2); τ) · log(σ(z · (s1 − s2); τ)).
This is equivalent to KLbin(σ(t1 − t2; τ) ‖σ(s1 − s2; τ)), i.e., swapping out the square loss for the KL divergence in Equation 3. In fact, one can make a stronger connection: observe that
∂
∂s1 `mmse(t, s) = 2 · ((s1 − s2)− (t1 − t2))
∂
∂s1 `sce(t, s) = − ∑ z∈{±1} σ(z · (t1 − t2); τ) · ∂ ∂s1 log(σ(z · (s1 − s2); τ)
= τ−1 ·
( eτ −1·s1
eτ−1·s1 + eτ−1·s2 − e
τ−1·t1
eτ−1·t1 + eτ−1·t2
)
= τ−1 · (
1 1 + eτ−1·(s2−s1) − 1 1 + eτ−1·(t2−t1) ) ∼ τ−2 · ( s1 − s2
4 − t1 − t2 4
) ,
as τ → +∞, where the last line follows from the Taylor series approximation
1
1 + exp(−z) =
1 2 + z 4 + O(z2)
as z → 0.

heading: B Experiment hyperparameters

text:
For all models, at the output layer we apply dropout at rate 0.1 and layer normalisation. We use a sequence length of 30 for queries, and 200 for passages. We optimise all methods for a maximum of 3 × 105 steps using Adam with weight decay, with a batch size of 128 and a learning rate of 2.8× 10−5 (i.e., a 4× scaling of the choices in Hofstätter et al. (2020a)). Following Hofstätter et al. (2020a), we perform early stopping based on the nDCG@10 metric.
For the Natural Questions dataset, we use the processed version used in Karpukhin et al. (2020), that contains questions, positive passages with a correct answer to each question, and a corpus of all Wikipedia passages. To train DE, CA, and multi-negative DIST models, we use 19 BM-25 hard-negative passages for each question along with a positive passage. For single-negative DIST models we use a single BM-25 negative from the same collection. To calculate MRR@10 and nDCG@10 metrics, we use the queries in the dev set with 200 passages containing positives, 100 BM-25 hard-negatives and up to 100 random negatives.

heading: C Additional experiments

text:
C.1 CA versus DE models: effect of base architecture
Figure 3 presents an an analogue of Figure 1, but using instead a 2-layer BERT model. As before, we initialise using the pre-trained models developed in Turc et al. (2019). The same general trends hold: for sufficiently large embedding dimension, the DE model closely matches the performance of the CA on the training set; however, there is a sizable gap in the test set performance. Table 2 presents a numeric summary of the train and test performance for CA and DE models, under different choices of base BERT-model architecture (in terms of number of layers, as well as the final embedding dimension).
C.2 CA versus DE models: evolution of train and test performance
The above results summarise the results at the completion of training. For a finer-grained understanding of how performance evolves during training, Figure 4 presents the learning curves for CA, DE, and distilled DE (DIST) models on the train and test set. Here, we use the small-bert-6-768 architecture for all models. We observe that the CA and DE models initially have a non-trivial gap in performance, but this shrinks over time; this further points to the initialisation of the CA models being more favourable for generalisation. Note also that beyond a certain point, both models continually improve their training performance, but are completely neutral in terms of test performance. Interestingly, distillation largely tracks the DE model performance, but then worsens on the training set while improving on the test set. This suggests that distillation helps explore a better part of the search space.
C.3 CA versus DE models: comparison of score distributions
Figure 5 compares the score distributions for the CA and DE models on the training set. We observe the same general trends as the test set (Figure 2). Figure 6 further presents the test score distributions without zero-centering. We see that the DE model exhibits a strong baseline shift, wherein the scores are of the order of ∼ 700. Figure 7 studies the score distributions for the CA and DE models after varying numbers of training steps. At initialisation, as expected, both models have essentially overlapping distributions for the positives and negatives. However, here too, we observe that the DE model scores operate on a much wider range than the DE model. The middle row continues this study by looking at behaviour after 10, 000 steps of training, which is a small fraction of the 300, 000 steps used for training of all models. Here, we start to observe a clearer separation between the positive and negative scores for both models. Interestingly, the CA model already sees a sharp peak on the negatives, unlike the DE model. The bottom row shows the test set score distributions of the final trained models, to complement Figure 2 from the body. We see the same general trend as the training set: the CA model is seen to more confidently distinguish positives from negatives, and operate at a narrower range of scores.
C.4 CA versus DE models: qualitative analysis
Table 3 provides a sample of (query, passage) pairs from MSMARCO-Passage dev set, where there is a large discrepancy between the CA and DE model scores. Specifically, we consider pairs that the CA model scores low, but the DE model scores high. Interestingly, these pairs typically involve strong token overlap between the query and passage — indicating a certain degree of topicality — but are fact genuine negative pairs. This reflects that the DE model may be unable to capture certain fine-grained distinctions.
Query Passage
effects of yeast on body The Side Effects of Chemotherapy on the Body. Chemotherapy drugs are powerful enough to kill rapidly growing cancer cells, but they also can harm perfectly healthy cells, causing side effects throughout the body.
age not to take off shoes at airline security Airline Identification Requirements. Airlines do not typically require identification from passengers under the age of 18, but there are exceptions. Children under the age of 2 may ride on a parent’s lap without purchasing a ticket, but the airline will require identification, such as a birth certificate, to prove the child’s date of birth . . .
actress who plays alice on the magicians Then portrayed as the animated Alice’s real life counterpart by actress Mia Wasikowska as a more mature, grown up Alice in Disney’s 2010 semi-sequel, live action/CGI film Alice and Wonderland Directed by Tim Burton. In the Broadway musical version, she will be played by Taylor Louderman . . .
can you absorb metals from plants Answer 1: Photosynthesis is the ability of plants to absorb the energy of light, and convert it into energy for the plant. To do this, plants have pigment molecules which absorb the energy of light very well. The pigment responsible for most light-harvesting by plants is chlorophyll, a green pigment.The green color indicates that it is absorbing all the non-green light– the blues ( 425-450 nm), the reds and yellows (600-700 nm).he pigment responsible for most light-harvesting by plants is chlor . . .
Table 3: Sample of (query, passage) pairs from MSMARCO-Passage dev set with largest discrepancy between the CA and DE model scores. In most of these cases, the passage is not relevant to the query; however, there is a high degree of token overlap between the two, indicating superficial similarity.
C.5 Impact of distillation: effect on score distributions
We have previously seen in Figure 7 that our proposed M3SE distillation strategy can improve the distribution of scores for positives and negatives. Figure 8 shows the impact that such distillation has on the margins between positive and negatives, compared to standard training on one-hot labels. The distilled model is seen to more confidently distinguish positives from negatives, as evidenced by the margin distribution having more mass on larger values.
C.6 Impact of distillation: effect of student model architecture
The preceding distillation results have all employed a small-bert-6-768 model. It is of interest how sensitive the results are to this choice. To study this, we report in Table 4 the re-ranking results on MSMARCO-Passage when using a small-bert-6-128 model. We observe largely consistent trends as before: distillation using our proposed losses can close the gap between CA and DE model performance. Table 5 repeats this for a small-bert-2-768 model.
C.7 Impact of distillation: effect of number of documents in supervision
The results reported in the body employed a total of 20 documents per query for the softmax CE and M3SE methods. As this involves significantly more information than the triplet data used, e.g., in training the margin MSE loss, it is of interest to tease apart how much of the gains from the method come from this increased supervision, versus the loss itself. Table 6 studies the effect of varying the number of documents. We see that there are diminishing returns beyond a certain point: using 100 documents yields qualitatively similar performance to using 20 documents. At the same time, there is a marked difference in performance when using 2 documents versus 10.
C.8 Results with ColBERT scorer
Table 7 presents results using the ColBERT scorer. We find that with distillation using the softmax CE loss (Equation 1), we can exceed the performance of the cross-attention teacher model for the re-ranking task. This further indicates the viability of dual-encoder models for neural ranking.
MSMARCO rerank TREC DL19 MRR@10 nDCG@10 MRR@10 nDCG@10
Baselines: one-hot Cross-attention BERT 0.370 0.430 0.829 0.749 Dual-encoder ColBERT 0.356 0.416 0.839 0.703
C.9 Beyond distillation? Effect of label smoothing
A special case of distillation is label smoothing (Szegedy et al., 2016), which can be understood as using a teacher that predicts the uniform distribution over all labels, and combining the result
Smoothing α MRR@10 train MRR@10 test
with the one-hot label. Specifically, this involves mixing the observed one-hot labels with a uniform distribution over all labels, with the mixing controlled by weight α ∈ [0, 1); α = 0 corresponds to standard one-hot training. Label smoothing has also proven effective as a means of preventing harmful overfitting, and thus improving generalisation (Müller et al., 2019; Lukasik et al., 2020) in classification settings.
For the MSMARCO-Passage dataset, we studied the effect of varying degrees of label smoothing for DE models, using the softmax CE loss on triplet data. We use a small-bert-6-768 model as before; for computational ease, we report results after 300, 000 steps of training, at which point all models see stable test error. Table 8 reveals that, surprisingly, smoothing has a minimal effect on test performance, even at high smoothing levels. This is despite smoothing reducing the training performance at higher values, which is expected given its equivalence to injecting symmetric label noise to each sample. Further study of the viability of smoothing for retrieval problems would be of interest.
C.10 Beyond distillation? Effect of alternate regularisation strategies
Distillation provides one means of preventing the DE model from overfitting to the training set. One may of course conceive of other plausible strategies, such as:
• increasing the dropout rate in the final (embedding) layer of the transformer. The experiments in the body employ 10% dropout (following prior work, e.g., Hofstätter et al. (2020a)), which improves performance considerably over having no dropout; it is thus of interest whether there are benefits to further increasing this.
• performing token dropout at the input layer. Intuitively, such corruption prevents the model from overly relying on individual tokens, thus preventing spurious token matching (as present in some cases in Table 3).
• adding a masked language model (Devlin et al., 2019) loss for the student. This loss is a core component of BERT, and has thus proven to be effective in learning generically useful sequence embeddings. Intuitively, adding such a self-supervised objective to training can plausibly improve the model’s robustness.
• modifying the base loss from square or log-loss to the focal loss (Lin et al., 2017). This loss has proven effective in vision tasks involving high class-imbalance. In our setting, as there are typically only a handful of relevant documents, but several millions of irrelevant documents, it is of interest whether such a loss can positively impact results.
Table 9 summarises results for each of these strategies on the MSMARCO Passage re-ranking task. Here, we focus on their effect on the standard (one-hot) training of a dot-product based DE model on triplet data. Unfortunately, we find that none of these strategies significantly improve the generalization performance of the baseline model. Thus, mitigating the overfitting observed in Figure 1 requires more effort than appealing to standard strategies effective in classification tasks.
C.11 Analysis of TREC DL19 predictions
Table 1 suggests a notable gap in performance for our proposed RankDistil variant and the margin MSE loss (Hofstätter et al., 2020a). Here, we take a closer look at the loss cases for our method. The
TREC DL19 data comprises a total of 43 queries, each with between∼ 100 to 500 rated passages. Of these queries, the MRR@10 of the DE models trained with margin MSE and softmax CE agree on 10. On the queries with disagreement, 6 cases favour margin MSE, and 4 cases favour RankDistil-SM.
For the 6 queries where RankDistil-SM ostensibly underperforms, we inspect the top-5 scoring passages for both margin MSE and softmax CE in Table 10 and 11. The cells shaded blue correspond to passages rated positive. Several other passages are however seen to be equally valid answers to the source query. Indeed, we submit that in all cases, the predictions from RankDistil are of at least the same quality as the margin MSE.
C.12 Impact of negative mining on re-ranking performance
We consider the value of adding additional negatives during DE model training. Following Karpukhin et al. (2020); Qu et al. (2021), we consider the use of within-batch (also referred to as in-batch) and uniform negatives. For the latter, in each training minibatch, we draw a sample of Buni documents drawn uniformly at random from the entire set of documents; each such document is treated as a negative document in the loss. For the former, from each training minibatch {(qi,di,yi)}Bi=1, we compute the set of all observed documents Dobs = ∪Bi=1 ∪Kk=1 {dik}. For each sample (qi,di,yi), we then use each element in Dobs − ∪Kk=1{dik} as a negative document in the loss. Table 12 summarises the results for the MSMARCO re-ranking task. We find that adding these negatives has a small gain for the one-hot model performance. However, for the distilled objectives, there is limited gain (and sometimes even a degradation) in performance. More study of this issue is warranted, but one hypothesis is that the re-ranking task is inherently concerned with ranking the outputs of a BM25 model. These outputs are precisely used to construct the training data used for distillation. It is thus possible that adding additional negatives — which are unlikely to appear as candidates for re-ranking — do not bring significantly useful information.
We emphasise here also that the results reported are for the re-ranking, as opposed to retrieval task. For the latter, adding within-batch and uniform negatives is intuitively and empirically valuable, as noted in Karpukhin et al. (2020); Qu et al. (2021). For example, in training the one-hot DE model on the triplet data, the retrieval MRR@10 is 0.281. When adding within-batch and uniform negatives, this increases to 0.314. Given that the re-ranking MRR@10 remains relatively unchanged (0.312 versus 0.310), this is further indication of the re-ranking and retrieval objectives not being perfectly aligned.
Query Top scoring passages Margin MSE RankDistil-SM
who is robert gray Who is Henry Gray? Henry Gray is an African-American blues piano player and singer. He has been play. . . Robert Grey (born 21 April 1951 in Marefield, Leicestershire) is an English musician best known as t. . .
Robert Gray was the Democratic candidate for governor of Mississippi in the 2015 elections. Gray won. . . Who is Henry Gray? Henry Gray is an African-American blues piano player and singer. He has been play. . .
Kenneth Gray (I) Kenneth Gray is an actor, known for Love, Lies and Murder (1991), Retribution (1987. . . Robert Gray was the Democratic candidate for governor of Mississippi in the 2015 elections. Gray won. . .
Kenneth Gray (I) Actor. Kenneth Gray is an actor, known for Love, Lies and Murder (1991), Retributio. . . Robert Gray. A surprise came on the Democratic side in the race for Mississippi Governor. Robert Gra. . .
Robert Grey (born 21 April 1951 in Marefield, Leicestershire) is an English musician best known as t. . . William Thomas Gray. Billy Gray was born on January 13, 1938 in Los Angeles, California, USA as Will. . .

title: In defense of dual-encoders for neural ranking