summaries: This paper investigate the gap problem between cross-attention (CA) and dual-encoder (DE) models for document reranking. In general, cross-attention models perform much better than dual-encoder models while consuming more computational cost. The authors first leverage Mercer's theorem to prove that dual-encoders models are sufficiently expressive to model a broad class of scores using countably infinite dimension, which mainly the gap between CA and DE models should not exist from theoretical analysis. They further identified the gap is caused by the generalization ability of the DE models. To reduce the overfitting on the training data, they proposed to use distillation algorithms. Specially, they design the loss function which matches the margins between the teacher (CA) and student (DE) models. They also extend the margin to probability matching by using softmax cross-entropy loss. Through the experiments, they demonstrated that their proposed distillation approaches can further reduce the gap between DE and CA models.  The main contributions of this paper is that they show there should not be a gap between DE and CA models from theoretical analysis. They found that the gap is caused by the generalization ability of CA model and proposed distillation approach to reduce the gap. In information retrieval problems, there are two kinds of models: dual-encoder models and cross-attention models. Cross-attention models generally outperform dual encoder models by a large margin. Due to the latency requirement of real search systems, cross-attention models might not meet the requirements. So one ideal solution is to transfer the knowledge from cross-attention models to dual encoder models. This paper follows this line of research ideas by reducing the gap between the two models.   Firstly, the paper theoretically analyzes the effects with sufficiently large embedding dimensions. Secondly, the paper empirically verifies the theoretical results. Thirdly, the paper proposes a new method that further bridges the gap between the two methods. The writing is clear and well-structured. I like the idea and logic of the paper. I have some questions about the experiment part.  This paper aims to narrow the performance gap between cross-attention BERT and dual-encoder BERT for re-ranking task. The authors empirically and theoretically analyze the underlying reasons of the performance gap. The gap could be mitigated by the proposed knowledge distillation method, where the original cross-attention BERT model acts as the “teacher” and the more efficient dual-encoder BERT model is used as student. Comprehensive experiments confirmed the effectiveness of the proposed KD method for various re-ranking tasks.    The paper contrasts cross-encoder vs. bi-encoder architectures for re-ranking in information retrieval.  The authors argue that bi-encoders under-perform cross-encoders, not for lack of capacity, but due to poor generalization, and back this claim with some theory and empirical results.  The authors then propose distillation of the cross-encoder model into a bi-encoder, which is shown to improve bi-encoder results.  While this has been proposed before, the distillation loss function the authors propose is slightly different than the standard cross-entropy (which the authors show is closely related).
abstractText: Transformer-based models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query. There are two broad flavours of such models: cross-attention (CA) models, which learn a joint embedding for the query and document, and dual-encoder (DE) models, which learn separate embeddings for the query and document. Empirically, CA models are often more accurate, which has motivated a series of works seeking to bridge this gap. However, a fundamental question remains less explored: does this performance gap reflect a limitation in the capacity of DE models, or in the training of such models? In this paper, we study this question, with three contributions. First, we establish theoretically that with a sufficiently large encoder size, DE models can capture a broad class of scores without cross-attention. Second, we show empirically that on real-world problems, the gap between CA and DE models may be due to the latter overfitting to the training set. To mitigate this behaviour, we propose a distillation strategy that focuses on preserving the ordering amongst documents, and confirm its efficacy on benchmark neural re-ranking datasets.
references.authors: Jimmy Ba Rich Caruana
references.title: Do deep nets really need to be deep
references.venue: Advances in Neural Information Processing Systems,
references.authors: Jane Bromley Isabelle Guyon Yann LeCun Eduard Säckinger Roopak Shah
references.title: Signature verification using a “Siamese” time delay neural network
references.venue: In Proceedings of the 6th International Conference on Neural Information Processing Systems,
references.authors: Wei-Cheng Chang Felix X. Yu Yin-Wen Chang Yiming Yang Sanjiv Kumar
references.title: Pre-training tasks for embedding-based large-scale retrieval
references.venue: In 8th International Conference on Learning Representations,
references.authors: J.H. Cho B. Hariharan
references.title: On the efficacy of knowledge distillation
references.venue: IEEE/CVF International Conference on Computer Vision (ICCV),
references.authors: Nick Craswell Bhaskar Mitra Emine Yilmaz Daniel Campos
references.title: Overview of the TREC 2020 deep learning track
references.venue: Proceedings of the Twenty-Ninth Text REtrieval Conference,
references.authors: Zhuyun Dai Jamie Callan
references.title: Deeper text understanding for IR with contextual neural language modeling
references.venue: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: pre-training of deep bidirectional transformers for language understanding
references.venue: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Tommaso Furlanello Zachary Chase Lipton Michael Tschannen Laurent Itti andAnimaAnandkumar
references.title: Born-again neural networks
references.venue: In Proceedings of the 35th International Conference on Machine Learning,
references.authors: Luyu Gao Zhuyun Dai Jamie Callan
references.title: Understanding BERT rankers under distillation
references.venue: In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval,
references.authors: Daniel Gillick Sayali Kulkarni Larry Lansing Alessandro Presta Jason Baldridge Eugene Ie Diego Garcia-Olano
references.title: Learning dense representations for entity retrieval, 2019
references.authors: Ruiqi Guo Philip Sun Erik Lindgren Quan Geng David Simcha Felix Chern Sanjiv Kumar
references.title: Accelerating large-scale inference with anisotropic vector quantization
references.venue: In Proceedings of the 37th International Conference on Machine Learning,
references.authors: Geoffrey Hinton Oriol Vinyals Jeff Dean
references.title: Distilling the knowledge in a neural network
references.authors: Sebastian Hofstätter Sophia Althammer Michael Schröder Mete Sertkan Allan Hanbury
references.title: Improving efficient neural ranking models with cross-architecture knowledge distillation
references.venue: CoRR, abs/2010.02666,
references.authors: Sebastian Hofstätter Markus Zlabinger Allan Hanbury
references.title: Interpretable & time-budget-constrained contextualization for re-ranking
references.venue: ECAI 2020 - 24th European Conference on Artificial Intelligence,
references.authors: Sebastian Hofstätter Bhaskar Mitra Hamed Zamani Nick Craswell Allan Hanbury
references.title: IntraDocument Cascading: Learning to Select Passages for Neural Document Ranking
references.venue: In Proc. of SIGIR,
references.authors: Gautier Izacard Edouard Grave
references.title: Distilling knowledge from reader to retriever for question answering, 2020
references.authors: Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li Fang Wang Qun Liu
references.title: TinyBERT: Distilling BERT for natural language understanding
references.venue: In Findings of the Association for Computational Linguistics: EMNLP
references.authors: Jeff Johnson Matthijs Douze Hervé Jégou
references.title: Billion-scale similarity search with gpus
references.venue: IEEE Transactions on Big Data,
references.authors: Vladimir Karpukhin Barlas Oguz SewonMin Patrick Lewis Ledell Wu Sergey Edunov Danqi Chen Wen-tau Yih
references.title: Dense passage retrieval for open-domain question answering
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781,
references.authors: Omar Khattab andMatei Zaharia
references.title: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, pp. 39–48
references.venue: Association for Computing Machinery,
references.authors: Tom Kwiatkowski Jennimaria Palomaki Olivia Redfield Michael Collins Ankur Parikh Chris Alberti Danielle Epstein Illia Polosukhin Matthew Kelcey Jacob Devlin Kenton Lee Kristina N. Toutanova Llion Jones Ming-Wei Chang Andrew Dai Jakob Uszkoreit Quoc Le Slav Petrov
references.title: Natural questions: a benchmark for question answering research
references.venue: Transactions of the Association of Computational Linguistics,
references.authors: Kenton Lee Ming-Wei Chang Kristina Toutanova
references.title: Latent retrieval for weakly supervised open domain question answering
references.venue: Proceedings of the 57th Conference of the Association for Computational Linguistics,
references.authors: Tsung-Yi Lin Priya Goyal Ross B. Girshick Kaiming He Piotr Dollár
references.title: Focal loss for dense object detection
references.venue: In IEEE International Conference on Computer Vision, ICCV 2017,
references.authors: Wenhao Lu Jian Jiao Ruofei Zhang
references.title: Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval
references.venue: In Proceedings of the 29th ACM International Conference on Information & Knowledge Management,
references.authors: Yi Luan Jacob Eisenstein Kristina Toutanova Michael Collins
references.title: Sparse, Dense, and Attentional Representations for Text Retrieval
references.venue: Transactions of the Association for Computational Linguistics, 9:329–345,
references.authors: Michal Lukasik Srinadh Bhojanapalli Aditya Krishna Menon Sanjiv Kumar
references.title: Does label smoothing mitigate label noise
references.venue: In Proceedings of the 37th International Conference on Machine Learning,
references.authors: Ulrike von Luxburg Olivier Bousquet
references.title: Distance–based classification with lipschitz functions
references.venue: J. Mach. Learn. Res.,
references.authors: Xueguang Ma Kai Sun Ronak Pradeep Jimmy Lin
references.title: A replication study of dense passage retriever
references.venue: CoRR, abs/2104.05740,
references.authors: Sean MacAvaney Andrew Yates Arman Cohan Nazli Goharian
references.title: Cedr: Contextualized embeddings for document ranking
references.venue: In SIGIR,
references.authors: Sean MacAvaney Franco Maria Nardini Raffaele Perego Nicola Tonellotto Nazli Goharian Ophir Frieder
references.title: Efficient Document Re-Ranking for Transformers by Precomputing Term Representations, pp. 49–58
references.venue: Association for Computing Machinery,
references.authors: Irina Matveeva Chris Burges Timo Burkard Andy Laucius Leon Wong
references.title: High accuracy retrieval with multiple nested ranker
references.venue: In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,
references.authors: Charles A. Micchelli Yuesheng Xu Haizhang Zhang
references.title: Universal kernels
references.venue: Journal of Machine Learning Research,
references.authors: Antoine Miech Jean-Baptiste Alayrac Ivan Laptev Josef Sivic Andrew Zisserman
references.title: Thinking fast and slow: Efficient text-to-visual retrieval with transformers
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021,
references.authors: Bhaskar Mitra Nick Craswell
references.title: An introduction to neural information retrieval
references.venue: Foundations and Trends® in Information Retrieval,
references.authors: Rafael Müller Simon Kornblith Geoffrey E. Hinton
references.title: When does label smoothing help? In Advances in Neural Information Processing Systems
references.venue: Annual Conference on Neural Information Processing Systems
references.authors: Tri Nguyen Mir Rosenberg Xia Song Jianfeng Gao Saurabh Tiwary RanganMajumder Li Deng
references.title: MS MARCO: A human generated machine reading comprehension dataset
references.venue: Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016,
references.authors: Rodrigo Nogueira Kyunghyun Cho
references.title: Passage re-ranking with BERT
references.venue: CoRR, abs/1901.04085,
references.authors: Yingqi Qu Yuchen Ding Jing Liu Kai Liu Ruiyang Ren Wayne Xin Zhao Daxiang Dong Hua Wu Haifeng Wang
references.title: RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering
references.venue: Proceedings of the 2021 Conference of the North American Chapter of the Association
references.authors: Dragomir R. Radev Hong Qi Harris Wu Weiguo Fan
references.title: Evaluating web-based question answering systems
references.venue: In Proceedings of the Third International Conference on Language Resources and Evaluation
references.authors: Sashank Reddi Rama Kumar Pasumarthi Aditya Menon Ankit Singh Rawat Felix Yu Seungyeon Kim Andreas Veit Sanjiv Kumar
references.title: RankDistil: Knowledge distillation for ranking
references.venue: Proceedings of The 24th International Conference on Artificial Intelligence and Statistics,
references.authors: Nils Reimers Iryna Gurevych
references.title: Sentence-bert: Sentence embeddings using Siamese BERTnetworks
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics,
references.authors: Stephen Robertson Hugo Zaragoza
references.title: The probabilistic relevance framework: Bm25 and beyond
references.venue: Found. Trends Inf. Retr.,
references.authors: Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf
references.title: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
references.venue: In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing
references.authors: Bharath K. Sriperumbudur Kenji Fukumizu Gert R.G. Lanckriet
references.title: Universality, characteristic kernels and RKHS embedding of measures
references.venue: Journal of Machine Learning Research,
references.authors: Ingo Steinwart Andreas Christmann
references.title: Support Vector Machines. Information Science and Statistics
references.authors: Siqi Sun Yu Cheng Zhe Gan Jingjing Liu
references.title: Patient knowledge distillation for BERT model compression
references.venue: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,
references.authors: Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou
references.title: MobileBERT: a compact task-agnostic BERT for resource-limited devices
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Zbigniew Wojna
references.title: Rethinking the inception architecture for computer vision
references.venue: IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Iulia Turc Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: Well-read students learn better: The impact of student initialization on knowledge distillation
references.venue: CoRR, abs/1908.08962,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,
references.authors: Wenhui Wang Furu Wei Li Dong Hangbo Bao Nan Yang Ming Zhou
references.title: Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers
references.venue: Advances in Neural Information Processing Systems,
references.authors: Lee Xiong Chenyan Xiong Ye Li Kwok-Fung Tang Jialin Liu Paul N. Bennett Junaid Ahmed Arnold Overwijk
references.title: Approximate nearest neighbor negative contrastive learning for dense text retrieval
references.venue: In International Conference on Learning Representations,
references.authors: Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou
references.title: BERT-of-theseus: Compressing BERT by progressive module replacing
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Sohee Yang Minjoon Seo
references.title: Is retriever merely an approximator of reader
references.venue: CoRR, abs/2010.10999,
references.authors: Zeynep Akkalyoncu Yilmaz Wei Yang Haotian Zhang Jimmy Lin
references.title: Cross-domain modeling of sentence-level evidence for document retrieval
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Chulhee Yun Srinadh Bhojanapalli Ankit Singh Rawat Sashank Reddi Sanjiv Kumar
references.title: Are transformers universal approximators of sequence-to-sequence functions
references.venue: In International Conference on Learning Representations,
references.authors: Jingtao Zhan Jiaxin Mao Yiqun Liu Min Zhang Shaoping Ma
references.title: RepBERT: Contextualized text embeddings for first-stage retrieval
references.venue: CoRR, abs/2006.15498,
references.authors: Jingtao Zhan Jiaxin Mao Yiqun Liu Jiafeng Guo Min Zhang Shaoping Ma
references.title: Optimizing dense retrieval model training with hard negatives
references.venue: CoRR, abs/2104.08051,
references.authors: Jingtao Zhan Jiaxin Mao Yiqun Liu Jiafeng Guo Min Zhang Shaoping Ma
references.title: Jointly optimizing query encoder and product quantization to improve retrieval performance
references.venue: In CIKM,
references.authors: Qu
references.title: Impact of negative mining on re-ranking performance We consider the value of adding additional negatives during DE model training
references.venue: Following Karpukhin et al
references.authors: Karpukhin
references.title: For example, in training the one-hot DE model on the triplet data, the retrieval MRR@10 is 0.281. When adding within-batch and uniform negatives, this increases to 0.314. Given that the re-ranking MRR@10 remains relatively unchanged (0.312 versus 0.310), this is further indication of the re-ranking and retrieval objectives not being perfectly aligned
references.title: Grey (born 21 April 1951 in Marefield, Leicestershire) is an English musician best known as t
references.authors: William Thomas Gray
references.title: Billy Gray was born on January 13, 1938 in Los Angeles, California, USA as Will
sections.text: Transformer-based models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query. There are two broad flavours of such models: cross-attention (CA) models, which learn a joint embedding for the query and document, and dual-encoder (DE) models, which learn separate embeddings for the query and document. Empirically, CA models are often more accurate, which has motivated a series of works seeking to bridge this gap. However, a fundamental question remains less explored: does this performance gap reflect a limitation in the capacity of DE models, or in the training of such models? In this paper, we study this question, with three contributions. First, we establish theoretically that with a sufficiently large encoder size, DE models can capture a broad class of scores without cross-attention. Second, we show empirically that on real-world problems, the gap between CA and DE models may be due to the latter overfitting to the training set. To mitigate this behaviour, we propose a distillation strategy that focuses on preserving the ordering amongst documents, and confirm its efficacy on benchmark neural re-ranking datasets.
sections.heading: 1 Introduction: transformer-based neural ranking
sections.text: Information retrieval (Mitra &Craswell, 2018) is the classic problem of identifying relevant documents for a given query. Typically, such retrieval is performed in a two-step manner (Matveeva et al., 2006): one uses an initial model to efficiently retrieve a candidate set of documents for a query, and then uses a second model to re-rank these candidates. The retrieval stage is generally implemented by a model with low inference cost, for which candidate document generation is tractable; if this candidate set is reasonably small, it is feasible to use more complex models in the re-ranking stage.
  Transformer-based models such as BERT (Devlin et al., 2019) have proven successful in both the retrieval and re-ranking stages. Such neural ranking models have two flavours: dual-encoder (DE) models (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Xiong et al., 2021; Luan et al., 2021), which learn separate (factorised) embeddings for the query and document; and cross-attention (CA) models (Nogueira & Cho, 2019; Dai & Callan, 2019; Yilmaz et al., 2019; MacAvaney et al., 2019; Gao et al., 2020), which learn a joint embedding for the query and document. Only DE models are applicable for retrieval, as they admit efficient nearest neighbour search (Guo et al., 2020; Johnson et al., 2021) to identify candidate documents. Both CA and DE models are applicable for re-ranking; however, empirically, CA models have proven more accurate (Hofstätter et al., 2020a).
  Does the re-ranking performance gap between CA and DE models reflect a limitation in the inherent capacity of the latter’s factorised representation, or in the training of DE models? These questions are of conceptual interest, but are also practically relevant: DE models allow for more efficient inference owing to the ability to pre-compute document embeddings, which enables fast similarity search when presented with a candidate query (Khattab & Zaharia, 2020; Hofstätter et al., 2020b). Several works have explored means of improving DE models, including by changing1 the scoring layer (Khattab & Zaharia, 2020; MacAvaney et al., 2020; Hofstätter et al., 2020b), and by distilling predictions from a CA model (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021). However, the root cause of the gap between CA and DE models remains elusive.
  In this paper, we study these questions, with the following contributions:
  1Unless otherwise stated, we use “DE” to mean a dual-encoder with dot-product scoring per Equation 2.
  (i) we establish theoretically that with a sufficiently large embedding dimension (and under mild assumptions), any continuous ground-truth scores can be modelled by generic DE models (Proposition 1), and in particular by sufficiently deep transformer-based DEmodels (Proposition 2). Thus, in principle, there is no fundamental restriction in using DE versus CAmodels for re-ranking.
  (ii) we show empirically (Figure 1) that on real-world problems, CA and DE models can achieve similar training performance, but DE models may do worse on test data due to overfitting (§3.2). Thus, the DE models may suffer due to poorer generalisation ability rather than capacity.
  (iii) to mitigate the above, we propose a distillation strategy focussing on mimicking the teacher’s ordering amongst documents (§4). This generalises the recent margin MSE loss (Hofstätter et al., 2020a) (§4.1), and justifies the utility of softmax cross-entropy based distillation for re-ranking (Proposition 3, §4.2). We confirm the efficacy of this strategy on the re-reranking benchmarks MSMARCO-Passage (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019).
  Overall, our results aim to shed light on the fundamental differences between CA and DE models, and give a simple yet effective distillation strategy to boost the performance of the latter.
sections.heading: 2 Background and notation
sections.text: Retrieval problems. Suppose we have query space Q and document space D. For a given query q ∈ Q, the goal of information retrieval is to identify a set of relevant documents Drel(q) ⊆ D (Mitra & Craswell, 2018). This is typically achieved by learning a scorer s : Q × D → R that predicts the relevance of a query and document, and simply selecting the top-k highest scoring documents for a given query. Typically, this scorer is itself implemented in two phases (Matveeva et al., 2006; Nogueira & Cho, 2019; Chang et al., 2020; Miech et al., 2021; Hofstätter et al., 2021). In the retrieval phase, one identifies an initial candidate set of documents via an initial scorer sret. In the re-ranking phase, one re-scores only these candidate documents via a distinct scorer srrk to obtain Drel(q). We may thus regard s : Q×D→ R as a suitable composition of both sret and srrk. To learn a scorer, suppose we have supervision {(qn,dn,yn)}Nn=1, where for each query qn we have a list of K associated documents dn . = (dn1, . . . , dnK) ∈ DK with ground-truth relevance labels yn . = (yn1, . . . , ynK) ∈ {0, 1}K . Typically, dn comprises a small subset of D (i.e., K |D|), since it is rarely feasible to collect relevance labels for all documents. Consider a parameterised class of scorers S .= {s(·, ·; θ) : Q × D → R | θ ∈ Θ}. For any given θ ∈ Θ, let sn(θ) . = (s(qn, dn1; θ), . . . , s(qn, dnK ; θ)) ∈ RK denote the vector of model scores on the provided
  documents. One may then learn a scorer s via minimising R(θ) .= 1N ∑ n∈[N ] `(yn, sn(θ)) for loss
  ` : RK × RK → R+, such as the mean square error `mse(yn, sn(θ)) . = ∑ k∈[K](ynk − snk(θ))2;
  and the softmax cross-entropy (assuming for simplicity 1>y = 1): for temperature τ > 0,
  `ce(yn, sn(θ)) . = − ∑ k∈[K] ynk · log pnk(θ) where pnk(θ) . = exp(τ−1 · snk(θ))∑ k′∈[K] exp(τ −1 · snk′(θ)) . (1)
  To instantiate either objective, one missing ingredient is the precise parameterisation of S. We now describe one possible parameterisation, given by a transformer model.
  Cross-attention and dual-encoders. Transformers (Vaswani et al., 2017) are sequence-to-sequence models with good empirical performance on language tasks. Suppose we have a sequence (x1, . . . , xL) of L tokens (e.g., words in a language), with X .= (x1,x2, . . . ,xL) ∈ RD×L being a corresponding sequence of token embeddings (e.g., word embeddings). A transformer encoder maps this sequence to Z .= (z1, z2, . . . , zL) ∈ RD×L , via a composition of attention and feedforward layers. The former is a mapping attn : RD×L → RD×L, which in its simplest (“single head”) form is attn(X) = X + WVXA, where A . = σ ( X>WQWKX/ √ D ) for softmax σ(·) and weights WQ,WK,WV.
  Intuitively, this embeds a token via a weighted sum of all tokens’ embeddings. The weighting is given by the attention matrix A ∈ RL×L, which is a learned similarity measure between pairs of tokens. BERT (Devlin et al., 2019) is a canonical instantiation of the transform encoder, coupled with effective pre-training tasks shown to improve downstream performance in small-sample regimes.
  Transformers can readily score (query, document) relevance: given a pair (q, d) ∈ Q×D of tokenised queries and documents, one may pass their concatenation through a transformer encoder T . The resulting embedding may be pooled (e.g., using the [CLS] token (Devlin et al., 2019)) to yield
  s(q, d) = w>pool(T (concat(q, d)))
  for weights w ∈ RD. Such cross-attention (CA) models apply attention layers on the queries and documents jointly, which intuitively allow for rich interactions.
  A distinct strategy is to separately embed queries and documents (e.g., via BERT encoders), and score them with their dot-product (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Zhan et al., 2020; Ma et al., 2021; Xiong et al., 2021; Luan et al., 2021; Qu et al., 2021; Zhan et al., 2021a):
  s(q, d) = pool(T (q))>pool(T (d)). (2)
  Such dual-encoder (DE) models have their roots in Siamese networks for image processing (Bromley et al., 1993), and have received recent interest in a range of language problems (Reimers & Gurevych, 2019; Gillick et al., 2019; Karpukhin et al., 2020). A salient feature of Equation 2 is that one can index the query and document embeddings. This implies that DE models can have significantly lower the inference cost compared to CA models, which makes them appealing to use for both the retrieval and re-ranking stages. Indeed, DE models are widely used for retrieval, owing to their amenability to fast approximate nearest neighbour search (Guo et al., 2020; Johnson et al., 2021). Unfortunately, for re-ranking, one typically observes a large gap between DE and CA performance (see §3.2).
  Cross-attention to dual-encoder distillation. One strategy to bridge the performance gap between CA and DE models is distillation (Ba & Caruana, 2014; Hinton et al., 2015). Suppose we have supervision {(qn,dn, tn,yn)}Nn=1, where tn ∈ RK are the scores from a “teacher” (e.g., CA) model. Then, we may minimise Rdist(θ) . = 1N ∑ n∈[N ] `(tn, sn(θ)), noting that we use the teacher supervision tn rather than “one-hot” labels yn; one may also combine the teacher and one-hot labels. Two canonical instantiations are logit matching (Ba & Caruana, 2014), which uses the the mean square error `mse(tn, sn(θ)) between the teacher and student scores, and probability matching (Hinton et al., 2015), which uses the softmax cross-entropy between the teacher and student softmax probabilities, i.e., `(tn, sn) = `ce(p teach n , sn) where pteachnk . = exp(τ −1·tnk)∑ k′∈[K] exp(τ −1·tnk′ ) for temperature τ > 0. Empirically, distillation has been shown to at least partially bridge the chasm between CA and DE performance (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021).
sections.heading: 3 The capacity of dual-encoder models
sections.text: We show that while DE models have sufficient capacity to represent complex score functions, in practice they may overfit and thus perform poorly on test data.
sections.heading: 3.1 How expressive are dual-encoders in theory?
sections.text: To study the expressivity of DE models, we ask: how well can such models approximate a given score function s∗ : Q×D→ R? This concerns the theoretical capacity of representing s∗, and side-steps the issue of estimating it from samples; we discuss the latter subsequently. For our purposes, the DE family comprises any model of the form s(q, d) = z(q)>w(d), where z : Q→ RD andw : D→ RD are (arbitrarily powerful) query- and document embedding functions. Under mild assumptions on D, we may show that DE models with sufficiently large embeddings can model a broad class of s∗. Proposition 1. Suppose D is a compact metric space, and s∗(q, ·) : D→ R is continuous for each q ∈ Q. Then, for any > 0, ∃ vectors zq,wd of at most countably infinite dimension such that
  (∀q ∈ Q, d ∈ D) |s∗(q, d)− z>q wd| ≤ .
  The proof relies on the classic Mercer’s theorem for kernel methods, and is provided in the Appendix. We make a few remarks on the above. First, note that by symmetry, we could equally swap the roles of Q and D above. Second, the assumption of D being a compact metric space is mild: we may do this, e.g., by identifying each document with a normalised Euclidean embedding. Third, while the above employs a kernel-based argument, we do not require that s∗ is symmetric; rather, s∗(q, ·) is simply a function to be approximated by an element in a reproducing kernel Hilbert space.
  Fourth, one can gain intuition by assuming Q and D are finite spaces. Here, we may associate with any s∗ : Q ×D → R a matrix S∗ ∈ R|Q|×|D|, where S∗q,d . = s∗(q, d). By the singular value decomposition, S∗ = ZW> for matricesZ ∈ R|Q|×r,W ∈ R|D|×r, where r .= rank(S∗). Thus, we may always write s∗(q, d) = S∗q,d = z(q)>w(d) by simply defining z(q) = Zq: and w(d) = W:d.
  Finally, the above does not restrict the function class used to represent either z orw. Restricting these to transformers, we may appeal to their universal approximation power (Yun et al., 2020): a sufficiently deep transformer encoder can approximate any continuous mapping T : RD×L → RD×L to arbitrary precision. Now suppose we represent queries Q ⊂ RD×L as embeddings of sequences of L tokenised elements, with a fixed [CLS] token at the first position. Then, any continuous z : Q→ RD can be approximated by reading the first token embedding of a suitable transformer T . Proposition 2. Fix some compact Q ⊂ RD×L. For any continuous z : Q → RD and ′ > 0, ∃ a transformer encoder T : RD×L → RD×L such that ‖z(Q)−T (Q):1‖22 ≤ ′ for almost everyQ ∈ Q.
  Remark. Luan et al. (2021) similarly analysed the expressive power of CA versus DE models, but through a slightly different lens: they formalised the expressiveness of DE score functions based on random projections, showing that the resulting embedding size may need to scale with the document length. This elegant analysis sheds light on one possible failure mode of DE models, i.e., scoring the relevance of queries and overly long documents. By contrast, we do not specifically restrict the form of the score function, which can be more reflective of transformer-based DE models; further, we shall now identify a distinct, fundamental reason for the gap between DE and CA model performance.
sections.heading: 3.2 How expressive are dual-encoders in practice?
sections.text: The previous section suggests that, at least in theory, DE models ought to be sufficiently expressive as to model a broad class of scores. However, this is at odds with the wealth of empirical results suggesting a non-trivial performance gap between DE and CA models (Lu et al., 2020; Izacard & Grave, 2020; Hofstätter et al., 2020a; Miech et al., 2021). What is the cause of this discrepancy? Note that the previous section considers only the capacity of DE models, i.e., the theoretical existence of a DE model capable of representing a given score function. A distinct question is whether it is feasible to learn such a model from a finite number of samples, and with a capacity restriction on the function class (e.g., a fixed embedding dimension and depth of transformer encoder).
  To study this, we perform a simple experiment on the benchmarkMSMARCO-Passage dataset (Nguyen et al., 2016), which concerns query to passage retrieval; we defer a detailed discussion of this dataset and our training protocol to §5. We focus on the re-ranking setting, wherein both the CA and DE models operate on the outputs of a BM25 retrieval model (Robertson & Zaragoza, 2009). We train a series of BERT-based CA and DE models on the standard (“small”) triplets training set, employing 6-layer BERT models from Turc et al. (2019) with varying embedding size. For each such model, we compute the mean reciprocal rank (MRR)@10 (Radev et al., 2002) on the provided train and dev set.
  Figure 1 compares the train and test MRR@10 for cross-attention (CA) and dual-encoder (DE) models. See Table 2 (Appendix) for a numeric summary. With increased embedding dimension, the DE model closely matches the performance of the CA on the training set. This is consistent with the previous section, which suggests suitably deep transformer-based DE models ought to model any ground truth scores; however, there is a sizable gap on the test set. This points to the DE models suffering from generalisation, rather than capacity. See Appendix C.1 for similar results with a 2-layer BERT model.
sections.heading: 3.3 What causes the dual-encoder generalisation gap?
sections.text: We now attempt to further understand the cause for the DE model’s generalisation gap. We perform a more fine-grained inspection of the CA versus DE model predictions by studying the predicted scores for positive and negative pairs. More precisely, for each query q in the test set, we consider the distribution of scores for positive and negative (query, document) pairs (q, d+) and (q, d−).
  Figure 2 shows that both CA and DE models possess clear modes for the positive and negative pairs. However, the CA model can be seen to not “waste” its capacity by unnecessarily modelling fine-grained distinctions amongst negative pairs; rather, it collapses most negatives to a small range of scores, and focusses intead on clearly separating the positives and negatives. By contrast, the DE model has far more diffuse negative scores. This results in a greater overlap in the positive and negative scores, implying the model has more difficulty making distinctions between these pairs. From the right plot, we also see that the DE model has smaller normalised margins (s(q, d+)− s(q, d−))/ρ between the positive and negative pairs, where ρ is the maximal score range.
  The DE score distribution for negatives is plausible: when updating a DE model for a given (q, d) pair, the factorised nature of the model implies a non-trivial influence on the scores for all other pairs (q, d′) and (q′, d) sharing either the query or document. Consequently, it is intuitively harder for the model to arrive at a solution that makes fine-grained distinctions. This can be verified qualitatively; the DE model is often seen to make errors on pairs which have only superficial relevance (Appendix C.4).
  We make a few remarks here. First, similar trends to the above holds on the training set; see Figure 5 (Appendix). Second, for visual clarity, in Figure 2 we apply a constant offset to each model’s scores to ensure they are mean zero. Figure 6 (Appendix) provides an uncentered plot, showing the DE model scores to have a strong baseline shift over the CA model.
  3.4 How can we mitigate the dual-encoder generalisation gap?
  Having identified the overfitting problem plaguing DE models, one may naturally ask how to mitigate this. One natural option is to apply standard regularisation strategies, such as dropout; however, our experiments reveal these to be largely ineffective in mitigating DE overfitting (see Appendix C.10). Alternately, the recent literature has identified two promising (complementary) strategies. The first is to modify the scoring function used to compute the final DE model score based on the query and document embeddings; i.e., we replace Equation 2 with s(q, d) = score(T (q), T (d)) for some scoring function score (see, e.g., MacAvaney et al., 2020; Khattab & Zaharia, 2020; Luan et al., 2021). Such strategies have proven effective; however, as these involve some overhead over dot-product scoring, it is natural to ask whether one can extract more gains out of the latter. The second strategy is
  to distill the predictions of the CA model into the DE model. By itself, this does not require changing the dot-product scoring in Equation 2. While distillation has been widely studied for classification settings, its use for re-ranking settings is less well-developed. We now study this issue more closely.
sections.heading: 4 Improving dual-encoders via distillation
sections.text: We now propose a family of distillation schemes to improve the performance of DE models.
sections.heading: 4.1 From logit to margin matching
sections.text: Recall that in distillation settings, each sample is the form (q,d,y, t), where for query q and documents d ∈ DK , we have ground-truth relevance labels y ∈ {0, 1}K and “teacher” model scores t ∈ RK (e.g., the outputs of a CA model). To fit scores s ∈ RK for a “student” model (e.g., a DE model), one can employ a suitable loss `(t, s). Per §2, the simplest choice is the logit matching or mean square error loss. Let us momentarily assume that for each query we have a single associated positive and negative document, with respective teacher scores t = (t1, t2) and student scores s = (s1, s2). Then, the loss simply minimises `mse(t, s) = ‖t− s‖22 = (t1 − s1)2 + (t2 − s2)2. While intuitive, this loss makes the implicit assumption that the range of scores for teacher and student are commensurate. When this is violated — as we empirially observe with the baseline shift between CA and DE model scores, per discussion in §3.2 and Figure 6 (Appendix) — the loss may be ineffective. To get around this, the margin MSE loss (Hofstätter et al., 2020a) matches margins, via
  `mmse(t, s) . = ((t1 − t2)− (s1 − s2))2. (3)
  Clearly, this loss is translation invariant, and thus is immune to baseline shifts in the scores. However, two issues remain. First, the loss is tied to triplet data with a single positive and negative document. Intuitively, it ought to be beneficial to leverage the teacher scores for multiple documents, with t, s ∈ RK for genericK. One naturalmeans of achieving this is `(t, s) = ∑ i∈P ∑ j∈N ((ti−tj)−(si−sj))2, where P,N denote the set of positive and negative documents, with P ∪N = [K]. Second, the above loss demands matching all documents’ margins exactly. However, this may be an ineffective use of model capacity: for re-ranking tasks, it is primarily important to ensure a clear separation between the positive and negative documents, and overcome the overlapping scores observed in Figure 2. To this end, suppose then that the negative documents are sorted in descending order of the teacher scores t = (t1, . . . , tK). We propose to extend Equation 3 via
  `m3se(t, s) = ∑ i∈P ((ti − tj∗)− (si − sj∗))2 + ∑ j∈N [sj − sj∗ ]2+, (4)
  where j∗ ∈ N denotes the index of the negative document with highest teacher score. This multimargin MSE (M3SE) loss only attempts to match the margins for the highest scoring negative j∗; for all other negatives j 6= j∗, we simply encourage them to have lower score than j∗. While one may design other intuitively plausible losses, an appealing property of Equation 4 is its relation to the recently proposed RankDistil framework (Reddi et al., 2021), as shall be discussed in §4.3.
sections.heading: 4.2 From margin to probability matching
sections.text: The above extended the logit matching distillation loss for re-ranking settings, with care to account for differing score ranges for teacher and student. Per §2, the other classic distillation loss is the softmax cross-entropy (Equation 1), which has seen prior use for dual-encoder distillation (Lu et al., 2020; Izacard & Grave, 2020; Yang & Seo, 2020; Hofstätter et al., 2020a; Miech et al., 2021). In fact, we now show that the softmax cross-entropy generalises the margin MSE loss in a precise sense. Proposition 3. Fix any teacher and student scores t, s ∈ R2. Let `sce(t, s; τ) denote the softmax cross-entropy loss (Equation 1) with temperature τ > 0, and `mmse the margin MSE loss (Equation 3). Then, for temperature scaled sigmoid function σ(z; τ) .= (1 + exp(−τ−1 · z))−1, we have
  `sce(t, s; τ) = KLbin(σ(t1 − t2; τ) ‖σ(s1 − s2; τ))
  lim τ→+∞
  [ τ2 · ∂
  ∂si `sce(t, s; τ)
  ] = 1
  8 · ∂ ∂si `mmse(t, s) for i ∈ {1, 2}.
  We make a few remarks. First, `sce swaps the square loss for a binary KL divergence KLbin(a‖b) . = −a · log ba + (1− a) · log 1−b 1−a in Equation 3. Unlike the square loss, the softmax cross-entropy is Lipschitz, which is typically favourable for generalisation (Luxburg & Bousquet, 2004). Second, the softmax and margin MSE loss derivatives with respect to the student logits converge in the high temperature limit. This gives a precise sense in which the softmax cross-entropy is a smooth approximation to the margin MSE. Intuitively, this is plausible: the former is similarly invariant to constant translations to the scores, thus allowing for differing ranges in the teacher and student.
  Third, our argument is subtly different from Hinton et al. (2015), which establishes convergence of the softmax and logit matching derivatives: crucially, the latter only holds under the assumption that the teacher and student logits are centered. We reiterate that this assumption typically does not hold for CA and DE models, per §3.2 and Figure 6 (Appendix). (While it is benign to translate all teacher logits by a fixed constant, this may not guarantee the assumption as the average score 1K ∑ i∈[K] ti is typically not constant across samples.) By contrast, Proposition 3 makes no such assumption.
  Finally, Proposition 3 assumes the supervision comprises a a single positive and negative document, to facilitate comparison with Equation 3. The case ofK > 2 documents with teacher and student scores t, s ∈ RK poses no conceptual issue: we may seamlessly minimise the softmax cross-entropy between the two vectors. Since typically K |D|, this does not correspond to matching the full teacher distribution over all possible “classes”. Despite this distinction to the standard use of distillation for classification problems, one may justify the loss as follows. For positive (negative) documents P (Q),
  `sce(t, s) ∝ 1
  τ ∑ k∈[K] eτ −1·tk∑ i∈P e τ−1·ti + ∑ j∈N e τ−1·tj · log ∑ i∈P eτ −1·(si−sk) + ∑ j∈N eτ −1·(sj−sk)  . This approaches max {maxi∈P [si − sk∗ ]+,maxj∈N [sj − sk∗ ]+} as τ → +∞, where k∗ is the highest scoring document under the teacher. The loss thus encourages the student to preserve the teacher’s highest scoring document, which is useful for re-ranking. Compared to Equation 4, this does not explicitly delineate the highest scoring negative from other negatives; however, empirically, we shall see that the departure from square loss endows the loss with comparable efficacy.
sections.heading: 4.3 Discussion and extensions
sections.text: Connection to RankDistil. Equation 4 can be related to the recently proposed RankDistil framework by Reddi et al. (2021). Here, the goal is to design distillation losses to match the teacher and student ranking over labels, particularly for the top-k elements. A canonical instantiation is the binary RankDistil formulation (RankDistil-B), `rankB(t, s) = ∑ i∈P (ti − si)2 + ∑ j∈N [sj − γ0]2+. Intuitively, this seeks to match the teacher and student score on positive documents, while ensuring all negative documents fall below some threshold γ0. The relation between this loss and Equation 4 is analogous to that between logit matching and margin MSE: Equation 4 matches margins rather than raw scores, which are thus suitable for settings of differing ranges of the teacher and student scores.
  Distillation for retrieval phase. Our focus thus far has been on bridging the observed gap between the CA and DE models for re-ranking. However, the losses presented above are equally applicable during the retrieval phase, where they can similarly be expected to improve DE model performance. These losses are complementary to proposals from several recent works improving the retrieval performance of dual-encoder models, including using suitable hard negative mining for training (Xiong et al., 2021; Zhan et al., 2021a), and quantisation (Zhan et al., 2021b).
sections.heading: 5 Experimental results
sections.text: We now present experiments that demonstrate the viability of the proposed distillation scheme in §4. We show that this scheme yields better performance compared to several baselines, helping reduce the gap between CA and DE models by mitigating the overfitting issue identified in §3.2.
  Per §4.3, we reiterate that our primary goal is to explore the feasibility of bridging the gap between CA and DE models for re-ranking. This is distinct from the more common use of DE models in the literature for retrieval (Chang et al., 2020; Xiong et al., 2021; Zhan et al., 2021a). While we do not make special effort to incorporate tricks from that literature that are beneficial for retrieval
  MSMARCO re-rank MSMARCO retrieval TREC DL19 re-rank NQ re-rank Model MRR nDCG MRR nDCG MRR nDCG MRR nDCG
  Baselines: one-hot BM25 (Robertson & Zaragoza, 2009) 0.194† 0.241† 0.194† 0.241† 0.689† 0.501† — — ANCE (Xiong et al., 2021) — — 0.330† — — 0.677† — — Cross-attention BERT (12-layer) 0.370 0.430 N/A N/A 0.829 0.749 0.746 0.673 Dual-encoder BERT (6-layer) 0.310 0.360 0.281 0.331 0.834 0.677 0.676 0.601
  highlight the best performing DE based model. Distilling the dual-encoder with our proposed techniques significantly improves performance over one-hot training and existing distillation techniques. Results marked † are quoted from the corresponding reference, “N/A” are not applicable (e.g., the cross-attention model is not feasible to apply for retrieval), and “—” are not available from the reference. ♦ See Appendix C.11 for analysis of potential label noise influencing the results. ? See text for discussion.
  (e.g., using hard negatives), we do demonstrate that our proposed distillation techniques can improve performance for this stage over simply using one-hot training labels.
  Datasets. We present results on MSMARCO-Passage (Nguyen et al., 2016) and Natural Questions (NQ) (Kwiatkowski et al., 2019). For MSMARCO, we report results on the standard dev set, and the TREC DL19 test set (Craswell et al., 2020). MSMARCO comprises (query, passage) records and their relevance labels from a human rater; for each query, the associated passages are retrieved from a BM25 model. The training set is canonically represented in triplet form, comprising a query, and a single positive and negative passage. NQ comprises user questions andWikipedia passages potentially containing their answers; we use a processed version from Karpukhin et al. (2020) (cf. Appendix B).
  Models. We use transformer encoders initialised with the standard pre-trained BERT model checkpoints. Following Hofstätter et al. (2020a), we use BERT-Base for the CA model, and a 6-layer BERT model (Turc et al., 2019) with embedding size 768 for all DE models. For the DE models, we tie the query and document encoder parameters. For distillation, we use the CA model as the “teacher” and the DE model as the “student”. On MSMARCO, for ease of comparison, we use the “T1” CA model annotations from Hofstätter et al. (2020a), which achieves similar re-re-ranking performance as our independently trained CA model. See Appendix B for further training details.
  Metrics. We report the MRR@10 and nDCG@10 for all methods. Note that we are primarily interested in re-ranking tasks, wherein the test set comprises documents retrieved by a BM25 baseline, and our models are used to re-rank these candidates. For MSMARCO, we additionally consider full retrieval performance obtained by scoring all (8.8M) passages, but reiterate that maximising performance on this metric requires additional negative mining that we do not explore.
  Baselines. We consider methods that leverage either the observed (“one-hot”) training labels on the triplet data, or the predictions from the cross-attention (“teacher”) model on the triplet data. For the former, we employ softmax cross-entropy against the using a cross-attention and dual-encoder model. For the latter, we employ the logit MSE loss (Ba & Caruana, 2014), margin MSE loss (Hofstätter et al., 2020a), and the binary version of RankDistil (RankDistil-B) (Reddi et al., 2021).
  These are compared against the proposed M3SE (Equation 4), and the softmax cross-entropy (Equation 1). The latter methods handle of an arbitrary number of documentsK in the supervision (i.e., they are not restricted to triplet data). We thus aggregate the triplet data by query, and retain the top-20 passages with highest teacher model score. We study the sensitivity toK in Appendix C.7.
sections.heading: 5.1 Results and discussion
sections.text: Table 1 summarises the results of all methods. We discuss a few salient findings.
  Baseline performance. Comparing the performance of CA and DE models in Table 1, we make two initial observations. First, there is a sizable performance gap between the models when using one-hot labels. Second, in line with Hofstätter et al. (2020a), naïve logit matching via the MSE underperforms, due to the vastly different scales of the two models; this is however assuaged by the margin MSE loss.
  Efficacy of distillation. Both the M3SE and the softmax cross-entropy losses significantly outperform existing methods on the MSMARCO and NQ re-reranking tasks, and shrinks the gap between the DE and CA models. On the TREC DL19 test set, our proposed methods appear to fall short of the margin MSE MRR@10; a closer inspection of the apparent loss cases (Appendix C.11) reveals these are unanimously the results of false negatives in the provided labels.
  Figure 1 shows that M3SE distillation nearly closes the gap between CA and DE models with sufficiently large embedding dimension. Intriguingly, the test gains comes with some sacrifice of training MRR for the highest embedding size. This is consistent with findings in the classical distillation literature, where distillation can harm training accuracy but improve test accuracy (Cho & Hariharan, 2019). To further study this, Figure 4 (Appendix) shows the learning curve on the MSMARCO train and test set, which demonstrates a consistent gap in test set performance as training progresses. Distillation yields a training MRR that closely matches that of the DE model for the first 100K training steps; subsequently, however, the training MRR under distillation begins to saturate, while that of the DE model with one-hot labels keeps increasing. Intuitively, this suggests that the DE model starts exploring a poorly generalising part of the parameter space beyond this point.
  As a final illustration, Figure 2 (right) shows that softmax cross-entropy distillation makes the DE scores better behaved: it shifts the margins of the DE model to be closer to the the CA model.
  Beyond the dot-product: generic scoring functions. We have thus far focussed on results using the standard dot-product based scoring for the DE models, per Equation 2. It is of interest as to what impact more complex scoring functions, per §3.4, have on final model performance. To study this, we consider the ColBERT model (Khattab & Zaharia, 2020), which computes the average of the maximum query-document token similarities. Consistent with Khattab & Zaharia (2020), using this model with one-hot labels by itself improves performance significantly over the dot-product.
  When further combined with distillation, the performance exceeds that of the CA teacher, reaching an MRR@10 of 0.376 for the MSMARCO re-ranking task under the softmax cross-entropy loss. (Full results in Appendix, Table 7.) This further highlights that factorised representations need not imply a loss in performance, provided they are suitably trained. We defer to future work a similar analysis of the conceptually similar multi-vector scoring layer of Luan et al. (2021).
  Full-retrieval performance. Thus far, we have focussed on the re-ranking performance of all models, wherein the predictions from a BM25 retriever are provided as input. This allows for the comparison of DE and CA models on an equal footing. One may however naturally wonder how the proposed methods fare in the retrieval phase, wherein the models score all possible passages. Table 1 also summarises the results in this setting on MSMARCO. We again see that the proposed distillation techniques offer strong gains over standard one-hot training of the DE model, as well as existing distillation techniques. We remark also that the poor retrieval performance of the MSE loss is a result of the model strongly overfitting to the teacher scores on the provided documents.
  Additional results. In the Appendix, we present several additional results and ablations, including the sensitivity of performance to the number of documentsK in the distilled supervision (Table 6); the effectiveness of label smoothing as opposed to full distillation (Appendix C.9); and the impact of other regularisation strategies aimed at minimising overfitting (Appendix C.10).
sections.heading: 6 Conclusion and future work
sections.text: Several avenues for future work remain open. For example, in the literature on distilling between cross-attention models, a common strategy is to match more than simply output logits (Sun et al., 2019; Turc et al., 2019; Sanh et al., 2019; Xu et al., 2020; Sun et al., 2020; Jiao et al., 2020; Wang et al., 2020). Are such strategies useful when distilling to a dual encoder? Further, our distillation strategy assumes access to scores from a cross-attention model. Can one do away with this altogether, e.g., using self-distillation and related ideas (Furlanello et al., 2018)? Finally, the efficacy of the proposed losses when combined with recent advances for retrieval (Xiong et al., 2021) is of interest.
  Ethics Statement
  This work concerns the mathematical and empirical analysis of techniques for neural re-ranking, a particular problem encountered in information retrieval. We do not foresee the former having undue societal effects. The latter does not explicitly consider issues of fairness in retrieval, which is an important yet under-studied dimension. We do not foresee our distillation techniques as unduly amplifying biases in existing algorithms; nonetheless, we consider a more careful study of the topic of fairness in retrieval an important avenue for future work.
sections.heading: A Proofs
sections.text: Proof of Proposition 1. For any q ∈ Q, let us write sq . = s(q, ·) : D→ R. Pick a reproducing kernel Hilbert space H over D with associated inner-product 〈·, ·〉H. Suppose further H has associated kernel k : D×D→ R that is continuous, and universal (Micchelli et al., 2006). Then,H is dense in the space of continuous functions (Sriperumbudur et al., 2011). Recall that by assumption sq is continuous. Consequently, each sq can be approximated to arbitrary precision (with respect to the `∞ metric) by some s∗q ∈ H. Since s∗q ∈ H, by the reproducing property of an RKHS,
  (∀d ∈ D) s∗(q, d) = 〈s∗q , kd〉H,
  where in an abuse of notation kd . = k(d, ·) ∈ H. Observe that we may interpret s∗q as an “embedding” for the query q, and kd a “weight vector” for the document d.
  It may appear at this stage that we are done; however, each of s∗q and kd may be uncountably infinite dimensional objects, and the inner product in H may not be the standard dot-product. Nonetheless, under the stated assumptions (in particular, compactness of D), we may appeal to a Mercer representation of the kernel, following Steinwart & Christmann (2008, Theorem 4.51). In particular, we may construct a Mercer feature map Φ: D→ `2 forH, such that
  (∀h ∈ H) (∃(an)∞n=1 ∈ `2)h(d) = ∞∑ n=1 an · Φn(d).
  Further, the inner product between any h1, h2 ∈ H with “Mercer coefficients” (an)∞n=1, (bn)∞n=1 is expressible as
  〈h1, h2〉H = ∞∑ n=1 an · bn,
  i.e., a familiar dot-product. In our context, write the Mercer coefficients for s∗q as (zqn)∞n=1, and for kd as (wdn)∞n=1. Thus, we may write
  〈s∗q , kd〉H = z>q wd,
  where the vectors zq,wd have at most countably infinite dimension.
  Proof of Proposition 2. Any continuous z : RD×L → RD can be trivially associated with z̃ : RD×L → RD×L, where each column z̃:i = z. Such a mapping preserves continuity. By Yun et al. (2020, Theorem 2), we may thus find a transformer encoder T : RD×L → RD×L such that ∫ Q ‖z̃(Q) − T(Q)‖22 dQ ≤ ′. Since the integrad is nonnegative, we must also have ‖z̃(Q)−T(Q)‖22 dQ ≤ ′ for any givenQ. Consequently, we must also have ‖z̃(Q):1−T(Q):1‖22 ≤ ′ almost surely, which by definition implies ‖z(Q)−T(Q):1‖22 ≤ ′,
  Proof of Proposition 3. For temperature τ > 0, simple algebra reveals Equation 1 to be
  `sce(t, s) = − ∑
  y∈{1,2}
  exp(τ−1 · ty) exp(τ−1 · t1) + exp(τ−1 · t2) · log exp(τ −1 · sy) exp(τ−1 · s1) + exp(τ−1 · s2)
  = ∑
  z∈{±1}
  1
  1 + ez·τ−1·(t2−t1) · log(1 + ez·τ −1·(s2−s1))
  = − ∑
  z∈{±1}
  σ(z · (t1 − t2); τ) · log(σ(z · (s1 − s2); τ)).
  This is equivalent to KLbin(σ(t1 − t2; τ) ‖σ(s1 − s2; τ)), i.e., swapping out the square loss for the KL divergence in Equation 3. In fact, one can make a stronger connection: observe that
  ∂
  ∂s1 `mmse(t, s) = 2 · ((s1 − s2)− (t1 − t2))
  ∂
  ∂s1 `sce(t, s) = − ∑ z∈{±1} σ(z · (t1 − t2); τ) · ∂ ∂s1 log(σ(z · (s1 − s2); τ)
  = τ−1 ·
  ( eτ −1·s1
  eτ−1·s1 + eτ−1·s2 − e
  τ−1·t1
  eτ−1·t1 + eτ−1·t2
  )
  = τ−1 · (
  1 1 + eτ−1·(s2−s1) − 1 1 + eτ−1·(t2−t1) ) ∼ τ−2 · ( s1 − s2
  4 − t1 − t2 4
  ) ,
  as τ → +∞, where the last line follows from the Taylor series approximation
  1
  1 + exp(−z) =
  1 2 + z 4 + O(z2)
  as z → 0.
sections.heading: B Experiment hyperparameters
sections.text: For all models, at the output layer we apply dropout at rate 0.1 and layer normalisation. We use a sequence length of 30 for queries, and 200 for passages. We optimise all methods for a maximum of 3 × 105 steps using Adam with weight decay, with a batch size of 128 and a learning rate of 2.8× 10−5 (i.e., a 4× scaling of the choices in Hofstätter et al. (2020a)). Following Hofstätter et al. (2020a), we perform early stopping based on the nDCG@10 metric.
  For the Natural Questions dataset, we use the processed version used in Karpukhin et al. (2020), that contains questions, positive passages with a correct answer to each question, and a corpus of all Wikipedia passages. To train DE, CA, and multi-negative DIST models, we use 19 BM-25 hard-negative passages for each question along with a positive passage. For single-negative DIST models we use a single BM-25 negative from the same collection. To calculate MRR@10 and nDCG@10 metrics, we use the queries in the dev set with 200 passages containing positives, 100 BM-25 hard-negatives and up to 100 random negatives.
sections.heading: C Additional experiments
sections.text: C.1 CA versus DE models: effect of base architecture
  Figure 3 presents an an analogue of Figure 1, but using instead a 2-layer BERT model. As before, we initialise using the pre-trained models developed in Turc et al. (2019). The same general trends hold: for sufficiently large embedding dimension, the DE model closely matches the performance of the CA on the training set; however, there is a sizable gap in the test set performance. Table 2 presents a numeric summary of the train and test performance for CA and DE models, under different choices of base BERT-model architecture (in terms of number of layers, as well as the final embedding dimension).
  C.2 CA versus DE models: evolution of train and test performance
  The above results summarise the results at the completion of training. For a finer-grained understanding of how performance evolves during training, Figure 4 presents the learning curves for CA, DE, and distilled DE (DIST) models on the train and test set. Here, we use the small-bert-6-768 architecture for all models. We observe that the CA and DE models initially have a non-trivial gap in performance, but this shrinks over time; this further points to the initialisation of the CA models being more favourable for generalisation. Note also that beyond a certain point, both models continually improve their training performance, but are completely neutral in terms of test performance. Interestingly, distillation largely tracks the DE model performance, but then worsens on the training set while improving on the test set. This suggests that distillation helps explore a better part of the search space.
  C.3 CA versus DE models: comparison of score distributions
  Figure 5 compares the score distributions for the CA and DE models on the training set. We observe the same general trends as the test set (Figure 2). Figure 6 further presents the test score distributions without zero-centering. We see that the DE model exhibits a strong baseline shift, wherein the scores are of the order of ∼ 700. Figure 7 studies the score distributions for the CA and DE models after varying numbers of training steps. At initialisation, as expected, both models have essentially overlapping distributions for the positives and negatives. However, here too, we observe that the DE model scores operate on a much wider range than the DE model. The middle row continues this study by looking at behaviour after 10, 000 steps of training, which is a small fraction of the 300, 000 steps used for training of all models. Here, we start to observe a clearer separation between the positive and negative scores for both models. Interestingly, the CA model already sees a sharp peak on the negatives, unlike the DE model. The bottom row shows the test set score distributions of the final trained models, to complement Figure 2 from the body. We see the same general trend as the training set: the CA model is seen to more confidently distinguish positives from negatives, and operate at a narrower range of scores.
  C.4 CA versus DE models: qualitative analysis
  Table 3 provides a sample of (query, passage) pairs from MSMARCO-Passage dev set, where there is a large discrepancy between the CA and DE model scores. Specifically, we consider pairs that the CA model scores low, but the DE model scores high. Interestingly, these pairs typically involve strong token overlap between the query and passage — indicating a certain degree of topicality — but are fact genuine negative pairs. This reflects that the DE model may be unable to capture certain fine-grained distinctions.
  Query Passage
  effects of yeast on body The Side Effects of Chemotherapy on the Body. Chemotherapy drugs are powerful enough to kill rapidly growing cancer cells, but they also can harm perfectly healthy cells, causing side effects throughout the body.
  age not to take off shoes at airline security Airline Identification Requirements. Airlines do not typically require identification from passengers under the age of 18, but there are exceptions. Children under the age of 2 may ride on a parent’s lap without purchasing a ticket, but the airline will require identification, such as a birth certificate, to prove the child’s date of birth . . .
  actress who plays alice on the magicians Then portrayed as the animated Alice’s real life counterpart by actress Mia Wasikowska as a more mature, grown up Alice in Disney’s 2010 semi-sequel, live action/CGI film Alice and Wonderland Directed by Tim Burton. In the Broadway musical version, she will be played by Taylor Louderman . . .
  can you absorb metals from plants Answer 1: Photosynthesis is the ability of plants to absorb the energy of light, and convert it into energy for the plant. To do this, plants have pigment molecules which absorb the energy of light very well. The pigment responsible for most light-harvesting by plants is chlorophyll, a green pigment.The green color indicates that it is absorbing all the non-green light– the blues ( 425-450 nm), the reds and yellows (600-700 nm).he pigment responsible for most light-harvesting by plants is chlor . . .
  Table 3: Sample of (query, passage) pairs from MSMARCO-Passage dev set with largest discrepancy between the CA and DE model scores. In most of these cases, the passage is not relevant to the query; however, there is a high degree of token overlap between the two, indicating superficial similarity.
  C.5 Impact of distillation: effect on score distributions
  We have previously seen in Figure 7 that our proposed M3SE distillation strategy can improve the distribution of scores for positives and negatives. Figure 8 shows the impact that such distillation has on the margins between positive and negatives, compared to standard training on one-hot labels. The distilled model is seen to more confidently distinguish positives from negatives, as evidenced by the margin distribution having more mass on larger values.
  C.6 Impact of distillation: effect of student model architecture
  The preceding distillation results have all employed a small-bert-6-768 model. It is of interest how sensitive the results are to this choice. To study this, we report in Table 4 the re-ranking results on MSMARCO-Passage when using a small-bert-6-128 model. We observe largely consistent trends as before: distillation using our proposed losses can close the gap between CA and DE model performance. Table 5 repeats this for a small-bert-2-768 model.
  C.7 Impact of distillation: effect of number of documents in supervision
  The results reported in the body employed a total of 20 documents per query for the softmax CE and M3SE methods. As this involves significantly more information than the triplet data used, e.g., in training the margin MSE loss, it is of interest to tease apart how much of the gains from the method come from this increased supervision, versus the loss itself. Table 6 studies the effect of varying the number of documents. We see that there are diminishing returns beyond a certain point: using 100 documents yields qualitatively similar performance to using 20 documents. At the same time, there is a marked difference in performance when using 2 documents versus 10.
  C.8 Results with ColBERT scorer
  Table 7 presents results using the ColBERT scorer. We find that with distillation using the softmax CE loss (Equation 1), we can exceed the performance of the cross-attention teacher model for the re-ranking task. This further indicates the viability of dual-encoder models for neural ranking.
  MSMARCO rerank TREC DL19 MRR@10 nDCG@10 MRR@10 nDCG@10
  Baselines: one-hot Cross-attention BERT 0.370 0.430 0.829 0.749 Dual-encoder ColBERT 0.356 0.416 0.839 0.703
  C.9 Beyond distillation? Effect of label smoothing
  A special case of distillation is label smoothing (Szegedy et al., 2016), which can be understood as using a teacher that predicts the uniform distribution over all labels, and combining the result
  Smoothing α MRR@10 train MRR@10 test
  with the one-hot label. Specifically, this involves mixing the observed one-hot labels with a uniform distribution over all labels, with the mixing controlled by weight α ∈ [0, 1); α = 0 corresponds to standard one-hot training. Label smoothing has also proven effective as a means of preventing harmful overfitting, and thus improving generalisation (Müller et al., 2019; Lukasik et al., 2020) in classification settings.
  For the MSMARCO-Passage dataset, we studied the effect of varying degrees of label smoothing for DE models, using the softmax CE loss on triplet data. We use a small-bert-6-768 model as before; for computational ease, we report results after 300, 000 steps of training, at which point all models see stable test error. Table 8 reveals that, surprisingly, smoothing has a minimal effect on test performance, even at high smoothing levels. This is despite smoothing reducing the training performance at higher values, which is expected given its equivalence to injecting symmetric label noise to each sample. Further study of the viability of smoothing for retrieval problems would be of interest.
  C.10 Beyond distillation? Effect of alternate regularisation strategies
  Distillation provides one means of preventing the DE model from overfitting to the training set. One may of course conceive of other plausible strategies, such as:
  • increasing the dropout rate in the final (embedding) layer of the transformer. The experiments in the body employ 10% dropout (following prior work, e.g., Hofstätter et al. (2020a)), which improves performance considerably over having no dropout; it is thus of interest whether there are benefits to further increasing this.
  • performing token dropout at the input layer. Intuitively, such corruption prevents the model from overly relying on individual tokens, thus preventing spurious token matching (as present in some cases in Table 3).
  • adding a masked language model (Devlin et al., 2019) loss for the student. This loss is a core component of BERT, and has thus proven to be effective in learning generically useful sequence embeddings. Intuitively, adding such a self-supervised objective to training can plausibly improve the model’s robustness.
  • modifying the base loss from square or log-loss to the focal loss (Lin et al., 2017). This loss has proven effective in vision tasks involving high class-imbalance. In our setting, as there are typically only a handful of relevant documents, but several millions of irrelevant documents, it is of interest whether such a loss can positively impact results.
  Table 9 summarises results for each of these strategies on the MSMARCO Passage re-ranking task. Here, we focus on their effect on the standard (one-hot) training of a dot-product based DE model on triplet data. Unfortunately, we find that none of these strategies significantly improve the generalization performance of the baseline model. Thus, mitigating the overfitting observed in Figure 1 requires more effort than appealing to standard strategies effective in classification tasks.
  C.11 Analysis of TREC DL19 predictions
  Table 1 suggests a notable gap in performance for our proposed RankDistil variant and the margin MSE loss (Hofstätter et al., 2020a). Here, we take a closer look at the loss cases for our method. The
  TREC DL19 data comprises a total of 43 queries, each with between∼ 100 to 500 rated passages. Of these queries, the MRR@10 of the DE models trained with margin MSE and softmax CE agree on 10. On the queries with disagreement, 6 cases favour margin MSE, and 4 cases favour RankDistil-SM.
  For the 6 queries where RankDistil-SM ostensibly underperforms, we inspect the top-5 scoring passages for both margin MSE and softmax CE in Table 10 and 11. The cells shaded blue correspond to passages rated positive. Several other passages are however seen to be equally valid answers to the source query. Indeed, we submit that in all cases, the predictions from RankDistil are of at least the same quality as the margin MSE.
  C.12 Impact of negative mining on re-ranking performance
  We consider the value of adding additional negatives during DE model training. Following Karpukhin et al. (2020); Qu et al. (2021), we consider the use of within-batch (also referred to as in-batch) and uniform negatives. For the latter, in each training minibatch, we draw a sample of Buni documents drawn uniformly at random from the entire set of documents; each such document is treated as a negative document in the loss. For the former, from each training minibatch {(qi,di,yi)}Bi=1, we compute the set of all observed documents Dobs = ∪Bi=1 ∪Kk=1 {dik}. For each sample (qi,di,yi), we then use each element in Dobs − ∪Kk=1{dik} as a negative document in the loss. Table 12 summarises the results for the MSMARCO re-ranking task. We find that adding these negatives has a small gain for the one-hot model performance. However, for the distilled objectives, there is limited gain (and sometimes even a degradation) in performance. More study of this issue is warranted, but one hypothesis is that the re-ranking task is inherently concerned with ranking the outputs of a BM25 model. These outputs are precisely used to construct the training data used for distillation. It is thus possible that adding additional negatives — which are unlikely to appear as candidates for re-ranking — do not bring significantly useful information.
  We emphasise here also that the results reported are for the re-ranking, as opposed to retrieval task. For the latter, adding within-batch and uniform negatives is intuitively and empirically valuable, as noted in Karpukhin et al. (2020); Qu et al. (2021). For example, in training the one-hot DE model on the triplet data, the retrieval MRR@10 is 0.281. When adding within-batch and uniform negatives, this increases to 0.314. Given that the re-ranking MRR@10 remains relatively unchanged (0.312 versus 0.310), this is further indication of the re-ranking and retrieval objectives not being perfectly aligned.
  Query Top scoring passages Margin MSE RankDistil-SM
  who is robert gray Who is Henry Gray? Henry Gray is an African-American blues piano player and singer. He has been play. . . Robert Grey (born 21 April 1951 in Marefield, Leicestershire) is an English musician best known as t. . .
  Robert Gray was the Democratic candidate for governor of Mississippi in the 2015 elections. Gray won. . . Who is Henry Gray? Henry Gray is an African-American blues piano player and singer. He has been play. . .
  Kenneth Gray (I) Kenneth Gray is an actor, known for Love, Lies and Murder (1991), Retribution (1987. . . Robert Gray was the Democratic candidate for governor of Mississippi in the 2015 elections. Gray won. . .
  Kenneth Gray (I) Actor. Kenneth Gray is an actor, known for Love, Lies and Murder (1991), Retributio. . . Robert Gray. A surprise came on the Democratic side in the race for Mississippi Governor. Robert Gra. . .
  Robert Grey (born 21 April 1951 in Marefield, Leicestershire) is an English musician best known as t. . . William Thomas Gray. Billy Gray was born on January 13, 1938 in Los Angeles, California, USA as Will. . .
title: In defense of dual-encoders for neural ranking