Transformer based re-ranking models can achieve high search relevance through context-aware soft matching of query tokens with document tokens. To alleviate runtime complexity of such inference, previous work has adopted a late interaction architecture with pre-computed contextual token representations at the cost of a large online storage. This paper proposes contextual quantization of token embeddings by decoupling document-specific and document-independent ranking contributions during codebook-based compression. This allows effective online decompression and embedding composition for better search relevance. This paper presents an evaluation of the above compact token representation model in terms of relevance and space efficiency. Modern search engines for text documents typically employ multi-stage ranking. The first retrieval stage extracts top candidate documents matching a query from a large search index with a simple ranking method. The second stage or a later stage uses a more complex machine learning algorithm to re-rank top results thoroughly. Recently neural re-ranking techniques from transformer-based architectures have achieved impressive relevance scores for top k document re-ranking, such as MacAvaney et al. (2019). However, using a transformer-based model to rank or re-rank is extremely expensive during the online inference (Lin et al., 2020). Various efforts have been made to reduce its computational complexity (e.g. Gao et al. (2020)). A noticeable success in time efficiency improvement is accomplished in ColBERT (Khattab and Zaharia, 2020) which conducts late interaction of query terms and document terms during runtime inference so that token embeddings for documents can be pre-computed. Using ColBERT re-ranking after a sparse retrieval model called DeepImpact (Mallia et al., 2021) can further enhance relevance. Similarly BECR (Yang et al., 2022), CEDR-KNRM (MacAvaney et al., 2019), and PreTTR (MacAvaney et al., 2020) have also adopted the late interaction architecture in their efficient transformer based re-ranking schemes. While the above work delivers good search relevance with late interaction, their improvement in time efficiency has come at the cost of a large storage space in hosting token-based precomputed document embeddings. For example, for the MS MARCO document corpus, the footprint of embedding vectors in ColBERT takes up to 1.6TB and hosting them in a disk incurs substantial time cost when many embeddings are fetched for re-ranking. It is highly desirable to reduce embedding footprints and host them in memory as much as possible for fast and high-throughput access and for I/O latency and contention avoidance, especially when an online re-ranking server is required to efficiently process many queries simultaneously. The contribution of this paper is to propose a compact representation for contextual token embeddings of documents called Contextual Quantization (CQ). Specifically, we adopt codebook-based quantization to compress embeddings while explicitly decoupling the ranking contributions of document specific and document-independent information in contextual embeddings. These ranking contributions are recovered with weighted composition after quantization decoding during online inference. Our CQ scheme includes a neural network model that jointly learns context-aware decomposition and quantization with an objective to preserve correct ranking scores and order margins. Our evaluation shows that CQ can effectively reduce the storage space of contextual representation by about 14 times for the tested datasets with insignificant online embedding recovery overhead and a small relevance degradation for re-ranking passages or documents. The problem of neural text document re-ranking is defined as follows. Given a query with multiple terms and a set of candidate documents, rank these documents mainly based on their embeddings and query-document similarity. With a BERT-based re-ranking algorithm, typically a term is represented by a token, and thus in this paper, word term is used interchangeably with token. This paper is focused on minimizing the space cost of token embeddings for fast online re-ranking inference. Deep contextual re-ranking models . Neural re-ranking has pursued representation-based or interaction-based algorithms (Guo et al., 2016; Dai et al., 2018; Xiong et al., 2017). Embedding interaction based on query and document terms shows an advantage in these studies. The transformer architecture based on BERT (Devlin et al., 2019) has been adopted to re-ranking tasks by using BERT's [CLS] token representation to summarize query and document interactions (Nogueira and Cho, 2019; Yang et al., 2019; Dai and Callan, 2019; Nogueira et al., 2019a; Li et al., 2020). Recently BERT is integrated in late term interaction (MacA-vaney et al., 2019; Hofsttter et al., 2020c,b; Mitra et al., 2021) which delivers strong relevance scores for re-ranking. Efficiency optimization for transformer-based re-ranking. Several approaches have been proposed to reduce the time complexity of transformer-based ranking. For example, architecture simplification (Hofsttter et al., 2020c; Mitra et al., 2021), late interaction with precomputed token embeddings (MacAvaney et al., 2020), early exiting (Xin et al., 2020), and model distillation (Gao et al., 2020; Hofsttter et al., 2020a; Chen et al., 2020b). We will focus on the compression of token representation following the late-interaction work of ColBERT (Khattab and Zaharia, 2020) and BECR (Yang et al., 2022) as they deliver fairly competitive relevance scores for several well-known ad-hoc TREC datasets. These late-interaction approaches follow a dual-encoder design that separately encodes the two sets of texts, studied in various NLP tasks (Zhan et al., 2020; Chen et al., 2020a; Reimers and Gurevych, 2019; Karpukhin et al., 2020; Zhang et al., 2020). Several previous re-ranking model attempted to reduce the space need for contextual token embeddings. ColBERT has considered an option of using a smaller dimension per vector and limiting 2 bytes per number as a scalar quantization. BECR (Yang et al., 2022) uses LSH for hashing-based contextual embedding compression (Ji et al., 2019). PreTTR (MacAvaney et al., 2020) uses a single layer encoder model to reduce the dimensionality of each token embedding. Following PreTTR, a contemporaneous work called SDR in Cohen et al. (2021) considers an autoencoder to reduce the dimension of representations, followed by an off-the-shelf scalar quantizer. For the autoencoder, it combines static BERT embeddings with contextual embeddings. Inspired by this study, our work decomposes contextual embeddings to decouple ranking contributions during vector quantization. Unlike SDR, CQ jointly learns the codebooks and decomposition for the document-independent and dependent components guided by a ranking loss. Vector quantization. Vector quantization with codebooks was developed for data compression to assist approximate nearest neighbor search, for example, product quantizer (PQ) from Jgou et al. (2011), optimized product quantizer (OPQ) from Ge et al. (2013); residual additive quantizer(RQ) from Ai et al. (2015) and local search additive quantizer (LSQ) from Martinez et al. (2018). Recently such a technique has been used for compressing static word embeddings (Shu and Nakayama, 2018) and document representation vectors in a dense retrieval scheme called JPQ (Zhan et al., 2021a). None of the previous work has worked on quantization of contextual token vectors for the re-ranking task, and that is the focus of this paper. Applying vector quantization naively to token embedding compression does not ensure the ranking effectiveness because a quantizer-based compression is not lossless, and critical ranking signals could be lost during data transformation. To achieve a high compression ratio while maintaining the competitiveness in relevance, we consider the ranking contribution of a contextual token embedding for soft matching containing two components: 1) document specific component derived from the self attention among context in a document, 2) document-independent and corpus-specific component generated by the transformer model. Since for a reasonable sized document set, the second component is invariant to documents, its storage space is negligible compared to the first component. Thus the second part does not need compres-696 Figure 1: Offline processing and online ranking with contextual quantization sion. We focus on compressing the first component using codebooks. This decomposition strategy can reduce the relevance loss due to compression approximation, which allows a more aggressive compression ratio. Our integrated vector quantizer with contextual decomposition contains a ranking-oriented scheme with an encoder and decoder network for jointly learning codebooks and composition weights. Thus, the online composition of decompressed document-dependent information with document-independent information can retain a good relevance. A vector quantizer consists of two steps as discussed in Shu and Nakayama (2018). In the compression step, it encodes a real-valued vector (such as a token embedding vector in our case) into a short code using a neural encoder. The short code is a list of reference indices to the codewords in codebooks. During the decompression step, a neural decoder is employed to reconstruct the original vector from the code and codebooks. The quantizer learns a set of M codebooks {C 1 , C 2 ,    , CM } and each codebook contains K codewords ( C m = { c m 1 , c m 2 ,    , c mK } ) of dimension h . Then for any D-dimensional real valued vector x  RD , the encoder compresses x into an M dimensional code vector s . Each entry of code s is an integer j , denoting the j -th codeword in codebook C m . After locating all M codewords as [ c 1 ,    , c M ] , the original vector can be recovered with two options. For a product quantizer, the dimension of codeword is h = D/M , and the decompressed vector is  x = c 1  c 2     c M where symbol  denotes vector concatenation. For an additive quantizerthe decompressed vector is  x = (cid:80) Mj =1 c j . Codebook-based contextual quantization. Now we describe how codebook-based compression is used in our contextual quantization. Given a token t , we consider its contextual embedding vector E ( t ) as a weighted combination of two components: E ( t  ) and E ( t ) . E ( t  ) captures the document-dependent component, and E ( t ) captures the document-independent component discussed earlier. For a transformer model such as BERT, E ( t ) is the token output from the last encoder layer, and we obtain E ( t ) by feeding [CLS]  t  [SEP] into BERT model and taking last layer's output for t . During offline data compression, we do not explicitly derive E ( t  ) as we only need to store the compressed format of such a value, represented as a code. Let  E ( t  ) be the recovered vector with codebook-based decompression, as a close approximation of E ( t  ) . Let  E ( t ) be the final composed embedding used for online ranking with late-interaction. Then  E ( t ) = g ( E ( t  ) , E ( t )) where g ( . ) is a simple feed-forward network to combine two ranking contribution components. The encoder/decoder neural architecture for contextual quantization. We denote a token in a document d as t . The input to the quantization encoder is E ( t )  E ( t ) . The output of the quantization encoder is the code vector s of dimension M . Let code s be ( s 1 ,    , s m ,    , s M ) and each entry s m will be computed below in Eq. 4. This computation uses the hidden layer h defined as: h = tanh( w 0 ( E ( t )  E ( t )) + b 0 ) . The dimension of h is fixed as 1  MK/ 2 . The hidden layer a is computed by a feed forward layer with a softplus activation (Eq. 2) with an output dimension of M  K after reshaping, Let a m be 697 the m -th row of this output. To derive a discrete code entry for s m , following the previous work (Shu and Nakayama, 2018), we apply the Gumbel-softmax trick (Maddison et al., 2017; Jang et al., 2017) as shown in Eq. 3, where the temperature  is fixed at 1 and  k is a noise term sampled from the Gumbel distribution  log(  log( Uniform [0 , 1])) . Here p m is a vector with dimension K . ( p m ) j is the j -th entry of the vector. Similarly, ( a m ) j is the j -th entry of a m . ( p m ) j = exp(log(( a m ) j +  j ) / ) (cid:80) Kj  =1 exp(log(( a m ) j  +  j  ) / ) . (3) s m = arg max 1  j  K ( p m ) j . (4) In the decompression stage, the input to the quantization decoder is the code s , and this decoder accesses M codebooks {C 1 , C 2 ,    , CM } as M parameter matrices of size K  h which will be learned. For each m -entry of code s , s m value is the index of row vector in C m to be used as its corresponding codeword. Once all codewords c 1 to c M are fetched, we recover the approximate vector  E ( t  ) as (cid:80) Mj =1 c j for additive quantization or c 1  c 2     c M for product quantization. Next, we perform a composition with a one-layer or two-layer feed-forward network to derive the contextual embedding as  E ( t ) = g ( E ( t  , E ( t )) . With one feed-forward layer,  E ( t ) = tanh( w 2 ( E ( t  )  E ( t )) + b 2 ) . The above encoder and decoder for quantization have parameter w 0 , b 0 , w 1 , b 1 , w 2 , b 2 , and {C 1 , C 2 ,    , CM } . These parameters are learned through training. Once these parameters are learned, the quantization model is fixed and the code for any new token embedding can be computed using Eq. 4 in offline processing. Figure 1 depicts the flow of offline learning and the online inference with context quantization. Given a query with l tokens { q 1 , q 2 , ..q l } , and a documents with n tokens { t 1 , t 2 , ..t n } , The query token embeddings encoded with a transformer based model (e.g. BERT) are denoted as E ( q 1 ) ,    , E ( q l ) . The embeddings for document tokens through codebook base decompression are  E ( t 1 ) ,     E ( t n ) . The online inference then uses the interaction of query tokens and document tokens defined in a re-ranking algorithm such as ColBERT to derive a ranking score (denoted as f q , d ). The purpose of injecting E ( t ) in Eq. 1 is to decouple the document-independent ranking contribution from contextual embedding  E ( t  ) so that this quantization encoder model will be learned to implicitly extract and compress the document-dependent ranking contribution. Table 1 gives an example with several token codes produced by CQ for different sentences representing different contexts, and illustrates context awareness of CQ's encoding with a small codebook dimension (M=K=4). For example, 1 in code [4, 4, 3, 1] means the 4-th dimension uses the first codeword of the corresponding codebook. Training of CQ uses the MS MARCO passage dataset discussed in Section 4 and these sentences are not from this dataset. Our observation from this example is described as follows. First, in general token codes in the same sentences are closer to each other, and token codes in different sentences, even with the same word bank, are far away with a visible Hamming distance. Thus CQ coding allows a context-based separation among tokens residing in different contexts. Second, by looking at boldfaced tokens at each sentence, their distance in terms of contextual semantics and proximity is reflected to some degree in their CQ codes. For instance, a small Hamming code distance of three words ac-tor, poet and writer resembles their semantic and positional closeness. A larger code distance of two banks in the 3 rd and 4 th sentences relates with their word sense and positional difference. Training loss for parameter learning . We have explored three training loss functions. The first option is to follow a general quantizer (Shu and Nakayama, 2018) using the mean squared error (MSE) between the reconstructed and original embedding vectors of all token t i . Namely LMSE = (cid:80)  E ( t i )   E ( t i )  22 . The second option is the pairwise cross-entropy loss based on rank orders. After warming up with the MSE loss, we further train the quantizer using L PairwiseCE = (cid:80) (  (cid:80) j = d + , d  P j log P j ) where d + and d  are positive and negative documents for query q . We adopt the third option which borrows the idea of MarginMSE loss from Hofsttter et al. (2020a) proposed for BERT-based ranking model distillation. In MarginMSE, a student model is trained to 698 Context Token codes William Shakespeare was widely regarded as the world's greatest writer actor poet actor , poet , writer and dramatist. mimic the teacher model in terms of both ranking scores as well as the document relative order margins. In our case, the teacher model is the ranking model without quantization and the student model is the ranking model with quantization. It is defined as L MarginMSE = (cid:80) (( f q , d +  f q , d  )  (  f q , d +   f q , d  )) 2 , where f q , d and  f q , d denote the ranking score with and without quantization, respectively. The above loss function distills the ColBERT ranking characteristics into the CQ model for better preservation of ranking effectiveness. Online space for document embeddings. The storage cost of the precomputed document embeddings in a late-interaction re-ranking algorithm is dominating its online space need. To recover token-based document embeddings, an online server with contextual quantization stores three parts: codebooks, the short codes of tokens in each document, and the document-independent embeddings. Given a document collection of Z documents of length n tokens on average, let V be the number of the distinct tokens. For M codebooks with M  K codewords of dimension h , we store each entry of a codeword with a 4-byte floating point number. Thus the space cost of codebooks is M  K  h  4 bytes, and the space for document-independent embeddings of dimension D is V  D  4 bytes. When M = 16 , K = 256 , D = 128 as in our experiments, if we use the product quantization with the hidden dimension h = 8 , the codebook size is 131 MB. In the WordPiece English token set for BERT, V  32 K and the space for document-independent embeddings cost about 16.4 MB. Thus the space cost of the above two parts is insignificant. The online space cost of token-based document embeddings is Z  n  ( M log 2 K 8 + 2) bytes. Here each contextual token embedding of length D is encoded into a code of length M and the space of each code costs log 2 K bits. For each document, we also need to store the IDs of its tokens in order to access document-independent embeddings. We use 2 bytes per token ID in our evaluation because the BERT dictionary based on WordPiece (Wu et al., 2016) tokenizer has about 32,000 tokens. In comparison, the space for document embeddings in ColBERT with 2 bytes per number costs Z  D  n  2 bytes. Then the space ratio of ColBERT without CQ and with CQ is about 2 D  8 M log 2 K +2  8 , which is about 14:1 when D = 128 , M = 16 and K = 256 . BECR uses 5 layers of the refinement outcome with the BERT encoder for each token and stores each layer of the embedding with a 256 bit LSH signature. Thus the space cost ratio of BECR over ColBERT-CQ is approximately 5  256 M log 2 K +2  8 , which is about 9:1 when M = 16 and K = 256 . We can adjust the parameters of each of ColBERT, BECR, and ColBERT-CQ for a smaller space with a degraded relevance, and their space ratio to CQ remains large, which will be discussed in Section 4. Time cost for online decompression and composition. Let k be the number of documents to re-rank. The cost of decompression with the short code of a token using the cookbooks is O ( M  h ) for a product quantizer and O ( M  D ) for an additive quantizer. Notice M  h = D . For a one-layer feed-forward network as a composition to recover the final embedding, the total time cost for decompression and composition is O ( k  n  D 2 ) with a product quantizer, and O ( k  n ( M  D + D 2 )) with an additive quantizer. When using two hidden layers with D dimensions in the first layer output, there is some extra time cost but the order of time complexity remains unchanged. Noted that because of using feed-forward layers in final recovery, our contextual quantizer cannot take advantage of an efficiency optimization called 699 asymmetric distance computation in Jgou et al. (2011). Since embedding recovery is only applied to top k documents after the first-stage retrieval, the time efficiency for re-ranking is still reasonable without such an optimization. Datasets and metrics. The well-known MS MARCO passage and document ranking datasets are used. As summarized the in Table 2, our evaluation uses the MS MARCO document and passage collections for document and passage ranking (Craswell et al., 2020; Campos et al., 2016). The original document and passage ranking tasks provide 367,013 and 502,940 training queries respectively, with about one judgment label per query. The development query sets are used for relevance evaluation. The TREC Deep Learning (DL) 2019 and 2020 tracks provide 200 test queries with many judgment labels per query for each task. Following the official leader-board standard, for the development sets, we report mean reciprocal rank (MRR@10, MRR@100) for relevance instead of using normalized discounted cumulative gain (NDCG) (Jrvelin and Keklinen, 2002) because such a set has about one judgment label per query, which is too sparse to use NDCG. For TREC DL test sets which have many judgement lables per query, we report the commonly used NDCG@10 score. We also measure the dominating space need of the embeddings in bytes and re-ranking time latency in milliseconds. To evaluate latency, we uses an Amazon AWS g4dn instance with Intel Cascade Lake CPUs and an NVIDIA T4 GPU. In all tables below that compare relevance, we perform paired t-test on 95% confidence levels. In Tables 3, 4, and 5, we mark the results with   ' if the compression method result in statistically significant degradation from the ColBERT baseline. In Table 6,   ' is marked for numbers with statistically significant degradation from default setting in the first row. Choices of first-stage retrieval models. To retrieve top 1,000 results before re-ranking, we consider the standard fast BM25 method (Robertson and Zaragoza, 2009). We have also considered sparse and dense retrievers that outperform BM25. We have used uniCOIL (Lin and Ma, 2021; Gao et al., 2021) as an alternative sparse retriever in Table 3 because it achieves a similar level of relevance as end-to-end ColBERT with a dense retriever, and that of other learned sparse representations (Mallia et al., 2021; Formal et al., 2021b,a). ColBERT+uniCOIL has 0.369 MRR while end-to-end ColBERT has 0.360 MRR on MSMARCO Dev set. Moreover, retrieval with a sparse representation such as uniCOIL and BM25 normally uses much less computing resources than a dense retriever. Relevance numbers reported in some of the previous work on dense retrieval are derived from the exact search as an upper bound of accuracy. When non-exact retrieval techniques such as approximate nearest neighbor or maximum inner product search are used on a more affordable platform for large datasets, there is a visible loss of relevance (Lewis et al., 2021). It should be emphasized that the first stage model can be done by either a sparse or a dense retrieval, and this does not affect the applicability of CQ for the second stage as the focus of this paper. Re-ranking models and quantizers compared. We demonstrate the use of CQ for token compression in ColBERT in this paper. We compare its relevance with ColBERT, BECR and PreTTR. We chose to apply CQ to ColBERT because assuming embeddings are in memory, ColBERT is one of the fastest recent online re-ranking algorithms with strong relevance scores and CQ addresses its embedding storage weakness. Other re-ranking models compared include: BERT-base (Devlin et al., 2019), a cross encoder re-ranker, which takes a query and a document at run time and uses the last layers output from the BERT [CLS] token to generate a ranking score; TILDEv2 (Zhuang and Zuccon, 2021), which expands each document and additively aggregates precomputed neural scores. We also evaluate the use of unsupervised quantization methods discussed in Section 2 for ColBERT, including two product quantizers (PQ and OPQ), and two additive quantizers (RQ and LSQ). Appendix A has additional details on the retrievers considered, re-ranker implementation, training, and relevance numbers cited. Table 3 and Table 4 show the ranking relevance in NDCG and MRR of the different methods and compare against the use of CQ with ColBERT (marked as ColBERT-CQ). We either report our experiment results or cite the relevance numbers from other papers with a  mark for such a model. For quantization approaches, we adopt M=16, K=256, i.e. compression ratio 14:1 compared to ColBERT. For the passage task, ColBERT outperforms other re-rankers in relevance for the tested cases. ColBERT-CQ after BM25 or uniCOIL retrieval only has a small relevance degradation with around 1% or less, while only requiring 3% of the storage of ColBERT. The relevance of the ColBERT-CQ+uniCOIL combination is also competitive to the one reported in Mallia et al. (2021) for the Col-BERT+DeepImpact combination which has MRR 0.362 for the Dev query set, NDCG@10 0.722 for TREC DL 2019 and 0.691 for TREC DL 2020. For the document re-ranking task, Table 4 similarly confirms the effectiveness of ColBERT-CQ. ColBERT-CQ and ColBERT after BM25 retrieval also perform well in general compared to the relevance results of the other baselines. From both Table 3 and Table 4, we observe that in general, CQ significantly outperforms the other quantization approaches (PQ, OPQ, RQ, and LSQ). As an example, we further explain this by plotting the ranking score of ColBERT with and without a Model Specs. quantizer in Figure 2(a). Compared to OPQ, CQ trained with two loss functions generates ranking scores much closer to the original ColBERT ranking score, and this is also reflected in Kendall's  correlation coefficients of top 1,000 re-ranked results between a quantized ColBERT and the original ColBERT (Figure 2(b)). There are two reasons that CQ outperforms the other quantizers: 1) The previous quantizers do not perform contextual decomposition to isolate intrinsic context-independent information in embeddings, and thus their approximation yields more relevance loss; 2) Their training loss function is not tailored to the re-ranking task. passage corpora, and compares CQ with other approaches. Each MS MARCO document is divided into overlapped passage segments of size up to 400 tokens, and there are 60 tokens overlapped between two consecutive passage segments, following the ColBERT setup. As a result, the number of WordPiece tokens per document changes from 1460 to about 2031 due to the addition of overlapping contextual tokens. To demonstrate the tradeoff, we also list their estimated time latency and relevance in passage re-ranking as a reference and notice that more relevance comparison results are in Tables 3 and 4. The latency is the total time for embedding decom-pression/recovery and re-ranking. For PreTTR and ColBERT, we assume that their passage embedding data cannot fit in memory given their large data sizes. The disk I/O latency number is based on their passage embedding size and our test on a Samsung 870 QVO solid-state disk drive to fetch 1,000 passage embeddings randomly. Their I/O latency takes 110ms or 182ms with single-thread I/O and with no I/O contention, and their disk access can incur much more time when multiple queries are processed in parallel in a server dealing with many clients. For example, fetching 1,000 passage embeddings for each of ColBERT and PreTTR takes about 1,001ms and 3,870ms respectively when the server is handling 16 and 64 queries simultaneously with multiple threads. in the 4-th column of Table 5 excludes the first-stage retrieval time. The default ColBERT uses embedding dimension 128 and 2 byte floating numbers. ColBERT-small denotes an optional configuration suggested from the ColBERT paper using 24 embedding dimensions and 2-byte floating numbers with a degraded relevance performance. As shown in Table 5, the embedding footprint of ColBERT CQ uses about 112GB and 10.2GB, respectively for document and passage re-ranking tasks. By looking at the latency difference of ColBERT with and without CQ, the time overhead of CQ for decompression and embedding recovery takes 1ms per query, which is insignificant. Compared with another quantizer ColBERT-OPQ, ColBERT-CQ can achieve the same level of space saving with K = 256 while having a substantial relevance improvement. ColBERT-CQ with K = 4 achieves the same level of relevance as ColBERT-OPQ while yielding a storage reduction of 67% and a latency reduction of about 70%. Comparing ColBERT-CQ with no contextual decomposition, under the same space cost, ColBERT-CQ's relevance is 4% higher. CQ with K = 16 achieves the same level relevance as ColBERT-CQ-undecomposed with K = 256 , while the storage of CQ reduces by 44%. Comparing with ColBERT-small which adopts more aggressive space reduction, ColBERT-CQ with K = 16 would be competitive in relevance while its space is about 4x smaller. Comparing with other non-ColBERT baselines (BECR, PreTTR, and TILDEv2), ColBERT-CQ strikes a good balance across relevance, space and latency. For the fast CPU based model (BECR, TILDEv2), our model achieves better relevance with either lower or comparable space usage. For BECR, its embedding footprint with 89.9GB may fit in memory for MS MARCO passages, it becomes very expensive to configure a machine with much more memory for BECR's MS MARCO document embeddings with about 791GB. Table 6 shows the relevance scores for the TREC deep learning passage ranking task with different design options for CQ. As an alternative setting, the codebooks in this table use M=16 and K=32 with compression ratio 21:1 compared to ColBERT. Row 1 is the default design configuration for CQ with product operators and 1 composition layer, 702 TREC19 TREC20 CQ, Product, 1 layer, MarginMSE 0.687 0.713 Different model configurations No decomposition. Different architecture or quantization options. Rows 2 and 3 of Table 6 denote CQ using product or additive operators without decomposing each embedding into two components, and there is about 4% degradation without such decomposition. Row 4 changes CQ using the raw static embeddings of tokens from BERT instead of the upper layer outcome of BERT encoder and there is an up to 4.7% degradation. Notice such a strategy is used in SDR. From Row 5 to Row 7, we change CQ to use additive operators or use a two-layer composition. The performance of product or additive operators is in a similar level while the benefit of using two layers is relatively small. Different training loss functions for CQ. Last two rows of Table 6 use the MSE and PairwiseCE loss functions, respectively. There is an about 1.2% improvement using MarginMSE. Figure 2 gives an explanation why MarginMSE is more effective. While CQ trained with MSE and MarginMSE generates ranking scores close to the original ranking scores in Figure 2(a), the distribution of Kendall's  correlation coefficients of 1,000 passages in Figure 2(b) shows that the passage rank order derived by CQ with the MarginMSE loss has a better correlation with that by ColBERT. Our evaluation shows the effectiveness of CQ used for ColBERT in compressing the space of token embeddings with about 14:1 ratio while incurring a small relevance degradation in MS MARCO passage and document re-ranking tasks. The quantized token-based document embeddings for the tested cases can be hosted in memory for fast and high-throughput access. This is accomplished by a neural network that decomposes ranking contributions of contextual embeddings, and jointly trains context-aware decomposition and quantization with a loss function preserving ranking accuracy. The online time cost to decompress and recover embeddings is insignificant with 1ms for the tested cases. The CQ implementation is available at https://github.com/yingrui-yang/ContextualQuantizer. Our CQ framework is also applicable to the contemporaneous work ColBERTv2 (Santhanam et al., 2021). Using uniCOIL scores for the first-stage sparse retrieval and ColBERTv2+CQ (M=16, K=256) for top 1,000 passage reranking, we achieve 0.387 MRR@10 on the MSMARCO passage Dev set, 0.746 NDCG@10 on TREC DL19, and 0.726 NDCG@10 on DL20 with about 10.2GB embedding space footprint. Notice that ColBERTv2 achieves a higher MRR@10 number 0.397 for the passage Dev set when used as a standalone retriever (Santhanam et al., 2021) and dense retrieval with such a multi-vector representation is likely to be much more expensive than retrieval with a sparse representation on a large dataset. The previous work in dense retrieval has often employed faster but approximate search, but that comes with a visible loss of relevance (Lewis et al., 2021). Thus the above relevance number using ColBERTv2+CQ for re-ranking with uniCOIL sparse retrieval is fairly strong, achievable with a reasonable latency and limited computing resource. Its embedding space size is 2.8x smaller than the 29GB space cost in the standalone ColBERTv2 (Santhanam et al., 2021) for MS MARCO passages. Our future work is to investigate the above issue further and study the use of CQ in the other late-interaction re-ranking methods. Acknowledgments . We thank Cindy Zhao, Ji-ahua Wang, and anonymous referees for their valuable comments and/or help. This work is supported in part by NSF IIS-2040146 and by a Google faculty research award. It has used the Extreme Science and Engineering Discovery Environment supported by NSF ACI-1548562. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
abstain abstain objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain objective result other other method method other other other other other other other other method other other other other other other other objective other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain objective abstain abstain abstain abstain objective abstain abstain abstain abstain abstain