Pinecone Systems ingber@pinecone.io BERT based ranking models have achieved superior performance on various information retrieval tasks. However, the large number of parameters and complex self-attention operations come at a significant latency overhead. To remedy this, recent works propose late-interaction architectures, which allow pre-computation of intermediate document representations, thus reducing latency. Nonetheless, having solved the immediate latency issue, these methods now introduce storage costs and network fetching latency, which limit their adoption in real-life production systems. In this work, we propose the Succinct Document Representation (SDR) scheme that computes highly compressed intermediate document representations, mitigating the stor-age/network issue. Our approach first reduces the dimension of token representations by encoding them using a novel autoencoder architecture that uses the document's textual content in both the encoding and decoding phases. After this token encoding step, we further reduce the size of the document representations using modern quantization techniques. Evaluation on MSMARCO's passage re-reranking task show that compared to existing approaches using compressed document representations, our method is highly efficient, achieving 4x11.6x higher compression rates for the same ranking quality. Similarly, on the TREC CAR dataset, we achieve 7.7x higher compression rate for the same ranking quality. Information retrieval (IR) systems traditionally comprise of two stages: retrieval and ranking. Given a user query, the role of the retrieval stage is to quickly retrieve a set of candidate documents  Both authors contributed equally to the paper. from a (very large) search index. Retrieval is typically fast but not accurate enough; in order to improve the quality of the end result for the user, the candidate documents are re-ranked using a more accurate but computationally expensive algorithm. Neural approaches have achieved the state of the art ranking performance in IR applications (Yates et al., 2021). Transformer networks such as BERT (Devlin et al., 2019) consistently show better ranking effectiveness at the cost of a higher computational cost and latency (Nogueira and Cho, 2019). To rank k documents, the ranker is called k times with an input of the form (query, document), where the query is the same, but the document is different. Several works (MacAvaney et al., 2020; Gao et al., 2020b; Chen et al., 2020; Cao et al., 2020; Nie et al., 2020; Gao et al., 2020b; Khattab and Zaharia, 2020) have proposed to modify BERT-based rankers in a way that allows part of the model to compute query and document representations separately, and then produce the final score using a low-complexity interaction block; we denote these models as late-6624 interaction rankers. Such approaches pre-compute document representations to improve latency significantly. Next, at runtime the model computes the query representation (once), retrieves the precomputed document representations, and is only required to run a low-complexity interaction block k times to produce the final ranking score. Precomputing document representations has shown to significantly reduce latency and at the same time retain comparable scores to BERT models (Gao et al., 2020b). However, this does not account for additional storage and/or network fetching latency costs. The representations typically consist of the contextual token embeddings in a transformer model, which consume orders of magnitude more storage than storing the entire corpus search index (cf.  5.1). In this work, we propose Succinct Document Representation (SDR), a general scheme for compressing document representations. It enables late-interaction rankers to be efficient in both latency and storage, while maintaining high ranking quality. SDR is suitable for any ranking scheme that uses contextual embeddings, and achieves extreme compression ratios (2-3 orders of magnitude) with little to no impact on retrieval accuracy. SDR consists of two major components: (1) embedding dimension reduction using an autoencoder with side information and (2) distribution-optimized quantization of the reduced-dimension vectors. In SDR, the autoencoder consists of two subnetworks: an encoder that reduces the vector's dimensions and a decoder that reconstructs the compressed vector. The encoder's output dimension represents the tradeoff between reconstruction fi-delity and storage requirements. To improve the compression-reliability tradeoff, we leverage static token embeddings, which are available since the ranker has access to the document text (as it needs to render it to the user), and are computationally cheap to obtain. We feed these embeddings to both the encoder and decoder as side information , allowing the autoencoder to focus more on storing  just the context  of a token, and less on its original meaning that is available in the static embeddings. Ablation tests verify that adding the static vectors significantly improves the compression rates for the same ranking accuracy. Since data storage is measured in bits rather than floating-point numbers, SDR uses quantization techniques to reduce storage size further. Given that it is hard to evaluate the amount of information in each of the encoder's output dimensions, we perform a randomized Hadamard transform on the vectors, resulting in (1) evenly spread information across all coordinates and (2) transformed vectors that follow a Gaussian-like distribution. We utilize known quantization techniques to represent these vectors using a small number of bits, controlling for the amount of quantization distortion. Existing late-interaction schemes either ignore the storage overhead, or consider basic compression techniques, such as a simple (1 layer) autoencoder and float16 quantization. However, this is insufficient to reach reasonable storage size (MacA-vaney et al., 2020); furthermore, this results in an increased fetching latency. For the MSMARCO dataset, we used a distilled model with a reduced vector width (Hofsttter et al., 2020a) as the initial pre-trained weights for the late-interaction model. On top of this, we used a non-linear autoencoder consisting of 2 dense layers followed by float16 quantization, a natural extension of MacAvaney et al. (2020). This baseline achieves compression rates of 30x with no noticeable reduction in retrieval accuracy (measured with the official MRR@10 metric). In comparison with this strong baseline, our SDR scheme achieves an additional compression rate of between 4x to 11.6x with the same ranking quality, reducing document representation size to the same order of magnitude as the retrieved text itself. In Figure 1 we include a high-level presentation of the baseline, a variant of our method with float16 quantization, and our full method. For the TREC CAR dataset, for which we do not have a reduced-width baseline, we used a BERT model as the pre-trained weights for the late-interaction model. The baseline with 2 dense layers and float16 quantization achieves a 30x compression rates with a slight reduction in accuracy. The SDR scheme reaches the same quality while improving compression rate by another 7.7x. We propose the Succinct Document Representation (SDR) scheme for compressing the document representations required for fast Transformer-based rankers. The scheme is based on a specialized autoencoder architecture and subsequent quantization. For the MSMARCO passage retrieval task, SDR shows compression ratios of 121x with no noticeable decrease in ranking performance. Compared to existing approaches for producing compressed representations, our method attains better compression rates (between 4x and 11.6x) for the same ranking quality. Similar results are demonstrated on the TREC CAR dataset. We provide a thorough analysis of the SDR system, showing that the contribution of each of the components to the compression-ranking effectiveness is significant. Late-interaction models. The idea of running several transformer layers for the document and the query independently, and then combining them in the last transformer layers, was developed concurrently by multiple teams: PreTTR (MacAvaney et al., 2020), EARL (Gao et al., 2020a), DC-BERT (Nie et al., 2020), DiPair (Chen et al., 2020), and the Deformer (Cao et al., 2020). These works show that only a few layers where the query and document interact are sufficient to achieve results close to the performance of a full BERT ranker at a fraction of the runtime cost. For each document, the contextual token vectors are stored in a cache and retrieved during the document ranking phase. This impacts both storage cost as well as latency cost of fetching these vectors during the ranking phase. MORES (Gao et al., 2020b), extends late-interaction models, where in the last interaction layers only the query attends to the document (and not vice-versa). As document are typically much longer, this results in additional performance improvements with similar storage requirements. ColBERT (Khattab and Zaharia, 2020) is another variant that runs all transformer layers independently for the query and the document, and the interaction between the final vectors is done through a sum-of-max operator. A similar work, the Transformer-Kernel (TK) (Hofsttter et al., 2020b), has an interaction block based on a low-complexity kernel operation. Both ColBERT and TK result in models with lower runtime latency at the expense of a drop in ranking quality. However, the storage requirements for both approaches are still significant. Some of the works above acknowledge the issue of storing the precomputed document representations and proposed partial solutions. In ColBERT (Khattab and Zaharia, 2020), the authors proposed to reduce the dimension of the final token embedding using a linear layer. However, even moderate compression ratios caused a large drop in ranking quality. In the PreTTR model (MacAvaney et al., 2020), it was proposed to address the storage cost by using a standard auto-encoder architecture and the float16 format instead of float32. Again, the ranking quality drops even with moderate compression ratios (they measured up to 12x). Several other works (Guu et al., 2020; Karpukhin et al., 2020; Xiong et al., 2021; Qu et al., 2020; Lu et al., 2020) proposed representing the queries and documents as vectors (as opposed to a vector per token), and using dot product as the interaction block. While this ranker architecture approach is simple (and can also be used for the retrieval step via an approximate nearest neighbor search such as FAISS (Johnson et al., 2017), ScaNN (Guo et al., 2020) or the Pinecone managed service 2 ), the overall ranking quality is generally lower compared to methods that employ a query-document cross-attention interaction. For that reason these methods are used mainly for first-stage retrieval, followed by a reranking step. Compressed embeddings. Our work reduces storage requirements by reducing the number of bits per floating-point value. Quantization gained attention and success in reducing the size of neural network parameters (Gupta et al., 2015; Essam et al., 2017; Wang et al., 2018; Wu et al., 2018) and distributed learning communication costs (Suresh et al., 2017; Alistarh et al., 2017; Konecn`y and Richtrik, 2018; Vargaftik et al., 2021, 2022). Specifically, compressing word embeddings has been studied as an independent goal. May et al. (2019) studied the effect of quantized word embeddings on downstream applications and proposed a metric for quantifying this effect with simple linear models that operate on the word embeddings directly. As our work is concerned with compressing contextual embeddings, these methods do not apply since the set of possible embeddings values is not bounded by the vocabulary size. Nevertheless, as in (May et al., 2019), we also observe that simple quantization schemes are quite effective. Our work uses recent advances in this area to further reduce storage requirements for document representation, which, to the best of our knowledge, were not previously attempted in this context. Our work is based on the late-interaction architecture (MacAvaney et al., 2020; Gao et al., 2020b; Chen et al., 2020; Cao et al., 2020; Nie et al., 2020), which separates BERT into L independent layers for the documents and the queries, and T  L interleaving layers, where T is the total number of layers in the original model, e.g., 12 for BERT-Base. Naively storing all documents embeddings consumes a huge amount of storage with a total of m  h  4 bytes per document, where m is the average number of tokens per document and h is the model hidden size (384 for the distilled version and 768 for the BERT version). For MSMARCO, with 8.8M documents and m = 76 . 9 , it leads to a high storage cost of over a terabyte, which is not affordable except in large production systems. Our compression scheme for the document representations consists of two sequential steps, (i) dimensionality reduction and (ii) block-wise quantization, described in  3.1 and  3.2 respectively. To compress document representations, we reduce the dimensionality of token representations (i.e., the output of BERT's L -th layer) using an autoencoder. Standard autoencoder architectures typically consist of a neural network split into an encoder and a decoder: the encoder projects the input vector into a lower-dimension vector, which is then reconstructed back using the decoder. Our architecture, AESI, extends the standard autoencoder by using the document's text as side information to both the encoder and decoder. Such an approach is possible since, no matter how the document scores are computed, re-ranking systems have access to the document's text in order to render it back to the user. In the rest of this section, we add the precise details of the AESI architecture. Side Information. In line with our observation that the ranker has access to the document's raw text, we propose utilizing the token embedding information, which is computed by the embedding layer used in BERT's architecture. The token embeddings encode rich semantic information about the token itself; however, they do not fully capture the context in which they occur; hence, we refer to them as static embeddings . For example, through token embeddings, we cannot disambiguate between the different meanings of the token bank , which can refer to either a geographical location (e.g., river bank) or a financial institution, depending on the context. Static embeddings are key for upper BERT layers, which learn the contextual representation of tokens via the self-attention mechanism. We use the static embeddings as side information to both the encoder and decoder parts of the autoencoder. This allows the model to focus on encoding the distilled context , and less on the token information since it is already provided to the decoder directly. AESI Approach. For a token whose representation we wish to compress, our approach proceeds as follows. We take the L -th layer's output contextual representation of the token together with its static embedding and feed both inputs to the autoencoder. The information to be compressed (and reconstructed) is the contextual embedding, and the side-information, which aids in the compression task, is the static embedding. The decoder takes the encoder output, along with the static embedding, and attempts to reconstruct the contextual embedding. Figure 2 shows the AESI architecture. AESI approach has two parameters that are determined empirically. First, the L -th transformer layer of the contextual representation provided as input, which has a direct impact on latency 3 . Second, the size of the encoder's output directly impacts the compression rate and thus storage costs. Encoding starts by concatenating the input vector (i.e., the output of layer L , the vector we compress) and the static token embedding (i.e., the output of BERT's embedding layer), and then passes the concatenated vector through an encoder network, which outputs a c -dimensional encoded vector . Decoding starts by concatenating the encoded vector with the static token embedding, then passes the concatenated vector through a decoder layer, which reconstructs the input vector. Specifically, we use a two-layer dense network for both the encoder and the decoder, which can be written using the following formula: e = E ( v, u ) := W e 2  (cid:0) gelu (cid:0) W e 1 ( v ; u ) (cid:1)(cid:1) (1) v (cid:48) = D ( e, u ) := W d 2  (cid:0) gelu (cid:0) W d 1 ( e ; u ) (cid:1)(cid:1) (2) where v  R h is the contextualized token em-3 A ranker has to compute layers L + 1 onward online. bedding (the output of the L -th layer), u  R h is the static token embedding (the output of the embedding layer, which is the input to BERT's layer 0 and includes token position embeddings and type embeddings), and u ; v means concatenation of these vectors. W e 1  R i  2 h , W e 2  R c  i , W d 1  R i  ( c + h ) , W d 2  R h  i are trainable parameters. h is the dimension of token embeddings (e.g., 384), i is the intermediate autoencoder size, and c is the dimension of the projected (encoded) vector. gelu(  ) is an non-linear activation function (Hendrycks and Gimpel, 2016). Additional autoencoder variations are explored in  5.3. Storing the compressed contextual representations in a naive way consumes 32 bits (float32) per coordinate per token, which is still costly. To further reduce storage overhead, we propose to apply a quantization technique, which uses a predetermined B bits per coordinate. However, different coordinates and different tokens have different importance and possibly also different scales, so using the same number of bits and same quantization threshold for all of them increases the quantization error. To remedy this issue, we follow an approach similar to EDEN quantization (Vargaftik et al., 2022), which uses a randomized Hadamard transform prior to quantization. Loosely speaking, this shuffles the information across all coordinates. Furthermore, each of the coordinates is guaranteed to follow Gaussian-like distribution, for which quantization boundaries can be computed optimally. For the sake of brevity, the full description of the quantization algorithm is deferred to Appendix A. Efficiently applying the Hadamard transform requires the size of the input to be a power of two. In addition, the input dimension should be large enough (specifically, larger than the output of AESI) so that information can be shuffled effectively. Therefore, we concatenate the AESI vectors of all tokens from a single document, then segment it to a larger block size (we use 128), padding the last block with zeros when necessary. The padding slightly increases space requirements and is considered when evaluating the compression efficiency. In this section we describe the datasets used to evaluate the competing approaches for ranking documents given a query. Next, we describe the baseline and the different configurations of SDR with emphasis on how we measure the compression ratio. To evaluate the effectiveness of our proposed approach (SDR) and the competing baseline, we consider two information retrieval datasets, each with different characteristics. MSMARCO passage re-ranking In this task (Nguyen et al., 2016), we are given a query and a list of 1,000 passages (retrieved via BM25), and the task is to rerank the passages according to their relevance to the query. The corpus consists of 8.8M passages, downloaded from the web. We consider two query sets: (1) MSMARCO-DEV , the development set for the MSMARCO passage reranking task, which consists of 6,980 queries. On average, each query has a single relevant passage, and other passages are not annotated. The models are measured using the mean reciprocal rank metric (MRR@10). (2) TREC 2019 DL Track . Here we consider the test queries from TREC 2019 DL Track passage reranking dataset. Unlike MSMARCO-DEV, there are multiple passages annotated for each query with graded relevance labels (instead of binary labels), allowing us to use the more informative nDCG@10 metric. Due to the excessive annotation overhead, this dataset consists of just 200 queries, so results are noisier compared to MSMARCO-DEV. TREC Complex Answer Retrieval (CAR) is a dataset (Dietz et al., 2017) curated from Wikipedia. It maps from article and section titles to relevant paragraphs. Following Nogueira and Cho (2019), 6628 we use the automatic by-article annotations variant, which considers all paragraphs within the same article as relevant. The dataset consists of 30M passages, making storage requirements a more significant challenge compared to the MSMARCO task. The test query set consists of 2,254 queries with an average of 2.74 positive passages per query. We use the MAP@1K official metric. For both datasets, in addition to the quality metrics, we also measure the Compression Ratio (CR) as the amount of storage required to store the token embeddings when compared to the baseline model. E.g., CR = 10 implies storage size that is one tenth of the baseline vectors. Our algorithm is based on the late-interaction architecture (MacAvaney et al., 2020; Gao et al., 2020a; Nie et al., 2020; Chen et al., 2020; Cao et al., 2020). We created a model based on this architecture, which we name BERTSPLIT , consisting of 10 layers that are computed independently for the query and the document with an additional two late-interaction layers that are executed jointly. For MSMARCO, we initialized the model from reduced width pre-trained weights 4 and fine-tuned it using knowledge distillation from an ensemble of BERT-Large, BERT-Base, and ALBERT-Large (Hofsttter et al., 2020b) on the MSMARCO small training dataset, which consists of almost 40M tuples of query, a relevant document, and an irrelevant document. For CAR, the model is initialized from pre-trained BERT-base model and trained on 50M samples curated by Nogueira and Cho (2019). We trained autoencoder variants on a random subset of 500k documents to reduce training time. We incorporate the quantization overhead into the computation of the compression ratios , including meta-data and the overhead of padding (cf. Appendix A). In the following sections, we denote the SDR variants as AESI-{c}-{B}b where { c } is replaced with the width of the encoded vector and { B } is replaced with the number of bits in the quantization scheme. When discussing AESI with no quantization, we simply write AESI-{c}. To measure end to end latency, we configured an OpenSearch 5 cluster in AWS. We used default pro-duction configurations, with 3 r6g.large datanode machines; disk space was set to 0.5TB. For ranking, we used a single g4dn.xlarge machine, featuring a single T4 GPU instance. This makes the cost of these two components similar. In this section, we present the end to end latency results ( 5.1), show compression ratios and quality tradeoff of the SDR scheme ( 5.2). We then examine how the proposed autoencoder ( 5.3) compares with other baselines and present additional measurements ( 5.4). Table 1 (top) shows the latency benefits of SDR on the MSMARCO dataset, assuming document embeddings are stored in the OpenSearch retrieval system and 1k documents are retrieved per query. The Distilbert model (full interaction architecture) has the highest quality and smallest index size (since it is only executed online). However, ranking latency is prohibitively expensive. As a baseline, we use a late interaction model, a two-layer autoencoder with code dimension 24 and float16 quantization, denoted Late+AE-24. For this baseline, the ranking latency is significantly reduced at a cost in terms of quality. However, the document representation 5 https://aws.amazon.com/ opensearch-service/ . is large, causing retrieval and overall latency to increase to 0.7 and 1.22 seconds, respectively. SDR, with a dimension of 16 and 6-bits quantization, reaches the same quality as the baseline while striking a better balance between retrieval and ranking latency, reaching overall latency of 1.1 seconds. The index size is also significantly reduced compared to the baseline compression algorithm. We also consider variants of the algorithms where the documents are pre-tokenized, and the tokenization output is retrieved instead of computing at runtime (marked as +tok in the table). This further improves the ranking latency at the expense of a slight increase in index size. Note that the baseline does not use the raw text and therefore does not benefit from precomputed tokens. Table 1 (bottom) shows the latency results on the CAR dataset. Here too, the BERT baseline has the highest ranking quality, at the cost of prohibitive latency. The late interaction variants we consider have the same configuration as in the MSMARCO case, where the baseline uses 24 features (with float16 quantization) and SDR uses 16 features (with 6 bits EDEN quantization). Unlike in the MSMARCO case, the quality (i.e., MAP@1k score) of these two options is not similar. This makes SDR better than the baseline in latency, index size, as well as quality (by a large margin of over 14%). In Appendix D we explore additional configurations and show that the baseline with 52 features reaches the same quality as SDR-16-6b. However, we do not measure end-to-end latency for this case due to the excessive storage size and indexing time. Note that using 52 features for the baseline is expected to have a negative impact on retrieval latency, making the benefits of SDR even more pronounced. Table 2 shows the results on the MSMARCO query sets for SDR and its compression ratio against storing contextual token embeddings uncompressed. In terms of compression ratio, it can be seen that AESI allows us to massively reduce storage requirements both with and without quantization. AESI -16-6b reduces storage requirements by 121x, while at the same time showing no significant ranking performance drop. Using AESI-16-6b, a document's embedding can be stored with only 947 bytes and the entire MSMARCO collection can Quant.bits( B ) be stored within 8.6GB. There are several advantages of fitting the entire collection's representation into the main memory of the hosting machine, allowing for fast access, further fine-tuning, etc. If further compression rates are required, AESI-8-5b uses just 5 bytes per token, reaching a compression rate of 277x and 487 bytes per document on average. At this level of compression, the entire MSMARCO corpus fits in 3.8GB. The MRR@10 drop is noticeable (0.0119) but still quite low. Finally, for TREC19-DL, the impact of compressing token embeddings is less evident. Only in the most extreme cases such as AESI-4-4b we see a significant drop in nDCG@10 performance. These results demonstrate that the performance drop is very small, showing the effectiveness of our method. To better understand the impact of the autoencoder, we present MRR@10 results as a function of autoencoder dimensions (i.e., number of floats stored per token) and with the different autoencoder configurations. In addition to the 2-layer AESI architecture we described in  3.1 ( AESI-2L ), we consider the following variations: 6630 0.20 0.22 0.24 0.26 0.28 0.30 0.32 0.34 0.36 0.38 0 4 8 12 16 20 24 28 32 Auto Encoder Dimension (width) MRR @ 10 AE1L AE2L AESIDEC2L AESI1L AESI2L Figure 3: MRR@10 was measured on the MSMARCO-DEV-25 dataset as a function of autoencoder dimensions. AutoEncoder with 2 Layers (AE-2L). Standard 2-layer autoencoder with gelu activation. This is the same as AESI, only without the side information. AutoEncoder with 1 Layer (AE-1L). Standard autoencoder with a single dense layer in the encoder and decoder. AESI with 1 Layer (AESI-1L). AESI with a single dense encoder and decoder layer. DECoder-only AESI (AESI-DEC-2L). Provides side information to the decoder but not the encoder. To reduce measurement overhead, we ran the experiment only over the MSMARCO dataset. In addition, we took only the top 25 BERTSPLIT passages for each query, denoted MSMARCO-DEV -25, which has a negligible impact on the results. Figure 3 shows the results for the different autoencoder configurations. Providing the side information to the autoencoder proves to be very effective in reducing storage costs, especially when the encoded vector size is small. A 2-layer encoder/decoder model, as expected, is more effective than a single-layer model. The gap is especially large when using side information, showing that the interaction between the encoded vector and the static token embeddings is highly nonlinear. Finally, providing the static embeddings only to the decoder is slightly inferior to providing it also to the encoder. the quantization technique we use to several other techniques, including Deterministic Rounding (Gersho and Gray, 1992), Stochastic Rounding (Connolly et al., 2021), and Subtractive Dithering (Roberts, 1962; Gray and Stockham, 1993). Due to lack of space, the results appear in Appendix B. We found that a randomized Hadamard transform improves quality (assuming similar bit rate), especially in the low-bits regime. Using a quantization technique fit-ted to the Gaussian distribution of post randomized Hadamard transform data further improve quality, making the EDEN quantization superior to other quantization techniques in our case. Our scheme uses a fixed number of bits per coordinate, which is essential for performance. However, variable-rate compression can further reduce storage. We used rate-distortion theory (from the information theory field) to upper bound the benefits of such techniques by 11%, which does not seem to justify the added system complexity (cf. Appendix B). Intrinsic Evaluation of AESI-Encoded Vectors To better understand the impact of side information, we measure the error rate between an input vector and its reconstructed vector (i.e., after encoding and decoding). As expected, in practically all cases, adding the side information reduces error rate compared to a 2-layer autoencoder (AE-2L) with the same code dimension. In IR, the document frequency of a token is known to be negatively correlated with the token's importance. We found that the error rate for AE-2L decreases with frequency, while the error rate for AESI increases with frequency. This shows that the AESI scheme can better focus on tokens that are important for ranking. A possible explanation for this phenomena is that the static embeddings for infrequent tokens are more informative (i.e., more helpful as side information) compared to static embeddings for frequent tokens (e.g., the'). We also found AESI excels more in compressing nouns, verbs, and adjectives, while AE-2L excels more in compressing punctuation, determiners, and ad-positions. Again, this demonstrate that the static embeddings is most helpful in encoding tokens that are crucial for ranking. The details of this evaluation are provided in Appendix C. 6 Conclusion In this paper, we proposed a system called SDR to solve the storage cost and latency overhead of existing late-interaction transformer based models for passage re-ranking. The SDR scheme uses a novel 6631 autoencoder architecture that uses static token embeddings as side information to improve encoding quality. In addition, we explored different quantization techniques and showed that the recently proposed EDEN performs well in our use case and presented extensive experimentation. Overall, the SDR scheme reduces pre-computed document representation size by 4x11.6x compared to a baseline that uses existing approaches. In future work, we plan to continue investigating means to reduce pre-computed document representation size.We believe that additional analysis of BERT's vector and their interaction with the context would be fundamental in such an advancement.
abstain abstain abstain abstain abstain objective objective method result result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain result method abstain abstain result method abstain abstain method method abstain result abstain method abstain abstain objective objective objective method objective objective other other other other other other other other other other other other other other other other other other other other method other other other method abstain method other other other other abstain other other method other objective other method other objective abstain abstain other method other other method method other other other other other other method other method other other other other other abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain