Many recent approaches towards neural information retrieval mitigate their computational costs by using a multi-stage ranking pipeline. In the first stage, a number of potentially relevant candidates are retrieved using an efficient retrieval model such as BM25. Although BM25 has proven decent performance as a first-stage ranker, it tends to miss relevant passages. In this context we propose CoRT, a simple neural first-stage ranking model that leverages contextual representations from pretrained language models such as BERT to complement term-based ranking functions while causing no significant delay at query time. Using the MS MARCO dataset, we show that CoRT significantly increases the candidate recall by complementing BM25 with missing candidates. Consequently, we find subsequent re-rankers achieve superior results with less candidates. We further demonstrate that passage retrieval using CoRT can be realized with surprisingly low latencies. The successful development of neural ranking models over the past few years has rapidly advanced state-of-the-art performance in information retrieval (Guo et al., 2020; Craswell et al., 2020). One key aspect of the success is the exploitation of query-document interactions based on token representations from self-supervised language models (LMs) (Devlin et al., 2019; Radford et al., 2019; Pennington et al., 2014). Due to high computational effort, however, these interaction-focused approaches are limited to re-ranking scenarios and thus they depend on the effectiveness of first-stage ranking models for candidate retrieval. Term-based retrieval models such as BM25 have proven decent performance in this task, but tend to miss relevant passages. In this context, we propose COmplementary Rankings from Transformers (CoRT), a simple neural first-stage ranking model that leverages contextual representations from transformer-based language models (Vaswani et al., 2017; Devlin et al., 2019) to complement term-based first-stage rankings. CoRT optimizes an underlying text encoder towards representations that reflect relevance through vector similarity. The model is trained to act complementary to term-based retrieval by using passages from BM25 rankings as negative examples. We study the characteristics of CoRT with four types of experiments based on the MS MARCO dataset. First, we measure various ranking metrics and compare the results with first-stage ranking baselines and competitors. In course of this, we demonstrate the portion of relevant candidates that are added by CoRT. Second, we combine the candidates from CoRT and BM25 with a state-of-the-art re-ranker based on BERT (Nogueira and Cho, 2019) and investigate how many candidates are needed to saturate the ranking quality. Third, we train CoRT with various representation sizes and measure its impact on the first-stage ranking quality. Fourth, we measure the retrieval latencies of CoRT with two retrieval modalities: a distributed exhaustive search on four GPUs and an approximate search based on a graph-based nearest-neighbor index with pruning heuristics (Iwasaki and Miyazaki, 2018). Finally, we build an exemplary end-to-end ranking pipeline using our first-stage ranking to demonstrate its efficiency. Our contribution is a first-stage ranking framework with the potential to improve end-to-end ranking pipelines by adding candidates that term-based retrieval models typically miss. With this it is possible to reduce the total number of re-ranking candidates without hurting end-to-end ranking quality. As a secondary contribution, we provide an open-source implementation 1 that enables other researchers to reproduce our results and test CoRT on other datasets. In this section we describe key concepts of neural ranking and refer to related work. We then present neural first-stage ranking approaches that allow direct comparison with our results. 2.1 Key Concepts of Neural Ranking According to Guo et al. (2016), Neural ranking approaches can be categorized into two types of models depending on the architecture. Representation-focused approaches (Huang et al., 2013; Shen et al., 2014; Zamani et al., 2018) produce representations for queries and documents to predict relevancy scores using a simple distance or similarity measure. In this context, exploiting local interactions between neighboring terms is a commonly used technique (Shen et al., 2014; Zamani et al., 2018). Models of the interaction-focused type exploit interactions between query and document terms (Guo et al., 2016; Xiong et al., 2017; Dai et al., 2018; Nogueira and Cho, 2019). Although this leads to superior ranking quality (Guo et al., 2016, 2020; Qiao et al., 2019), it is computationally much more intensive, since a given query has to be processed together with each potentially relevant document. Hence, this type of neural ranking model is only applicable in a ranking pipeline, where limited numbers of documents are given as potentially relevant candidates. These candidates are selected by an efficient retrieval model that is able to retrieve documents directly from the corpus in a reasonable amount of time. The multi-stage ranking technique is also known as cascade ranking and optimizing the configuration of such a pipeline towards maximized efficiency and effectiveness has been extensively studied in the past (Wang et al., 2011; Chen et al., 2017). Many interaction-focused neural ranking models employ a dedicated layer to explicitly perform a matching between query and document terms (Guo et al., 2016; Xiong et al., 2017; Dai et al., 2018). Another approach is using the attention mechanism (Vaswani et al., 2017), or more specifically, a pretrained transformer encoder such as BERT (Devlin et al., 2019) to exploit both local and query-document interactions (Nogueira and Cho, 2019; Qiao et al., 2019). Recently, some hybrid approaches have been proposed that combine typical representation-focused techniques with interaction-focused approaches to reduce computational cost: Gao et al. (2020) propose a model architecture comprising three modules for document understanding, query understanding and relevance judging respectively. The understanding modules produce token-level representations, which can be cached as usual in representation-focused approaches. The relevance judging module uses those cached representations to apply query-document interactions more quickly. Each module is a stack of transformer layers (Vaswani et al., 2017), initialized with weights from BERT. In a related approach, MacAvaney et al. (2020) investigate the relationship between different numbers of dedicated layers of BERT for query-document interactions and measure the resulting speedup that is due to token representation caching, as well as its impact on the end-to-end ranking quality. Khattab and Zaharia (2020) propose a related approach, namely ColBERT . The model architecture incorporates an inexpensive max-similarity mechanism to perform token-level query-document interactions. The authors propose to store token representations in an Approximate Nearest Neighbor (ANN) index to quickly retrieve only those documents that have token representations in the proximity to those of the query. Thus, ColBERT can be described as an end-to-end ranking approach that brings its own first-stage retrieval mechanism allowing to perform end-to-end ranking in a reasonable amount of time. Now we discuss neural ranking approaches that can be used to retrieve passages or documents directly from an entire corpus in a reasonable amount of time and thus qualify for first-stage ranking. Many proposed methods make use of existing infrastructure for sparse bag-of-words retrieval or at least inverted indexing (Manning et al., 2008). Zamani et al. (2018) propose SNRM , a representation-focused approach with sparse representations that can be used with an inverted index as if each feature dimensions corresponds to a term in a bag-of-words representation. SNRM uses pretrained GloVe Word Embeddings (Pennington et al., 2014) to model soft-matched n-grams which are encoded in sparse representations. Nogueira et al. (2019) predict queries for given documents to expand those documents by corresponding query terms. In their first work, known as doc2query , they used a sequence-to-sequence (seq2seq) transformer model (Vaswani et al., 2017). In a subsequent work, Nogueira and Lin (2019) reported large effectiveness gains for their follow-up model docTTTTTquery by replacing the seq2seq model with T5 (Raffel et al., 2020). Another approach aims at predicting optimal document term weights as a function of the term's context. DeepCT , proposed by Dai and Callan (2020), utilizes BERT to predict these context-aware weights based on associated queries in the training data. Inverted indexing is only applicable to sparse representations. Representation-focused models using dense representations can instead employ an ANN index, which heuristically prunes documents that are unlikely to be in the topk proximity of the query representation to realize low response latencies (Boytsov et al., 2016; Gysel et al., 2018). Karpukhin et al. (2020) recently used this technique in combination with a fine-tuned BERT encoder for open question answering. We describe a first-stage ranking model that acts as a complementary ranker to existing term-based retrieval models such as BM25. To achieve this, we make use of a transformer-based pretrained language model and its inherent ability to make use of token-level local interactions. Its complementary behavior is further supported by negative sampling from BM25 rankings. The model architecture of CoRT, illustrated in Figure 1, follows the idea of a Siamese Neural Network (Bromley et al., 1993). Passages and queries are encoded using an identical model with shared weights except for one detail: The passage encoder   and the query encoder   use different segment embeddings (Devlin et al., 2019). CoRT computes relevance scores as angular similarity between query and passage representations while training a pairwise ranking objective. CoRT can incorporate any BERT-like encoder as underlying text encoder. Here, we use a pretrained ALBERT (Lan et al., 2020) encoder for its smaller model size, the tougher sentence coherence pretraining and increased first-stage ranking quality throughout our early-stage experiments compared to BERT. The tokenizer of ALBERT is a Word-Piece tokenizer (Wu et al., 2016) including the special tokens [CLS] and [SEP] known from BERT. From the text encoder we seek a single representa-Figure 1: CoRT's model architecture and pair-wise learning objective (simplified). tion vector for the whole passage or query, which we call context representation . From ALBERT we take the [CLS] embedding of the last layer for this purpose. We denote the context representation obtained from the underlying encoder for an arbitrary string s with  ( s )  R h where h is the output representation size. ALBERT's language modeling approach involves sentence coherence prediction for which segment embeddings are used to signal different input segments. Although we only feed single segments to the encoder, i.e. a query or a passage, we use segment embeddings which allow the model to encode queries differently from passages. The segment embeddings EA and EB (illustrated in Figure 1) are part of the context encoder functions   and   for passages and queries respectively. The context representation is further projected to the desired representation size e using a linear layer followed by a tanh activation function. Thus, the complete passage encoder function is   ( s ) := tanh( W   ( s ) + b ) where W  R h  e and b  R e are parameters of the linear layer. The query encoder   is defined analogous. Training CoRT corresponds to updating the parameters of the encoder  towards representations that reflect relevance between queries and passages through vector similarity. Each training sample is a triple comprising a query q , a positive passage d + and a negative passage d  . While positive passages are taken from relevance assessments, negative passages are sampled from term-based rankings (i.e. BM25) to support the complementary property of CoRT. The relevance score for a query-passage pair ( q, d ) is calculated using the angular cosine similarity function 2 : 2 Similar to Cer et al. (2018), we found angular similarity performs better than cosine similarity. As illustrated in Figure 1, the training objective is to score the positive example d + by at least the margin  higher than the negative one d  . As part of our loss function, we use the triplet margin objective: l ( q, d + , d  ) := max (0 , sim ( q, d  )  sim ( q, d + ) +  ) Inspired by Song et al. (2016), we aim to take full advantage of the whole training batch. For each query, each passage in the batch is used as a negative example, except for the given positive passage. Thus, we define our batch-wise loss function as follows: L := (cid:88) 1  i  n   (cid:88) 1  j  n l ( q i , d + i , d  j ) + (cid:88) 1  k  n, k (cid:54) = i l ( q i , d + i , d + k )   q i , d + i and d  i denote the triple of the i th sample in the batch and n the number of samples per batch. We found this technique to make the training process more robust against exploding gradients. Otherwise we need to employ gradient clipping (Zhang et al., 2020) to stabilize the training process. Also, it positively affects first-stage ranking results 3 . For retrieval with CoRT, each passage must be encoded by the passage encoder   . Subsequent normalization of each vector allows us to use the dot product as a proxy score function for sim , which is sufficient to form rankings accurately. Given a query q , we calculate its representation   ( q ) and the dot product with each normalized passage vector. From those, the k highest scores are selected and sorted to form the CoRT ranking. This procedure can be implemented heavily parallelized using GPU matrix operations. Alternatively, the passage representations can be indexed in an ANN index to avoid exhaustive similarity search. In contrast to the first-stage ranking of Khattab and Zaharia (2020) and MacAvaney et al. (2020), we 3 We achieve 2.0 p.p. higher MRR@10 compared to the plain triplet margin loss on the MS MARCO passage task. only index one representation per passage rather than one per token. Finally, we combine the resulting ranking of CoRT with the respective BM25 ranking by interleaving the positions beginning with CoRT to create a single merged ranking of equal length. During this process, each passage that was already added by the other ranking is omitted. For example, merging two ranking lists beginning with [ a, b, c, d, . . . ] and [ e, c, f, a, . . . ] would result in [ a, e, b, c, (cid:1) c, f, d, (cid:1) a, . . . ] . The interleaving procedure stops as soon as the desired ranking size has been reached. The result is a compound ranking of CoRT and BM25, which we denote with CoRT BM25 . We present four experiments studying the ranking quality and recall of CoRT, the connection between the number of candidates and re-ranking effectiveness, the impact of the representation size e , and CoRT's retrieval latencies. Finally, we outline a competitive end-to-end ranking setup with CoRT and a BERT-based re-ranker. The Microsoft Machine Reading Comprehension (Nguyen et al., 2016) dataset for passage ranking was introduced in 2018. It provides a benchmark for passage retrieval with real-world queries and passages gathered from Microsoft's Bing search. The MS MARCO passage ranking task comprises 8.8M passages sampled from web pages and about 1M queries that are formulated as questions. The objective is to rank those passages high that were labeled as relevant to answer the respective question. The annotations, however, are sparse. There are 530k positive relevance labels distributed over 808k queries in the training set, whereby most queries are associated to one passage. The validation and evaluation sets, dev and eval , comprise 101k queries each. An official subset of dev , called dev.small comprises 6980 queries and 7437 relevance labels and is often used for publicly reported evaluations. We follow this convention and use dev.small for testing. The creators suggest to use the mean reciprocal rank cut at the tenth position (MRR@10) as primary evaluation measure. Additionally, we measure NDCG@20 (Manning et al., 2008) as less punishing ranking quality measure and the recall at various positions Table 1: First-stage ranking results on the MS MARCO dev.small set. The passage retrieval section of the TREC 2019 Deep Learning Track (Craswell et al., 2020) provides on average 215 manual relevance assessments per query for a set of 43 MS MARCO queries. Each assessment corresponds to a rating on a scale from 0 (not relevant) to 3 (perfectly relevant). We adopt the evaluation metrics MRR (uncut), NDCG@10 and MAP from the official TREC overview. In contrast to the original MS MARCO benchmark, this evaluation set provides dense annotations, but only for few queries. We train CoRT as described in Section 3.3 while using a representation size of e = 768 . In this section we discuss the first-stage ranking results of our model using the datasets and their associated metrics described in Section 4.1. The results of CoRT and its baselines on the MS MARCO passage retrieval task ( dev.small ) are reported in Table 1. Next to BM25 as a baseline, we include DeepCT (Dai and Callan, 2020), doc2query (Nogueira et al., 2019) and its successor docTTTTTquery (Nogueira and Lin, 2019). All three are recent first-stage rankers with average retrieval latencies below 100ms per query on the MS MARCO passage corpus. The metrics MRR@10 and NDCG@20 reveal a quite decent ranking quality for the standalone CoRT ranker. Since CoRT's primary use is candidate retrieval rather than standalone ranking, we pay particular attention to the recall at various cuts. From the perspective of BM25, the absolute increase of recall due to merging with CoRT ranges between Table 2: First-stage ranking results on the TREC 2019 DL passage task. 15.1 (RECALL@50) and 9.2 (RECALL@1000), which we consider the complementary portion of CoRT. Greater increases of recall can be noticed for lower cuts, which is particular useful when re-ranking is performed with low numbers of candidates. The top-200 candidates from CoRT BM25 comprise higher recall than the top-1000 candidates from BM25. The metrics for DeepCT and docTTTTTquery have been calculated using published top-1000 rankings from the respective authors. Thus, we were able to merge those rankings with CoRT. However, the used instance of CoRT was trained on BM25 and not on the external ranker. Although we consider the relevance assessments from TREC 2019 DL to be dense, we found 112 unlabeled passages among the 43 top-10 CoRT rankings while the assessments for the BM25 rankings are complete. This means there are, on average, 2.6 unlabeled passages in the top-10 CoRT rankings, which might make this evaluation somewhat unfavourable for CoRT and explain the drop in recall. Still, Table 2 shows superior results for CoRT compared to BM25 in terms of ranking quality (MRR, NDCG and MAP). Merging CoRT with BM25 slightly increases MRR and NDCG, while a decent gain in terms of recall can be noticed. We can not report any results for DeepCT or docTTTTTquery , since only rankings for the MS MARCO dev.small set are available online from the respective authors. We re-rank candidates from both BM25 and CoRT BM25 to study the impact of the candidates on a subsequent interaction-focused re-ranking. By varying the numbers of candidates, we investigate at which point adding more candidates becomes ineffective. Similar to (Nogueira and Cho, 2019), we use a simple binary classifier based on BERT. The query-passage pair ( q, p ) is concatenated to one token sequence of two segments. This sequence is processed by the BERT encoder while the [CLS] embedding of the last layer, which we denote with  ( q, p ) , is projected to a single classification logit. We then apply the sigmoid activation function  to obtain the relevance confidence for query q and passage p . This procedure can be formalized as  ( q, p ) =  ( W (cid:48)  ( q, p ) + b (cid:48) ) where W (cid:48)  R h  1 and b (cid:48)  R are the parameters of a linear layer with a single output activation. To form a ranking at inference time, we sort the candidates by the model's confidence. Following (Nogueira and Cho, 2019), this model is trained using a point-wise objective. We sample query-passage pairs, each associated with a binary relevance label y  { 0 , 1 } and minimize the binary cross-entropy loss: l (cid:48) ( q, p, y ) = y  log  ( q, p )+(1  y )  log (1   ( q, p )) 4.3.2 Re-ranking Results As illustrated in figure 2, using CoRT BM25 as first-stage ranking appears to result in superior end-to-end ranking quality in terms of MRR@10. This is especially true, if low numbers of candidates are used. We also notice earlier saturation 4 of MRR@10 for CoRT BM25 , which is illustrated in Figure 2. Only 64 candidates from CoRT BM25 are sufficient to achieve top results with this re-ranker. In contrast, 256 candidates from BM25 are needed to reach the point of saturation, which translates in quadrupled re-ranking time. 4 In our definition, saturation is reached, when doubling the number of candidates results in less than 0 . 5% increase of the respective metric 8 16 32 64 128 256 512 1000 Number of Re-ranking Candidates 27 29 31 33 35 37 39 MRR @ 10 CoRT BM25 + BERT BM25 + BERT Saturation Point Figure 2: Re-ranking quality by number of candidates for BM25 (black) and CoRT BM25 (red) on the MS MARCO passage task. As described in Section 3.2, CoRT projects the context representation of the underlying encoder  to an arbitrary representation size e . This size determines the size of the final index and also influences the retrieval latency. The total size of the encoded corpus is easy to calculate. For example, with e = 128 and the MS MARCO corpus, the index size (without overhead) amounts 8 . 8 M documents  128 floats/document  4 bytes/float  4 . 5  10 9 bytes  4 . 2 GB . Thus, e is proportional to the total size and reducing e to 64 would halve the memory footprint. If e is small, however, it is more difficult to attain the training objective. Thus, e can be used for a trade-off between ranking quality and computational effort / resource cost. We investigate the relation between the representation size e and the ranking quality by conducting identical training runs with different numbers for e . The results in Table 3 show that MRR@10 already saturates at e = 128 . Interestingly, even with an representation size of e = 32 , CoRT outperforms BM25 in terms of MRR@10 and nDCG@20 by a big margin. We propose two methods for the deployment of CoRT. The first exhaustively calculates similarity scores using multiple GPUs while the second incorporates an Approximate Nearest Neighbor index (ANN). We measure retrieval latencies of those methods and compare them with BM25 as representative for term-based retrieval models based on inverted indexing. We conduct the latency measurement based on the top-1000 retrieval for the dev.small split of the MS MARCO passage corpus. Since some approaches profit from batch computing, we also measure the latency for batches of 32 queries. As representation size, we have cho-sen e = 128 , since it is the smallest representation size investigated in Section 4.4 that does not hurt the ranking quality of CoRT BM25 . As retrieval latency baseline, we use a Lucene index generated by the Anserini toolkit (Yang et al., 2017). Please note, this is not perfectly representative for sparse bag-of-words retrieval: Retrieval latency can be reduced due to index pruning without significantly hurting retrieval quality (Mackenzie et al., 2020). The retrieval was performed on a machine with an Intel Core i9-9900KS processor (16 logical cores, 8 physical) and enough memory to hold the whole corpus. Single queries were processed using the single-threaded search function, while batch-wise search has been performed with 16 threads. Multiple GPUs can be used to deploy CoRT for fast large-scale ranking. We propose to uniformly distribute the vector representations of the corpus on the available GPUs. Each GPU ranks its own partition of the corpus as described in Section 3.4. Afterwards, the results for each partition are aggregated by selecting the top-k candidates with highest scores. Since CoRT operates on vector similarities, it can make use of ANN search. We measure the retrieval latency and the loss of recall, which occurs due to the pruning heuristics. We use a graph-based index optimized with the ONNG method (Iwasaki and Miyazaki, 2018). An implementation of this method is publicly available as part of the NGT Table 4: Retrieval latencies averaged over 6980 queries of the dev.small split. Library 5 . To adjust the trade-off between retrieval latency and accuracy of the index, we alter the search range coefficient (cid:15) . We always retrieve 1000 candidates from the ANN index, even if we use a smaller number of candidates in a ranking pipeline. 4.5.4 Latency Measurements The latency measurements are reported in Table 4. For CoRT the total retrieval latency per query consists of two factors: Query encoding and retrieval. The query encoding has to be performed by the query encoder   , which we highly recommend to run on a GPU. The latency of the retrieval depends on the retrieval methods described above. Employing multiple GPUs appears to reduce retrieval latency on a linear scale: The exhaustive search using four GPUs takes 17ms for a single query, while a single GPU takes four times as long. The total retrieval time per query sums up to 17 + 8 = 25 ms for the quad GPU setting, which is below the BM25 baseline. It is worth noting, that batch-wise processing results in a substantial efficiency increase: Retrieval for 32 queries at once only takes about twice as long as a single query. This can be useful if multiple queries queue up while the system is busy. The tested BM25 index (Anserini), on the other side, seems to suffer from multiprocessing overhead or some sort of bottleneck. The latencies for the ANN index has been measured with three different values for the search range coefficient (cid:15) . While this significantly affects the retrieval latency, only slight differences regarding the quality of the first-stage ranking are 5 https://github.com/yahoojapan/NGT observed. Latencies for CoRT BM25 comprise latencies from BM25, CoRT's query encoding and the corresponding retrieval method. For simplicity, we assume sequential processing of all three components, although BM25 could be processed in parallel. Intrigued by the remarkable ratio of retrieval latency and ranking quality of ColBERT's full-ranking approach (Khattab and Zaharia, 2020), we used our above findings to create a competitive end-to-end ranking setup. We suggest to re-rank the top-64 candidates from CoRT BM25 with e = 128 , retrieved by an ANN index ( (cid:15) = 0 . 1 ). The end-to-end latency comprises 8 ms for query encoding, 17 ms for CoRT retrieval based on ONNG, 38 ms for BM25 retrieval, and 192 ms for re-ranking. Although the BM25 candidates could be retrieved in parallel, we report sequential processing latencies. As shown in Table 5, we outperform ColBERT's end-to-end ranking performance in terms of MRR@10 and retrieval latency. It is worth noting that the "RTX 2080 Ti" we used for latency measurements is less powerful than the "Tesla V100" Khattab and Zaharia (2020) used for their measurements. CoRT's representations for the MS MARCO corpus only weight 4.3GB when e is set to 128, or 7.0GB when indexed in an ONNG index. The size of the query encoder only amounts about 50MB, which is due to ALBERT's parameter sharing. To compile the full CoRT BM25 candidates, the corresponding BM25 index is needed, which amounts 2.2 GB on disk. Although more memory is needed to deploy and operate both indexes, this is by far less than the 154GB footprint reported by Khattab and Zaharia (2020) for ColBERT's end-to-end approach. In this paper, we propose CoRT, a framework and neural first-stage ranking model that leverages contextual representations from transformer-based language models to complement term-based ranking functions. As a result, we observe decently increased recall measures and improved end-to-end ranking quality on the MS MARCO passage task. Also, we are able to decrease the number of candidates for re-ranking without hurting the final performance. Our further experiments reveal sweet spots for CoRT's representation size and the number of re-ranking candidates. We presented two deployment strategies for CoRT and measured their performances in terms of efficiency and effectiveness. Finally, we demonstrate CoRT can be used with a simple BERT-based re-ranker to create a competitive ranking pipeline. We would like to thank Felix Hamann and Prof. Dr. Adrian Ulges for helpful discussions and comments on the manuscript, as well as the anonymous reviewers for their valuable feedback. This work was funded by German Federal Ministry of Education and Research (Program FHprofUnt, Project DeepCA (13FH011PX6)).
abstain abstain abstain objective result result objective abstain abstain abstain abstain abstain abstain abstain method objective objective abstain objective method objective objective abstain objective abstain objective other other other other other other other other other other other other other other other other other other other objective other other other other other other other other other other other method method abstain abstain abstain abstain abstain method abstain method method method method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method method abstain abstain method method abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain objective result abstain result method objective other other