The ability to integrate context, including perceptual and temporal cues, plays a pivotal role in grounding the meaning of a linguistic utterance. In order to measure to what extent current vision-and-language models master this ability, we propose a new multimodal challenge, Image Retrieval from Contextual Descriptions (IMAGECODE ). In particular, models are tasked with retrieving the correct image from a set of 10 minimally contrastive candidates based on a contextual description. As such, each description contains only the details that help distinguish between images. Because of this, descriptions tend to be complex in terms of syntax and discourse and require drawing pragmatic inferences. Images are sourced from both static pictures and video frames. We benchmark several state-of-the-art models, including both cross-encoders such as ViLBERT and bi-encoders such as CLIP, on IMAGECODE . Our results reveal that these models dramatically lag behind human performance: the best variant achieves an accuracy of 20.9 on video frames and 59.4 on static pictures, compared with 90.8 in humans. Furthermore, we experiment with new model variants that are better equipped to incorporate visual and temporal context into their representations, which achieve modest gains. Our hope is that IMAGECODE will foster progress in grounded language understanding by encouraging models to focus on fine-grained visual differences. We make code and dataset publicly available. 1 1 Introduction Natural languages are highly contextual (Fodor, 2001): for a listener, recovering the speaker's intended meaning requires integrating information from different streams, such as grounding in perception (Pecher and Zwaan, 2005), shared world knowledge, and temporal reasoning (Wilson and Sperber, 1998). These processes, more generally, 1 https://github.com/McGill-NLP/imagecode fall under the umbrella term of pragmatics (Grice, 1957). Despite recent progress in multimodal systems, it remains unclear to which extent they can handle settings where context plays a major role, such as in real-world communication. To this end, we present a new challenge that requires multimodal models to leverage context to retrieve images from text. In particular, given a contextual description and a set of minimally contrastive candidate images, i.e. differing only in some details, the model has to retrieve the target image. In order to discriminate between similar images, human annotators naturally produce highly nuanced and grammatically complex descriptions. An example of our new challenging dataset, Image Retrieval from Contextual Descriptions (IMAGECODE ), is shown in Figure 1. During the data collection process, sets of similar images are selected among static pictures from Open Images (Kuznetsova et al., 2020) and (a larger portion) among video frames from diverse domains. Including both types of images allows for diversifying the dataset while representing different degrees of visual similarity within each set. Next, we crowdsource a contextual description of a target image (presented together with the rest of the set) that contains only differences relevant for retrieval. After a filtering phase involving human retrievers, we obtain a large-scale dataset with 94,020 images and 21,202 descriptions associated with image sets of size 10. As a result of this annotation protocol, successfully completing the task requires models to integrate several kinds of context: i ) the image set, as the descriptions often only make sense in the context of several other images and are not suitable as stand-alone captions. In fact, aspects of the image that are very salient and that therefore would normally be emphasized are not useful in our proposed task. Instead, the focus of our descriptions are fine-grained details that help discriminate between images (see Figure 1); ii ) the speaker's intention. Due to their high degree of image similarity, contextual descriptions may be literally true for multiple images; however, once the speaker's intention is taken into account, the correct image can be determined by virtue of pragmatics, i.e. Grice's maxim of quality 2 (see Figure 2, Figure 7); iii ) temporal sequences: for video frames temporal reasoning is also required to compare different moments of an unfolding event. On our new dataset IMAGECODE , we benchmark a series of vision-and-language models that achieve state-of-the-art performance on other multimodal tasks, specifically ViLBERT (Lu et al., 2019) and UNITER (Chen et al., 2020) as two cross-encoder variants and CLIP as a strong bi-encoder (Radford et al., 2021). We report several findings. First, accuracy on static images is vastly superior than on video frames. Therefore, the degree of similarity among the candidate images has an overwhelming impact on retrieval performance. Second, all state-of-the-art models generally struggle with image retrieval from contextual descriptions, whereas humans consistently achieve high accuracy. Hence, we propose model variants capable of better taking context into account: i ) once an image-description pair is encoded, we refine this representation by attending to the other images in the set; 2 Note: While we do not model pragmatics explicitly in our baselines, we find that the IMAGECODE contains many examples suitable for pragmatic modeling ii ) we augment image encodings with temporal embeddings. Based on our results, models take advantage of this additional information fruitfully but only to a limited degree. Because of its challenging nature, due to the minimally contrastive images and complex descriptions, we believe that IMAGECODE will help make visio-linguistic models more context-aware and sensitive to fine-grained details. There is a long tradition of grounding language understanding on single images, in the form of visual question answering (Goyal et al., 2017; Hudson and Manning, 2019), visual dialogue (de Vries et al., 2017; Das et al., 2017), or visual entailment (Xie et al., 2019). Recently, more and more focus has been directed to settings where the visual context consists of multiple images, either conventional static pictures (Vedantam et al., 2017; Hu et al., 2019; Suhr et al., 2019; Forbes et al., 2019; Hendricks and Nematzadeh, 2021; Yan et al., 2021; Hosseinzadeh and Wang, 2021; Bogin et al., 2021; Liu et al., 2021), or video frames (Jhamtani and Berg-Kirkpatrick, 2018a; Bansal et al., 2020). While many of these benchmarks involve just two images, COVR (Bogin et al., 2021) and ISVQA (Bansal et al., 2020) provide more images, similar to our sets of 10 images. ISVQA and Spot-the-diff (Jhamtani and Berg-Kirkpatrick, 2018a) are most similar to our dataset, IMAGECODE . ISVQA is based on several video frames that are synthetic and cover a restricted do-main, with short questions for Visual Question Answering. Spot-the-diff provides two frames from surveillance video cameras and descriptions of all their differences. IMAGECODE is unique as a) we cover a wider range of domains; b) we construct image sets that are maximally similar while being distinguishable through natural language (Section 3) and c) we limit descriptions to relevant differences. This results in (a) diverse, (b) complex and (c) pragmatically informative descriptions. We do not claim to explicitly model pragmatics in this paper, i.e. with Rational Speech Acts (Goodman and Frank, 2016). Instead we present a dataset that is naturally suitable for pragmatic reasoning (Andreas and Klein, 2016; Cohn-Gordon et al., 2018) as a listener has to consider the context, assume a Gricean speaker and resolve ambiguities resulting from nuanced differences. The 3427 reasoning in our task and data collection is therefore also similar to ReferItGame and subsequent work (Kazemzadeh et al., 2014; Mao et al., 2016) where one crowdworker generates a referring expressing for an object in a single image and another worker picks an object based on the expression. Our data collection involves two steps with a human describer and retriever. The describer is given a set of 10 highly similar images S = [ I 1 , I 2 , ..., I 10 ] , one of them marked as the target image I t , and has to write a description D that clearly distinguishes I t from the other distractor images. In the second step, the retriever is given the same 10 images and the description from the first step and has to identify the target image based on the description. S and D are only added to our dataset if the retrieval is successful. Below, we outline the main stages of data collection: first, the collection of similar, contrastive images in Section 3.1. Then, the crowdsourcing of contextual descriptions in Section 3.2 and validation of the examples via image retrieval (Sec-tion 3.3). The final IMAGECODE dataset consists of 94,020 images (partitioned into 9,402 sets) and 21,202 contextual descriptions (16,594 in the train split, 2,302 and 2,306 in the validation and test split respectively). In the first stage, we collect sets of images that are highly similar but still distinguishable from each other by a human. To quantitatively measure the pairwise similarity of two images, we compute the Euclidean distance between their encodings extracted from a pre-trained CLIP model (Radford et al., 2021). 3 To study the effect of different degrees of similarity, further variegate our dataset, and enable temporal reasoning, we source our candidate images from collections of static pictures as well as videos, as detailed below. Static Pictures. We obtain image sets from one of the largest repositories of static pictures, the Open Images Dataset V6 (Kuznetsova et al., 2020), containing 1.74M images. For each image, we retrieve the 9 closest images from the training set based on their CLIP encodings. We then randomly sample 4,845 of these image sets. 3 We also experimented with ResNet-50 features, but we found CLIP results to be more similar to that of humans in preliminary experiments. Video Frames. As sources for our video frames, we use i ) Video-Storytelling (Li et al., 2019), covering social events (wedding, birthday, Christmas, camping); ii ) general-domain MSR-VTT (Xu et al., 2016); and iii ) YouCook (Das et al., 2013), covering cooking events. We choose these datasets as they contain publicly available and general-purpose videos (not specific to downstream tasks). We retain the original splits for train, validation, and test. To obtain disjoint sets of 10 similar frames, we first segment the videos into smaller scenes (also known as shots) via the scene detection functionality of ffmpeg (Tomar, 2006). Then, for each scene, we add its first frame to the set of selected images. We then iterate over every following frame and add it to the set if its pairwise Euclidean distance with each of the previously selected frames is larger than a threshold. 4 Once the set contains 10 images, we reiterate the procedure for a new set. If the scene ends and the current set contains less than 10 images, the set is discarded. During this process, we additionally remove frames that i ) are too blurry, i.e. their BRISQUE score (Mittal et al., 2012) is larger than 0.65; or ii ) contain too much text, which is detected with the OCR tool Tesseract (Smith, 2007). 5 We use all of YouCook's image sets and (due to cost constraints) randomly sample image sets from Video-Storytelling and MSR-VTT for crowdsourcing (cf. Table 1). We remark that image sets are further filtered at the final stage of annotation (Section 3.3). After creating sets of highly-similar images in Section 3.1, we request annotators from Amazon Mechanical Turk (AMT) to write contextual descriptions for each target image in a set. Each round, a set of images is presented in random order for static pictures and respecting temporal order for 4 The distance threshold was manually chosen as 0.35 based on qualitative results. video frames. This encourages annotators to take the dynamics of the event into account. We then (randomly) select 3 target images per set, and ask annotators to produce a description that discriminates them from the other images in the set. To encourage pragmatic reasoning, we do not ask for all the differences (just those sufficient for retrieval) and do not allow explicit mentions of other images (see Figure 2). We select high-quality annotators according to criteria in Appendix B and assign partly disjoint sets of annotators to train and test in order to avoid annotator bias (Geva et al., 2019). 6 3.3 Human Validation via Image Retrieval Finally, we validate the annotation crowdsourced in Section 3.2 by asking AMT workers to retrieve the correct target image from a set given its contextual description. For the final dataset, we retained only the examples that i) were retrieved successfully in the training set by a single worker or ii) were retrieved successfully by at least 2 out of 3 workers in the validation and test sets. As a consequence, we filtered out 26.5% of the contextual descriptions generated in Section 3.2. Table 1 compares the number of examples retained at each stage throughout the dataset creation. 7 6 For further details on crowdsourcing instructions, analysis of annotator bias and the AMT interface, please refer to Appendix C and Appendix D. 7 Again, the set of workers validating train and test sets were partly disjoint to avoid annotator bias. To quantify the reliability of the process outlined in Section 3, we report the inter-annotator agreement on our final dataset in Table 3. We use Krippendorff's  as a metric (the higher the better), which accounts for incomplete data, since the number of annotators per example is not fixed. We treat the index of the target image either as a nominal variable for static images or as an ordinal variable for video frames. In both cases, we find a high degree of agreement. Moreover, in Table 3, we also report human accuracy the percentage of times an annotator retrieved the correct target image from a contextual description (as described in Section 3.3). This provides an upper ceiling for the model performances (see Section 6). In Table 4, we measure a series of statistics of the descriptions collected for IMAGECODE and compare them with other vision-and-language datasets with multiple naturalistic images (cf. Section 2), such as NLVR2 (Suhr et al., 2019) and Spot-the-diff (Jhamtani and Berg-Kirkpatrick, 2018b). 8 In particular, we count the average description length, the number of distinct word types, the average dependency tree depth of each sentence, 9 and the average number of sentences per description. Based on these metrics, we find evidence that IMAGECODE 's descriptions are longer and more syntactically complex than in the other datasets. Moreover, they include multiple sentences (11.8% of examples have 3 or more). By calculating the average pairwise Euclidean distance between CLIP-based encodings of images in the same set, we find that video frames are more similar than static pictures  as expected  by a fac-tor of 1.13. Moreover, we find that descriptions of video frames mention human body parts (72.1%) more often than static pictures (30.2%). On the other hand, names of colors appear in descriptions of static pictures (61.4%) more frequently than 8 For comparability, we measured the statistics for all the datasets with the same tools. video frames (33.6%). 10 Thus, annotators resort to different strategies to discriminate between different types of image sets, focusing on the aspects that vary the most. Finally, we identify 9 interesting and challenging phenomena in IMAGECODE and annotate whether they are present in 200 examples from the validation set. We provide the definition of each phenomenon, its frequency, and an illustrative example in Table 2. An example for each phenomena is given in Appendix G. For 4 of these phenomena unique to IMAGECODE , we further annotated 800 examples for the purpose of error analysis in Section 6. Inspecting these examples, we find a high number of cases where the visual context (47.0%) is required to complete the task. For instance, consider Figure 2: the description No bridesmaid visible at all. requires a retriever to resolve the co-references of the entities in 5 frames. In particular, the body parts of the bridesmaids (red boxes) visible in frames 2 and 4 would not be identifiable as such without frame 1 and 5, respectively (where they appear with matching dresses and flowers in their hands). A common example we find in the data are "gradable" scenarios, i.e.  The person is looking down  might be semantically true for more than one image but it fits best to the image where the person is looking down the most. Another group of phenomena characteristic for IMAGECODE originates from its minimally contrastive setup: annotators might focus on how an 10 We calculated these percentages based on a list of 171 body parts in English collected by Tjuka (2021) and a list of colors in English from games4esl.com. event unfolds over time ( temporal context ), on what is missing in a specific frame but visible in the others ( negation ), on what moved out of frame ( visibility / occlusion ), or on small regions and patches of pixels ( nuances ). Importantly, these phenomena are less prominent in static pictures than in video frames (cf. Table 2). In order to assess whether vision-and-language models can retrieve the correct image from a contextual description on a par with humans, we benchmark three state-of-the-art models that represent three main families of multimodal architectures (Bugliarello et al., 2021; Miech et al., 2021): i ) ViLBERT, a cross-encoder where language and vision streams can interact via cross-attention at intermediate layers (Lu et al., 2019); ii UNITER, a single-stream encoder where language and vision tokens are concatenated as inputs and processed with a single Transformer (Chen et al., 2020); iii ) CLIP, a bi-encoder where language and vision streams are independent (Radford et al., 2021). It is worth noting that ViLBERT and UNITER are more expressive due to their architecture, whereas CLIP boasts a higher parameter count, is pre-trained on a larger dataset and uses a contrastive objective. We evaluate these models under two different regimes: i) zero-shot inference, where pre-trained models are deployed on the IMAGECODE test set directly; and ii) fine-tuning , where the models are refined on the full training set before evaluation. We cast the training objective as binary classification for ViLBERT and as 10-class classification for CLIP. 11 Crucially, in both cases, positive and negative examples during training are sampled at random independently from the image set they belong to (see the first column of Figure 3). Thus, the visual context of the other images in a set is only indirectly accessible at inference time, where the image with the highest probability is predicted. and temporal context into the model. First, we use an alternative objective where all three models are trained on 10-class classification, but the 1 positive and 9 negatives are sourced from the same image set. The consequence of including positive and negative examples from the same image set in the same mini-batch is providing a wider visual context. We refer to this variant as +C ONTEXTBATCH (second column of Figure 3). This setup only conveys the visual context as a weak signal, since the model has no chance to directly compare the images in the same set. Hence, we experiment with enhancing the architecture of vision-and-language models with a mechanism inspired by Bogin et al. (2021). In particular, given an encoder (CLIP, ViLBERT or UNITER), we obtain the representations of a contextual description x L  R e (where e is the model hidden size) and of the images in a set ( x ( 1 ) V , . . . , x ( 10 ) V ) , x ( i ) V  R e from their final layer. 12 Then, we create a series of multimodal embeddings via element-wise multiplication: m = ( x L  x ( 1 ) V , . . . , x L  x ( 10 ) V ) . Finally, we feed these to a l -layer Transformer Tf  R 10  e  R 10  e to obtain context-aware multimodal embeddings ( Tf ( m ) 1 , . . . , Tf ( m ) 10 ) . Since each descriptionimage pair can now attend on the others in a set, the model can fully exploit the visual context. We obtain the score for the i -th pair through a linear classifier head W  R 1  e . The target image is predicted as arg max i softmax [ W ( Tf ( m ) i + m ( i ) )] (1) Note that we add a highway layer from the input to the output of the Transformer. We label this model variant +C ONTEXTMODULE . Finally, in addition to visual context, we make models aware of the temporal context too, as shown in the fourth column of Figure 3. For video-based examples only, the multimodal embeddings of each description-image pair are summed with a learnable positional embedding t  R e that reflects the temporal order of the frames. 13 Thus, m = ( x L  x ( 1 ) V  t ( 1 ) , . . . , x L  x ( 10 ) V  t ( 10 ) ) . Multimodal embeddings are then fed to a Transformer as above. We label this variant encapsulating both visual and temporal context +T EMPORALEMBEDDINGS . For all CLIP experiments, we use a pre-trained model with the vision backbone VI T-B/16. 14 We train the full models with a batch size of 360 examples (i.e., 36 image sets) for CLIP and 150 examples for ViLBERT/UNITER. We perform early stopping based on the validation accuracy with a maximum of 30 epochs. In the variants that adopt the base version of a model, we select a learning rate of 4  10  6 for CLIP, 5  10  6 for ViLBERT, 4  10  5 for ViL-BERT+C ONTEXTBATCH , 8  10  6 for UNITER, and 7  10  6 for UNITER++C ONTEXTBATCH . We find these values via hyper-parameter search on the range [ 10  4 , 10  7 ] . For CLIP variants that modify the model architecture, we adopt the following setup: first, we fine-tune the full model in the +C ONTEXTBATCH regime as detailed above. Afterwards, we freeze the encoder parameters and train the components responsible for processing the multimodal embeddings, described in Equation (1). More details are provided in Appendix F. For ViLBERT and UNITER we finetune the whole architecture at the same time. All descriptions in IMAGECODE exceeding the maximum length of the three models are truncated. Due to their negligible amount, this does not affect 14 https://github.com/openai/CLIP performance significantly. In Table 5, we report the performance of the models from Section 5 for all the test examples in IMAGECODE as well as for the subsets containing only video frames or static pictures (see Appendix E for validation scores). Note that the random chance baseline has an accuracy of 10%. In what follows, we compare the results across several dimensions. Zero-shot vs. fine-tuning. In the zero-shot setting, we observe that CLIP representations are surprisingly superior to UNITER/ViLBERT even though CLIP has separate streams to encode an image and its description. In the simplest fine-tuning setting (i.e., if negatives are randomly sampled independent of the image set), we find that overall there is only a small increase in performance compared to zero-shot inference. This demonstrates that in the regime where images in the same set do not appear in the same batch during training, models cannot extrapolate how to leverage the visual context at inference time. Adding context. For the fine-tuning regime, we observe instead a different trend once the visual context of the other images in a set is provided during training (+C ONTEXTBATCH ): CLIP and UNITER receive a significant boost in performance (i.e. +14.4% for CLIP), which is 3432 all video static ZERO-SHOTCLIP 22.4 15.6 47.8 FINE-TUNINGCLIP 24.3 17.1 51.3 +C ONTEXTBATCH 28.4 20.0 60.0 +C ONTEXTMODULE 27.7 19.6 58.4 +T EMPORALEMBEDDINGS 29.9 22.0 59.8 ZERO-SHOTUNITER 19.8 13.6 42.9 FINE-TUNINGUNITER 21.9 14.4 50.1 +C ONTEXTBATCH 24.8 17.4 52.8 +C ONTEXTMODULE 24.4 16.7 53.0 +T EMPORALEMBEDDINGS 25.7 19.1 50.5 ZERO-SHOT ViLBERT 19.3 13.5 40.8 FINE-TUNING ViLBERT 20.9 13.1 49.9 +C ONTEXTBATCH 20.9 15.0 42.7 +C ONTEXTMODULE 22.3 16.1 45.6 +T EMPORALEMBEDDINGS 24.5 18.0 49.3 Table 5: Performance (test accuracy) on IMAGECODE across two training regimes (zero-shot and fine-tuning), three models (CLIP, UNITER, ViLBERT) and 4 model variants. particularly accentuated for static pictures. On the other hand, ViLBERT's performance remains the same. Stacking a special module for con-textualizing multimodal representations on top of the encoders (+C ONTEXTMODULE ), instead, yields gains for ViLBERT compared to +C ONTEXTBATCH , whereas CLIP and UNITER are unaffected (slight drop). This shows that all models can exploit visual context, but different strategies (contrastive training or dedicated modules) may be necessary. Finally, all three models achieve the highest performance when fine-tuned with both visual and temporal context. Adding temporal positional embeddings on top of the contextual module (+T EMPORALEMBEDDINGS ) yields an accuracy of 29.9 for CLIP, 25.7 for UNITER and 24.5 for ViLBERT. Crucially, even the best-performing models lag significantly behind the (micro-averaged) human accuracy of 90.8 (cf. Table 3). Hence, despite some limited ability to integrate context, models are currently incapable of the fine-grained reasoning and pragmatic inferences needed to solve IMAGECODE . Pre-trained model. Across all model variants and training regimes, CLIP consistently achieves higher accuracy than ViLBERT or UNITER. This implies that a larger amount of parameters, pretraining examples or the contrastive objective are more beneficial than ViLBERT's or UNITER's more expressive model architecture. Thus, these results violate the expectations that attention between vision and language would be more suitable to jointly encode highly nuanced visual details and descriptions (Miech et al., 2021). Additionally UNITER slightly outperforms ViLBERT as its single-stream architecture might enable richer cross-modal interactions. Video frames vs. static pictures. The highest accuracy on the subset of the data with video frames (20.9) is far lower than that for static pictures (59.4). This confirms that videos represent the main challenge in IMAGECODE , both because of the higher similarity of images in a set and of the particular factors of variation that help differentiate among them (cf. Section 4.3 and examples in Appendix G). Additionally, model performance on video frames seems to increase more consistently as more context (both visual and temporal) is provided, whereas there is no clear trend in the case of static pictures. Error Analysis. On a broad level, we have seen that video frames are much more challenging for models. Next, to identify more fine-grained causes for the overall low performance of the vision-and-language models on IMAGECODE , we compute the Pearson's correlation between accuracy and a series of possible explanatory variables. In particular, we find a weak negative correlation with the number of tokens in the description ( r =  0 . 11 ) and a weak positive correlation with the average pair-wise Euclidean distance between CLIP encodings of the images in a set ( r = 0 . 22 ), which represents visual similarity. By focusing on the 1000 annotated examples in Table 2 we observe a stark drop from overall performance on the subset of examples containing nuances, visibility/occlusion, and negation (Fig-ure 4). This confirms insights from Kassner and Schtze (2020) and Hosseini et al. (2021) on the difficulty of modeling negation in text-only models. We created a new challenge, Image Retrieval from Contextual Descriptions (IMAGECODE ), which is designed to evaluate the ability of vision-and-language models to integrate visual, pragmatic, and temporal context into their predictions. In particular, given a complex and nuanced contextual description, a model is required to retrieve the corresponding image from a set of highly similar candidates. We benchmarked state-of-the-art bi-encoder and cross-encoder models, such as CLIP and ViLBERT. Moreover, we proposed new variants of these models that are more suitable to solve this task, by augmenting them with a module to attend on the other images in a set and temporal embeddings. We found that IMAGECODE is highly challenging for all variants: even the best model (28.9) lags behind human performance (90.8) dramatically. Images sourced from video frames display the largest gap in performance. The most challenging phenomena in IMAGECODE include pragmatics, negation, fine-grained distinctions between images, and occlusion among others. IMAGECODE wouldn't have been possible without the herculean effort of the Amazon Mechanical Turkers and their feedback on the interface. We also thank Emanuelle Bugliarello for his help with VOLTA , an excellent codebase for several vision and language models. We thank the members of SR's research group for their feedback on the ideas presented here. IMAGECODE is funded by the Mila-Samsung grant program. We thank Microsoft for providing us Azure credits. SR acknowledges the support of the NSERC Discovery Grant program and the Facebook CIFAR AI Chair program. We distribute the descriptions in IMAGECODE under MIT and adopt the licenses of the video and image sources on which our image sets build on top. We report details about crowdsourcing such as payment and selection criteria in Section 3.2 and Appendix B. For the tested model variants, we only train a single run for each hyperparameter setting due to long run times.
abstain objective abstain abstain abstain abstain abstain result objective method abstain abstain other abstain abstain objective abstain abstain objective abstain abstain method result abstain objective result other abstain abstain abstain abstain abstain objective result method other other method abstain other other other abstain method method other other other other abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain method method abstain abstain method abstain abstain abstain abstain method abstain abstain method method method method abstain abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method method method method method method abstain other result abstain result abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain objective abstain abstain objective result abstain abstain other other other other other other method method