As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influenc-ing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, presenting an approach for good-enough entity linking without entity recognition first. Last, we explore some geographical and economic factors that may explain the observed dataset distributions. 1 1 Introduction The lack of linguistic, typological, and geographical diversity in NLP research, authorship, and publications is by now widely acknowledged and documented (Caines, 2019; Ponti et al., 2019; Bender, 2011; Adelani et al., 2021). Nevertheless, the advent of massively multilingual models presents opportunity and hope for the millions of speakers of under-represented languages that are currently under-served by language technologies. Broadening up the NLP community's research efforts and scaling from a handful up to the almost 7000 languages of the world is no easy feat. In order for this effort to be efficient and successful, the community needs some necessary foundations to build upon. In seminal work, Joshi et al. (2020) provide a clear overview of where we currently stand with respect to data availability for the world's languages and relate them to the languages' representation in NLP conferences. Choudhury and 1 Code and data are publicly available: https://github. com/ffaisal93/dataset_geography . Additional visualizations are available in the project page: https://nlp.cs.gmu. edu/project/datasetmaps/ . Deshpande (2021) study how linguistically fair are multilingual language models, and provide a nuanced framework for evaluating multilingual models based on the principles of fairness in economics and social choice theory. Last, Blasi et al. (2022) provide a framework for relating NLP systems' performance on benchmark datasets to their downstream utility for users at a global scale, which can provide insights into development priorities; they also discuss academic incentives and socioeconomic factors that correlate with the current status of systematic cross-lingual inequalities they observe in language technologies performance. These works provide insights into current data availability and estimated utility that are paramount for making progress, as well as an evaluation framework for future work. However, there is one missing building block necessary for real progress: a way to estimate how representative of the underlying language speakers is the content of our datasets. Any evaluation framework and any utility estimates 3381 we build can only be trustworthy as long as the evaluation data are representative. Gebru et al. (2021) and Bender and Friedman (2018) recognize the importance of this information, including them in their proposed guidelines for datasheets and data statements respectively; but most datasets unfortunately lack such meta-information. To the best of our knowledge, MaRVL (Liu et al., 2021) is the only dataset that is culturally-aware by design in terms of its content. 2 We propose a method to estimate a dataset's cultural representativeness by mapping it onto the physical space that language speakers occupy, producing visualizations such as Figure 1. Our contributions are summarized below:  We present a method to map NLP datasets unto geographical areas (in our case, countries) and use it to evaluate how well the data represent the underlying users of the language. We perform an analysis of the socio-economic correlates of the dataset maps we create. We find that dataset representativeness largely correlates with economic measures (GDP), with geographical proximity and population being secondary. We test a simple strategy for performing entity linking by-passing the need for named entity recognition. We evaluate its efficacy on 19 languages, showing that we can get within up to 85% of a NER-informed harder-to-obtain model. We also show that encouragingly, using either model largely leads to similar dataset maps. Assumptions This work makes two assumptions: that (a) data locality matters, i.e., speakers of a language are more likely to talk about or refer to local news, events, entities, etc as opposed to ones from a different side of the world, and (b) that we can capture this locality by only focusing on entities. Kumar et al. (2019) discuss these topical correlations that are present in datasets, 3 noting that they exist and that L1 language identification models tend to pick up on them, i.e. if a text mentions Finland, a L1 langid model is probably going to predict that the speaker is Finnish, because p ( Finland  L1 = Finnish ) is generally high. In that work Kumar et al. (2019) make explicit effort 2 Datasets designed to capture dialectal variations, e.g., SD-QA (Faisal et al., 2021), are culturally-aware in terms of annotator selection, but there is no guarantee that their content is also culturally-relevant for the language speakers. 3 See 2 of their paper. to avoid learning such correlations because they are interested in building models for p ( L1  text ) (i.e. p ( L1 = Finnish  Finland ) ) that are not confounded by the reverse conditional. The mere fact they need to do this, though, confirms that real-world text has such topical confounds. As for our second assumption that we can capture these topical correlations by only looking at entities, one need only to take a look at Table 2 of Kumar et al. (2019), which lists the top topical confounding words based on log-odds scores for each L1 language in their dataset: all lists include either entities related to a country where that language is spoken (e.g. Merkel', the name of a former chancellor, for German) or topical adjectives (e.g. romanian' for Romanian). Approach For a given dataset, our method follows a simple recipe: 1. Identify named entities present in the dataset. 2. Perform entity linking to wikidata IDs. 3. Use Wikidata to link entities to countries. We discuss each step below. Entity Recognition Step Standard entity linking is treated as the sequence of two main tasks: entity recognition and entity disambiguation. One approach is to first process the text to extract entities and then disambiguate these entities to the correct entries of a given knowledge base (eg. Wikipedia). This approach relies on NER model quality. However, to perform analysis on several datasets spanning several low-resource languages, one needs good-quality NER models in all these languages. The interested reader will find a discussion on the cross-lingual consistency of NER models in Appendix F. 4 As we show in Section 4, we can bypass this NER step if we tolerate a small penalty in accuracy. Entity Linking Step In this step we map named entities to their respective Wikidata IDs. We further discuss this step in Section 4. From Entities to Countries We produce maps to visualize the geographical coverage of the datasets we study, discussing their properties and our findings in Section 3. 4 Discussion summary: state-of-the-art NER models are not cross-lingually consistent, i.e. they do not produce the same entity labels when presented with translations of the same sentence. We recommend using parallel data as part of the evaluation sets in multiple languages to measure this important aspect of models' performance. To link entities to countries, 5 we rely on Wikidata entries, depending on the type of entity:  for persons, we log their places of birth (P19) and death (P20), and country of citizenship (P27);  for locations, we search for their associated country (P17); and  for organizations, we use the links of the lo-cated_at' (P276) and headquartered_at' (P159) relations. Since places of birth/death and headquarters are not necessarily at the country level, we perform a second step of associating these locations with countries. In cases where the result does not correspond to a modern-day country (as can often be the case with historical figures), we do not make any attempts to link it to any modern day countries, excluding them from the analysis. For example, the entry for Nicolaus Copernicus (Q619) lists him as born in Torun (Q47554) which is then mapped to Poland; as having died in From-bork (Q497115) that also maps to Poland; and as a citizen of the Kingdom of Poland (Q1649871) which is not mapped to any modern-day country; so he is only linked to Poland. Albert Einstein is similarly mapped to both Germany and the United States, due to his places of birth (Ulm) and death (Princeton). Before delving into our case studies, we first list a set of statistics of interest that one could extract from our produced dataset-country maps, in order to gauge a dataset's representativeness. Representativeness Measures We will avoid providing a single metric, largely because the ideal metric to use will be very dataset-specific and related to the goals of the creators of the dataset and the socioeconomic correlates they are interested in (see discussion in Section 3.3). As a first straightforward representativeness measure, we will compute the percentage of entities associated with countries where the language is largely spoken . For example, according to Ethnologue (Eberhard et al., 2021), most Swahili speakers 6 reside in Tanzania, Kenya, Uganda, DR. Congo, and Rwanda. For a Swahili dataset, then, we compute the percentage of all entities associated with this set of countries ( in-country ). Notions of equity or fairness across countries could be measured by various fairness metrics, given the distribution of entities over countries in a dataset: from simply computing the standard deviation of the observations, 7 to treating countries as a population and computing fairness indices like the popular Gini index (Gini, 1912; Gastwirth, 1972) or the indices proposed by Speicher et al. (2018). We will opt for a simpler, much more interpretable measure, the number of countries not represented in the dataset i.e. countries with associated entity count below a given threshold (we use zero for simplicity but higher values would also be reasonable for large datasets). Last, especially for languages with significant amounts of speakers in more than one country, it is important to go deeper and measure the representativeness of this in-country portion. For a simple example, an English dataset with entities only from the UK is probably not representative of Nigerian or Jamaican English speakers. Hence, we will create two distributions over the countries where the language is largely spoken: the distribution of speaker populations (as available from Ethnologue and other public data), and the distribution of entities observed in the dataset. Discrepancies between these two distributions will reveal potential issues. While one could easily compute some measure of distance between the two distributions (e.g. the Bhattacharyya coefficient (Bhattacharyya, 1943)), in this work we will rely on the interpretable advantages of the visualizations. Measures of fairness could be computed for this portion of the dataset, similarly as discussed above. In the example dataset of the Swahili portion of MasakhaNER in Figure 1, the utility of our method is apparent. Through the visualization, a researcher can quickly confirm that the dataset seems to not reflect the users of the language to a large extent: only about 17% of the entities indeed correspond to Tanzania, Kenya, Uganda, DR. Congo, or Rwanda (where Swahili and its varieties are treated as a lingua franca, at least in portions of these coun-tries). Wealthy or populous countries like USA, France, and China, are well-represented, 8 as one would expect, while 156 countries and territories have no representation. At the same time, the visualization allows a researcher to identify gaps: 7 Or approximations thereof such as the max-min of the observations, as used by (Debnath et al., 2021). beyond the neighboring African countries and perhaps the Middle East, north-west African countries as well as central America or central/south-east Asia are clearly under-represented in this dataset. Between the main Swahili-speaking countries, Tanzania, Kenya, and Uganda are well-represented (DR Congo and Rwanda less so, but they have less Swahili speakers), with the former two perhaps slightly over-represented and the latter (as well as Rwanda) being under-represented relative to the speakers population, c.f. red (dataset entities) and green (proportional to population) bars in Figure 1. 3.1 Datasets and Settings We apply the process described above on several datasets, chosen mostly for their language and typological diversity. Our process is not datasetor language-dependent, 9 and could easily be applied on any NL dataset. We briefly describe the datasets we include in our study below, with detailed statistics in Appendix C. NER Datasets We study the WikiANN dataset (Pan et al., 2017) that is commonly used in the evaluation of multilingual models. We additionally study the MasakhaNER dataset (Ade-lani et al., 2021), which was created through participatory design (  et al., 2020) in order to focus on African languages. Since these datasets are already annotated with named entities, we only need to perform entity linking. rather than contexts), namely SQuAD (Rajpurkar et al., 2016), MLQA (Lewis et al., 2020), TyDiQA (Clark et al., 2020), and Natural Questions (Kwiatkowski et al., 2019, NQ;), which have unique characteristics that lend themselves to interesting comparisons. SQuAD is a large English-only dataset (although it has been translated through efforts like XQuAD (Artetxe et al., 2020)). MLQA is a n -way parallel multilingual dataset covering 7 languages, created by translating an English dataset. TyDi-QA is another multilingual dataset covering 11 languages, but each language portion is derived separately, without translation involved. Last, NQ is an English QA dataset created based on real-world queries on the Google search engine for which annotators found relevant Wikipedia context, unlike the other datasets that were created by annotators forming questions given a context. Additional Datasets While not further discussed in this paper, additional visualizations for more datasets (e.g. for the X-FACTR benchmark (Jiang et al., 2020), and several machine translation benchmarks) are available in the project's webpage: https://nlp.cs.gmu.edu/ project/datasetmaps/ . Beyond Figure 1, we also show example maps in Figure 2 for NQ, MLQA, SQuAD, and the English portion of TyDi-QA. We provide additional maps for all other datasets in Appendix G. Comparing datasets The comparison of MasakhaNER to the WikiANN dataset (see Appendix G) reveals that the former is rather more localized (e.g. more than 80% of the identified entities in the Dholuo dataset are related to Kenya) while the latter includes a smaller portion from the countries where most native speakers reside (between 10%-20%) and almost always also includes several entries that are very Europeanor western-centric. The effect of the participatory design (  et al., 2020) approach on creating the MasakhaNER dataset, where data are curated from local sources, is clear in all language portions of the dataset, with data being highly representative of the speakers. In Figures 89 (App. G) the majority of entities in the Wolof portion are from Senegal and neighboring countries (as well as France, the former colonial power of the area), and the Yoruba and Igbo ones are centered on Nigeria. Figure 2 allows for a direct comparison of different QA datasets (also see maps for other TyDiQA languages in Appendix G). The first notable point has to do with NQ, which was built based on real-world English-language queries to the Google search engine. Since such queries happen all over the world, this is reflected in the dataset, which includes entities from almost all countries in the world. Two types of countries are particularly represented: ones where English is an official language (USA, UK, Australia, but also, to a lesser extent, India, Nigeria, South Africa, and the Philippines); and wealthy ones (European, Japan, China, etc). In our view, NQ is an exemplar of a representative dataset, because it not only includes representation of most countries where the language is spoken (with the sum of these entities being in their large majority in-country: 80%) but due to its size it also includes entities from almost all countries. SQuAD also has a large percentage in-country (63%) but it is less representative of different Englishes than NQ. India, for instance, is relatively under-represented in all datasets; in SQuAD it ranks 7 th , but it ranks 3 rd in NQ (see red bars in bottom left of figures). On the other hand, the geographical representativeness of both MLQA and TyDi-QA (their English portion) is lacking. Since these datasets rely on Wikipedia articles for their creation, and Wikipedia has a significant western-country bias (Greenstein and Zhu, 2012; Hube and Fetahu, 2018), most entities come from Europe, the US, and the Middle East. All these datasets underrepresent English speakers from English-speaking countries of the Global South like Kenya, South Africa, or Nigeria, since there are practically almost no entities from these countries. MLQA further under-represents the speakers of all other languages it includes beyond English, since all data are translations of the English one. Contrast this to TyDi-QA and its visualized Swahili portion which, even though still quite western-centric, does have a higher representation from countries where Swahili is spoken than the TyDi-QA English portion. This discussion brings forth the importance of being cautious with claims regarding systems' utility, when evaluated on these datasets. One could argue that a QA system that is evaluated on NQ does indeed give a good estimation of real-world utility; a system evaluated on TyDi-QA gives a distorted notion of utility (biased towards western-based speakers and against speakers from the Global 3385 TyDi-QA (11) MLQA (1) SQUAD (1) NaturalQ. South); a system evaluated on MLQA will give an estimation as good as one evaluated on TyDi-QA, but only on the English portion. We clarify that this does not diminish the utility of the datasets themselves as tools for comparing models and making progress in NLP: MLQA is extremely useful for comparing models across languages on the exact same data , thus facilitating easy comparisons of the cross-lingual abilities of QA systems, without the need for approximations or additional statistical tests. But we argue that MLQA should not be used to asses the potential utility of QA systems for German or Telugu speakers. Similar observations can be made about comparing two similar projects that aim at testing the memorization abilities of large language models, namely X-FACTR and multi-LAMA (mLAMA; Kassner et al., 2021)  see corresponding Figures in Appendix G. Both of these build on top of Wikidata and the mTREx dataset. However, mLAMA translates English prompts and uses entity-relation triples mined from the English portion of Wikidata, unlike X-FACTR which uses different data for each language, mined from their respective portion of Wikidata. Both are still western-biased, since they rely on Wikipedia, but one (X-FACTR) is better at giving an indication of potential downstream utility to users. In this section we attempt to explain our findings from the previous section, tying them to socioeconomic factors. Empirical Comparison of Factors We identify socioeconomic factors  that could be used to explain the observed geographic distribution of the entities in the datasets we study. These are:  a country's population  pop  a country's gross domestic product (GDP)  gdp  a country's GDP per capita  gdppc  a country's landmass  land  a country's geographical distance from coun-try/ies where the language is spoken  geo The first four factors are global and fixed. The fifth one is relative to the language of the dataset we are currently studying. For example, when we focus on the Yoruba portion of the mTREx dataset, we use Nigeria (where Yoruba is spoken) as the focal point and compute distances to all other countries. The assumption here is that a Yoruba speaker is more likely to use or be interested in entities first from their home country (Nigeria), then from its neighboring countries (Cameroon, Chad, Niger, Benin) and less likely of distant countries (e.g. Argentina, Canada, or New Zealand). Hence, we assume the probability to be inversely correlated with the country's distance. For macro-languages or ones used extensively in more than one country, we use a population-weighted combination of the factors of all relevant countries. To measure the effect of such factors it is common to perform a correlational analysis, where one measures Spearman's rank correlation coefficient  between the dataset's observed geographical distribution and the factors  . It is important, though, that the factors are potentially covariate, particularly population and GDP. Hence, we instead compute the variance explained by a linear regression model with factors  as input, i.e., a  pop + b  gdp + c  gdppc + d  geo + e with a  e learned parameters, trained to predict the log of observed entity count of a country. We report explained variance and mean absolute error from five-fold cross-validation experiments to avoid overfitting. results with different combination of factors for the QA datasets are listed in Table 1. 10 The best sin-10 sin-10 See Appendix H for NER datasets, and Appendix I for breakdown by language for all datasets. gle predictor is, perhaps unsurprisingly, the GDP of the countries where the language is spoken: all datasets essentially over-represent wealthy countries (e.g. USA, China, or European ones). Note that GDP per capita is not as good a predictor, neither is landmass. A combination of geographical distance with GDP explains most of the variance we observe for all datasets, an observation that confirms the intuitions we discussed before based solely on the visualizations. Importantly, the fact that including population statistics into the model deteriorates its performance is further proof that our datasets are not representative of or proportional to the underlying populations. The only dataset that is indeed better explained by including population (and GDP per capita) is NQ, which we already argued presents an exemplar of representativeness due to its construction protocol. Limitations It is important to note that our assumptions are also limiting factors in our analyses. Mapping languages to countries is inherently lossy. It ignores, for instance, the millions of immigrants scattered throughout the world whose L1 language could be different than the dominant language(s) in the region where they reside. Another issue is that for many languages the necessary granularity level is certainly more fine than country; if a dataset does not include any entities related to the Basque country but does include a lot of entities from Spain and France, our analysis will incorrectly deem it representative, even though the dataset could have been a lot more culturally-relevant for Basque speakers by actually including Basque-related entities. Another limitation lies in the current state of the methods and data resources on which our approach relies. Beyond discrepancies in NER/EL across languages (addressing which is beyond the scope of this work), we suspect that Wikidata suffers from the same western-centric biases that Wikipedia is known for (Greenstein and Zhu, 2012). As a result, we might be underestimating the cultural representativeness of datasets in low-resource languages. An additional hurdle, and why we avoid providing a single concrete representativeness score or something similar, is that the ideal combination of socioeconomic factors can be subjective. It could be argued, for instance, either that geographic proximity by itself should be enough, or that it should not matter at all. Even further, other factors that we did not consider (e.g. literacy rate or web access) might influence dataset construction decisions. In any case, we share the coefficients of the NQ model, since it is the most representative dataset we studied, at least for English: a = 0 . 1 . 46 (for  pop ), b = 0 . 87 (  gdp ), c = 25 . 4 (  gdppc ), d = 0 . 41 (  geo ). We believe that ideally GDP should not matter ( b  0) and that a combination of speaker population and geographic proximity is ideal. 11 3.4 Geographical Breakdown of Models' Performance Beyond the analysis of the datasets themselves, we can also break down the performance of models by geographical regions, by associating test (or dev) set samples containing entities with the geographical location of said entities. Since most test sets are rather small (a few hundred to a couple thousand instances) we have to coarsen our analysis: we map each country to a broader region (Africa, Americas, Asia, Europe, Oceania), keeping historical entities in a separate category (History). 12 We perform such a case study on TyDi-QA, comparing the performance on the TyDi-QA development sets of two models: one trained mono-lingually on the training set of each language of TyDi-QA (gold task), and another model trained by Debnath et al. (2021) on English SQuAD and automatically generated translations in the target languages. Example results on Telugu shown in Figure 3 reveal some notable trends. 13 First, training set representation (green bars in the Figures) is not a necessary condition for good test set performance (red bars). Some test set instances (e.g. with historical and African entities) receive similar test F1 score from both models. Perhaps the most interesting though, is the comparison of the Asian and European portions of the test set: the Telugu monolingual model achieves similar performance in these two subsets; but the SQuAD-trained model is almost 20 percentage points worse on the Asian subset, showing the potential unfairness of translation-based models (Debnath et al., 2021). For most TyDi-QA languages (Indonesian being an exception, see Table 2) the macro-standard deviation (computed over the averages of the 6 region subsets) is larger for the SQuAD-trained model (which is, hence, less fair than models trained on 11 However regrettable a fact, it is undeniable that western culture and politics have world-wide effects. So their (over)representation as a result of their high influence (and GDP) might actually reflect the true interests of people everywhere! 12 Future work could explore a different clustering. 13 See Table 4 in Appendix D for all languages. We use mGENRE (Cao et al., 2021) for the task of multilingual entity linking, a sequence to sequence system that predicts entities in an auto-regressive manner. It works particularly well in a zero-shot setting as it considers 100+ target languages as latent variables to marginalize over. Typically, the input to mGENRE can be informed by a NER model that provides the named entity span over the source. For instance, in the Italian sentence "[START] Einstein [END] era un fisico tedesco." ( Einstein was a German physicist. ) the word Einstein is enclosed within the entity span. mGENRE is trained to use this information to return the most relevant Wikidata entries. Due to the plasticity of neural models and mGE-BRE's auto-regressive token generation fashion, we find that by simply enclosing the whole sentence in a span also yields meaningful results. In particular, for the previously discussed Italian sentence now the input to mGENRE is "[START] Einstein era un fisico tedesco. [END]" . The advantage of this approach is two-fold. First, one does not need a NER component. Second, exactly because of bypassing the NER component, the EL model is now less constrained in its output; in cases where the NER component made errors, there's a higher chance that the EL model will re-b e n h i n e s t q u e j a v r u s t u r c m n j p n language 0.0 0.2 0.4 0.6 0.8 1.0 a g r ee m e n t @ k WiKiANN k i n p c m y o r w o l i b o s w a h a u l u g l u o a m h language 0.0 0.2 0.4 0.6 0.8 1.0 MasakhaNER Comparing top-k k=1 k=2 k=3 Figure 4: For some languages a NER-Relaxed model is within 60% of a NER-Informed model. Experiments and Results We conduct experiments to quantify how different a model uninformed by a NER model (NER-Relaxed) will perform compared to one following the typical pipeline (NER-Informed). Given the outputs of the two models over the same set of sentences, we will compare their average agreement@ k , as in the size of the intersection of the outputs of the two models divided by the number of outputs of the NER-Informed model, when focusing only on their topk outputs. 14 We aggregate these statistics at the sentence level over the whole corpus. We focus on two datasets, namely WikiANN and MasakhaNER, summarizing the results in Figure 4. 15 Comparing the general performance between these two datasets, it is clear that general agreement is decent. In 7 Out of 9 typologically diverse languages from WikiANN, more than 60% top-1 entities are linked by both models. The African languages from MasakhaNER are low-resource ones yielding less than 40% EL agreement to English in all cases. Given that most of these languages have not been included in the pre-training of BART (the model mGENRE is based on), we expect that using AfriBERTa (Ogueji et al.) or similar models 14 Both models typically output between 13 entity links ranked according to their likelihood. Effect on downstream maps We compare the dataset maps we obtain using NER-Relaxed and NER-Informed (using gold annotations) models in our pipeline for the MasakhaNER dataset. Overall, the maps are very similar. An example visualization of the two maps obtained for Swahili is in Figure 5 in Appendix E.1. The NER-Informed model produces slightly fewer entities overall (likely exhibiting higher precision for lower link recall) but there are minimal differences on the representativeness measures e.g., the in-country percentage changes from 15.3% (NER-Informed) to 16.9% (NER-Relaxed). We can compare the distributions of the topk countries obtained with the two models using Ranked Biased Overlap (RBO; higher is better; Webber et al., 2010). 16 The results for varying values for k (topk countries) are presented in Table 6 in Appendix E.1. We overall obtain very high RBO values ( > . 8 for k = 10) for all language portions and all values of k . For example for 8 of the 10 MasakhNER languages the two models almost completely agree on the top-10 countries with only slight variations in their ranking. Dholuo and Amharic are the ones exhibiting the worse overlap (but still > . 5 RBO). We present a recipe for visualizing how representative NLP datasets are with respect to the underlying language speakers. We plan to further improve our tool 17 by making NER/EL models more robustly handle low-resource languages. We will also expand our dataset and task coverage, to get a broader overview of the current utility of NLP systems. This work is generously supported by NSF Awards 2040926 and 2125466.
abstain abstain objective objective objective abstain abstain abstain abstain method other abstain other abstain abstain abstain abstain abstain method abstain abstain objective method result method result result abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method method method abstain abstain abstain abstain abstain other result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain method method abstain method method abstain abstain method abstain abstain abstain abstain abstain result abstain method method abstain abstain abstain abstain abstain result method abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain other