Character-level information is included in many NLP models, but evaluating the information encoded in character representations is an open issue. We leverage perceptual representations in the form of shape, sound, and color embeddings and perform a representational similarity analysis to evaluate their correlation with textual representations in five languages. This cross-lingual analysis shows that textual character representations correlate strongly with sound representations for languages using an alphabetic script, while shape correlates with featural scripts. We further develop a set of probing classifiers to intrinsically evaluate what phonological information is encoded in character embeddings. Our results suggest that information on features such as voicing are embedded in both LSTM and transformer-based representations. On the one hand, writing is an essential form of human communication. Writing systems and orthographies differ across languages and impact our reading behavior. Psycholinguists have extensively studied the effect of orthographic depth, i.e., the transparency of grapheme-to-phoneme mappings, on reading acquisition as well as skilled reading (Seymour et al., 2003). On the other hand, the wide range of cross-linguistic diversity is still a major challenge for natural language processing (NLP) and for the study of language more generally (Mielke et al., 2019; Gutierrez-Vasques and Mijangos, 2020), especially on sub-word levels (Gutierrez-Vasques et al., 2021). This increases the importance of cross-lingual analyses of character-level language models (LMs), because anglocentrism in linguistic research is not only prevalent in NLP, but also in (reading and) orthography research (Share, 2008). with Latin scripts, since they contain meaningful information on various linguistic levels and enhance the robustness of models. Oh et al. (2021) suggest that character LMs provide a more human-like account of sentence processing, which assumes a larger role of morphology, phonotactics, and orthographic complexity than was previously thought. Moreover, including character and sub-character information in LMs for Asian scripts is a standard practice. Despite of this recent attention, work focusing on getting a deeper understanding of character representation is scarce (Kann and Monsalve-Mercado, 2021), in particular regarding the comparison between languages and different types of scripts. The goal of this work is to improve our understanding of learned character representations, for better interpretability of the models. Like other neural network based models, character-level LMs can be seen as black-box methods and reveal limited insights about the causes for their predictions (Gilpin et al., 2018). We investigate the information encoded in character embeddings by comparing them to perceptual representations. Such representations we design by mimicking features of human language processing, from reading, writing and speaking, by the creation of embeddings based on the shape of characters, the sound (phonological features derived from grapheme-to-phoneme mappings) and color (elicited in the form of grapheme-color mappings from synesthetes). Contributions We train models to learn three types of character embeddings: a positive pointwise mutual information (PPMI) vectorization, a recurrent model, and a transformer model. As an intrinsic evaluation method, we conduct a representational similarity analysis (RSA) between the distances of textual character representations and the perceptual representations in the form of shape, sound, and color embeddings. Furthermore, to pro-6819 vide more interpretable evaluation methods for character embeddings, we propose a novel probing task of predicting phonological features. Crucially, we address the cross-linguistic challenges that arise with character-level modeling by taking into account languages of varying scripts and orthographic depths. We argue that character-level black-box models can only be understood through cross-linguistic approaches and not on individual languages. We perform analyses of five languages: Dutch, English, Japanese, Korean, and Spanish. We discuss the compelling patterns of significant correlations and show the effectiveness of the probing classifiers even in a zero-shot scenario. The implementation and character representations are available online 1 . Character-level information in LMs. Including character-level information in LMs of languages with Latin scripts has become a common practice in NLP in recent years. This has been the case for different tasks, such as language modeling (Kim et al., 2016; Al-Rfou et al., 2019), part-of-speech tagging (Ling et al., 2015), morphological inflection (Faruqui et al., 2016; Kann and Schtze, 2016; Kann et al., 2020), named entity recognition (Lample et al., 2016), machine translation (Sen-nrich et al., 2016; Ngo et al., 2019). Character-level information can enhance the models by providing background knowledge in the form of the underlying structures of words in a language (Adouane et al., 2018). Ma et al. (2020) showed how combining characterand word-level information in pre-trained LMs improves not only the performance but also the robustness of the model. For certain languages, it is standard practice to include sub-token information in LMs, which happens naturally due to the compositional structure of their orthographies. This is the case for East Asian languages such as Korean and Japanese (e.g., Misawa et al. 2017; Chen et al. 2015). Korean LMs are often trained on Jamos (i.e., letters, as opposed to syllables), the smallest unit of the Korean script (Ahn et al., 2017; Park et al., 2018). This reduces the vocabulary size and injects syntactic and semantic information to the model that is difficult to access with conventional character-or token-level units (Stratos, 2017). Recently, Lee 1 https://github.com/syssel/ Interpreting-character-embeddings et al. (2020b) showed that a Korean BERT model using sub-character information requires less training data than previous models. Similarly, Japanese LMs also benefit from sub-character information (Nguyen et al., 2017). Evaluating character embeddings. Character-based language models are most often evaluated on downstream NLP tasks or on next character or word prediction (e.g., Takase et al. 2019; Tay et al. 2021; Clark et al. 2021). Additionally, they can be evaluated on word-level intrinsic evaluation tasks such as word analogy or similarity (e.g., Li et al. 2015). While work on intrisic evaluation of character embeddings is scarce (Kann and Monsalve-Mercado, 2021), the evaluation of neural models trained on phonemes have received more attention, focusing on what phonological knowledge is embedded within (Silfverberg et al., 2018; Kolachina and Magyar, 2019; Mayer and Nelson, 2020; Mayer, 2020; Silfverberg et al., 2021). Mayer (2020) and Mayer and Nelson (2020) use characters as an approximation of phonemes in the case of Samoa and Finnish, respectively, as graphemes are closely connected to phonemes in these orthographies. The methods we leverage in this paper, previously applied for evaluating different types of representations, are representational similarity analysis (RSA) and probing classifiers. The former was first proposed by Kriegeskorte et al. (2008) for comparing brain activity vectors in heterogeneous representational spaces, but has also been applied in NLP as an interpretability metric as it allows us to study the relation between language representations (Abnar et al., 2019; Abdou et al., 2019; Chrupaa and Alishahi, 2019). RSA enables a transparent comparison between the representational geometries of different models and modalities (Sgaard, 2021). Contrarily, probing classifiers learn to classify output representations in supervised settings (Et-tinger et al., 2016). The intuition behind probing is that if a classifier can be learned to accurately predict certain linguistic properties from the representations of a neural model, then this model has "learned" this property. Typically, lightly parametrized classifiers (like logistic regression) are applied, however, the exact trade-off between accuracy and complexity of a probe is an open question (Belinkov, 2021). In recent years, NLP studies have used probing classifiers to investigate 6820 whether LMs encode linguistic properties including morphological features (such as person and number , Torroba Hennigen et al. (2020)) and word sense (Coenen et al., 2019). However, we apply probing classifiers for the first time to character representations. Impact of different orthographies on linguistics and human language learning. Orthographic depth, i.e., the transparency of grapheme-phoneme correspondences in written language (Frost et al., 1987; Katz and Frost, 1992), is a well-studied fac-tor influencing reading acquisition and skilled reading behavior (Seymour et al., 2003; Landerl et al., 2013; Richlan, 2020). For instance, English is considered to be a deep orthography , as there are often multiple different pronunciations for the same spelling patterns (e.g., <gh> in tough and though ). This contrasts shallow orthographies with more reliable grapheme-phoneme correspondences, such as Spanish. The consistency and complexity with which print reflects speech is one of the prime factors of cross-linguistic differences in reading flu-ency (Ziegler et al., 2010; Schmalz et al., 2015). It is the starting point for any discussion that centers on reading development across languages (Pa-padopoulos et al., 2021). Since the orthography has such a high impact on human reading behavior, its effect should also be considered more carefully in the development of NLP models. models. While orthographic depth has been discussed at length in reading research and psychology, it has rarely been addressed in NLP. This partly due to the prevalent anglocentrism and missing resources (Bender, 2018). Some research has gone into studying the differences between languages when it comes to train computational LMs (Mielke et al., 2019), showing the impact of the vocabulary size and sentence length, but there is lack of NLP research analyzing or taking into account the varying orthographies across languages. Two notable exceptions are the recent methods proposed by Mar-jou (2021) and Sproat and Gutkin (2021), who use neural networks to estimate the transparency of orthographies and degree of logography, respectively. Moreover, Gorman et al. (2020) conducted a shared task on grapheme-to-phoneme prediction. Their results show an urgency for improving these systems and the pronunciation dictionaries used to train them across languages and scripts. We train three types of character embeddings based on textual input: count-based PPMI embeddings, and embeddings learned by LSTM and transformer language model. We use the Wiki40B multilingual dataset (Guo et al., 2020) to train the character models. For each of the five languages, English (en), Dutch (nl), Spanish (es), Korean (ko), and Japanese (ja), we extract training sets of 3 million characters. See Appendix A for details on preprocessing. The first three languages all use variants of the Latin script, while Hangul (Korean) and Hiragana (one of three scripts used in Japanese) are syllabic scripts, in which most graphemes denote entire syllables. We preprocess Korean Hangul characters, decomposing them into constituent Jamos , each corresponding roughly to a single phoneme. For Japanese, we convert Kanji symbols to Hiragana and train the language model on Hiragana and Katakana characters. The representational similarity analyses are then only performed on Hiragana . Figure 1 shows 2-dimensional plots of the learned textual character representations. Count-based PPMI embeddings. We generate vectorized character representations in a purely count-based manner with a positive pointwise mutual information (PPMI) weighting. While the importance of positional information is less obvious for modelling word semantics, it is crucial for modelling the distribution of sounds. Following the approach by Mayer (2020), we let our PPMI weighting diverge from traditional bag-of-words models by distinguishing contexts by their relative position to a target. Thus, embeddings will have indepen-dent values for the contexts AB_ , _AB , and A_B , counting the number of times a target follows, precedes, and mediates a string AB . Using bigram contexts, the resulting embeddings have a dimension of 3  c 2 , where c is the number of characters in a given language, and 3 indicating the number of possible relative positions. LSTM. We train a recurrent language model consisting of two unidirectional long-short term mem-ory (LSTM) layers. It receives sequences of 40 characters as input at each time step and is trained for next character prediction. The model is trained with an Adam optimizer (Kingma and Ba, 2015), 6821 o m k u t p b r h d z s g y l f x q e i a j c w n v en PPMI vowelconsonant a b c d e f g h i j k l m n o p q r s t u v w x y z en LSTM vowelconsonant a b c d e f g h i j k l m n o p q r s t u v w x y z en Transformer vowelconsonant ko PPMI vowelconsonant ko LSTM consonantvowel ko Transformer consonantvowel Figure 1: tSNE cluster plots of the character distances from the three types of character language models for English and Korean (see Appendix Figure 5 for the plots for Dutch, Spanish and Japanese). an initial learning rate of 0.01, and a batch size of 128. We extract the hidden representations of 128 dimensions as the character embeddings. See Appendix C.1 for training specifications and Appendix C.2 for perplexity metrics. We additionally experimented with bidirectional LSTMs (see Table 1) and 1-layer LSTMs without any substantial changes in the results (Appendix C). Transformer. Similarly, we also train a transformer character model on the same data (Vaswani et al., 2017). The input layer consists of character and positional embeddings, followed by a single transformer block with 2 heads and a hidden layer size of 128. We follow the same training procedure as for the LSTM and extract the representations of the hidden layer as the character embeddings. Again, see Appendix C for additional details, model modifications, and perplexity metrics. Sound. The first perceptual representation that we consider is sound . To retrieve this representation, we map characters to a phonological distinctive feature space. This method has previously been applied to phonemes as a means of generalisation compared to sparse representations (Rumelhart and McClelland, 1986; Mirea and Bicknell, 2019), and to evaluate the knowledge embedded representations learned from neural networks (Silfverberg et al., 2018; Kolachina and Magyar, 2019). As sound and speech are only indirectly reflected in writing, we approximate sound representations of characters using grapheme-to-phoneme alignment: For all languages, we extract data from the WikiPron pronunciation dictionary (Lee et al., 2020a) and use the m2m-aligner (Ji-ampojamarn et al., 2007) to align graphemes with phonemes in an unsupervised manner. Having alignments from the WikiPron data, we chose the most frequent phoneme mapping to represent the sound of each character (resulting mappings are listed in the Appendix D) We also considered extracting the most frequent phoneme mapping only from word-initial positions. The intuition behind this approach was to retrieve representations as close to phonemic as possible, as sounds in the initial position are expected to be less prone to phenomena such as reduction and assimilation reflected in the WikiPron data (e.g., reduction of English "o" to @ ). However, the word-initial position is also subject to phonotactic restrictions: For exam-6822 ple, in Korean only including consonants occurring word-initially heavily reduces the inventory considered. Having phonemes mapped to characters, we are able to associate it with a set of phonological distinctive features, which we use to form our final sound representation: Using the ipapy 2 toolkit, we retrieve International Phonetic Alphabet (IPA) descriptions of the phoneme mappings from which we create a sparse vector that describes what phonological features (e.g., consonant manner of articulation,  plosive, or vowel height,  front) are active. For every language, this provides us with a sound embedding table, S | V || F | , where V is the set of characters and F is the set of distinctive features: S i,j = (cid:40) 1 if F j  phonmap( V i ) . Color. Inspired by Kann and Monsalve-Mercado (2021), we compute color character representations from synesthesia data. Grapheme-color synesthesia is a neurological phenomenon in which viewing a grapheme elicits an automatic, involuntary, and consistent sensation of color (Eagleman et al., 2007). Color-to-letter associations in synesthesia allow to examine the relationships between visual, acoustic, and semantic aspects of language. Recent research in this area has found cross-linguistic similarities in synesthesia, suggesting that some influences on grapheme-color associations in synesthesia might be universal and highlighting the importance of multilingual analyses (Root et al., 2018). Figure 2 shows example grapheme-color associations from individual subjects for each of our studied languages. It emphasizes the preference for red color tones for the first letter of the alphabet irrespective of the language (Root et al., 2018). We use the cross-linguistic synesthesia data collected by Root et al. 2018 (see Appendix B for the dataset statistics). In order to extract color representations we compute the Euclidean distances between the 3-dimensional CIELuv color coding scheme for all character combinations. We average the distances across all participants of the same language. The resulting vector representations re-flect the finding of Root et al. (2018) that the first grapheme in any language is unusually distinct (see Figure 4 in Appendix) . Shape. Lastly, we also create simple character representations based on their shape. Previous works (Brang et al., 2011; Watson et al., 2012) have relied on Gibson (1969) or Courrieu et al. (2004) to build shape-related embeddings from human similarity judgements. However, we create shape embeddings directly from their visual expressions. We create an image for each printed character as shown in Figure 6 in the appendix. For each script, all images have the same width and height (the largest width among all characters incremented with 10 pixels, and the same for the height, which results approximately in 35  45 pixels) and all characters are drawn at position {5,5} . We use the font Arial Unicode MS with size 28 . From these images, we create shape representations by reading the images as gray scale images row-wise from top to bottom and flattening the matrix into vectors. In order to analyze the relation between the learned character representations and the three perceptual representations  sound, shape, and color  we first compute the pairwise distances between characters of a single model/representation type to analyze how similar the model's representations for each character are to each other 3 . For each pair of experimental conditions, the spatial correlation is calculated between the distances of all characters of a language. Figure 3 shows the Pearson correlations between the character distances of all embedding types. The figure also includes a baseline, where the correlation between random distances and the distances of the respective character representations is computed. We correct the significance results by applying the Bonferroni correction for multiple comparisons. As expected, the textual character representations show high correlation amongst each other for all five languages. The correlations between the textual embeddings and the perceptual representations show that even though the first are purely trained on written language, they still learn to encode certain inherent characteristics of human language processing and production. As a general pattern, the textual character representations correlate strongly with sound representations, moderately with color representations, and not at all with the shape representations (with the 3 We use cosine distance for all textual, sound and shape representations; and Euclidean distance for color. exception of Korean, discussed below). Japanese character embeddings behave differently. For instance, the correlation with the sound representations is weaker than for the other languages, which might be due to the syllabic nature of the Japanese script. In the following, we discuss the results for each of the perceptual embedding types in detail. The PPMI character embeddings show the highest correlation with sound representations, followed closely by transformer embeddings. This is notable in the three languages with Latin scripts (en, es, nl). To explain this finding, we speculate that the context and learning direction available to the LMs provide phonetic information. While the PPMI embeddings have access to contextual information in both directions, the unidirectional LSTM and transformer learn from left-to-right only. Therefore, as an addition, we trained a bidirectional LSTM (hid-den dimension = 256) to show that the addition of right-to-left information improves the correlation to the sound representations. The results are shown in Table 1. Moreover, comparing the results across Latin script, we note that Spanish character embeddings from all models achieve higher correlations than Dutch and English. The shallow orthography of the Spanish language explains this finding. This is also the case for Korean. Our findings on the correlation between English character embeddings and synesthesia data are in line with Kann and Monsalve-Mercado (2021), who find that LSTMs agree with human letter-color perceptions more than transformers on a dataset with more participants (0.08 for LSTM-LM and 0.0 for transformer-LM). Moreover, we reach the same conclusion for the other alphabetic scripts, Dutch and Spanish, while for Korean and Japanese there is no clear pattern evident from the correlation coefficients. This might be due to the smaller 6824 number of synesthete participants in the dataset. The character embeddings of non-featural Latin scripts show low (or even negative) correlation to the shape embedding. However, due to their featural writing systems (Sampson, 1985; Marjou, 2021), Japanese and especially Korean embeddings correlate significantly with shape. The fact that the Korean consonant graphemes were designed to resemble the place of articulation (Lee, 2021; Gale, 1912), can explain the high correlations between character and shape embeddings for this language. This is also shown in the positive correlation between sound and shape representations, which is absent for the other languages. To analyze this further, we compare our initial results with transformer character representations computed based on Jamos (e.g., individual phonemes such as "  "), to character representations of full Hangul characters (e.g., syllables such as "  "). Table 2 shows higher correlations for characters decomposed into Jamos. The correlation between sound and shape is also lower for full syllables (0.31). In this light, the result is unsurprising and can be interpreted as an effective proof-of-concept of using a correlation analysis between textual and perceptual representations. More genuine shape representations, for example learned by a convolutional neural network, could be applied to reveal more accurate correlation patterns for Latin scripts. Except for Japanese, the results show that the neural embeddings correlate the most with the perceptual sound representations. To get a closer look at the information that may be encoded in the dense embeddings, we design a probing task in which classifiers are trained to predict whether certain distinctive features are present given character embeddings as input. For each distinctive feature, we train a binary Logistic Regression to predict whether the the feature is present (1), or not (0). The labels are given by the sound representations as explained in Section 3.2. As the number of samples is small (limited to the number of characters in a language), we do this in a leave-one-out manner, training a classifier for each character, while using the rest for training. In both test and training, for features that only concern consonants (e.g., manner of articulation and voicing ), we exclude vowels, and similarly, for features that only concern vowels (e.g., vowel height and vowel rounding ), we exclude consonants. The performance of the probes are evaluated for each distinctive feature using F1 scores and by comparison with two baseline strategies, namely, (a) to predict labels uniformly at random, and (b) to always predict the most frequent label according to the training distribution. The former is given as the average across 1000 runs. For some features, choosing the most frequent label is a good strategy and will yield good results. To further challenge the knowledge learned by the embeddings and distinguish the classifiers from the strategy of choosing the most frequent baseline, we create a zero-shot setup in which the classifiers will have to be able to transfer knowledge between features in order to excel in the task. In particular, we test 1) if a classifier trained to predict whether a consonant is voiced is able to identify vowels and 2) if labial consonants are retrieved by a classifier trained to predict vowel rounding. While the intuition behind 1) relates to the sonority sequencing principle (Clements, 1990), which states that the nucleus of a syllable (vowels in the majority of the cases) represents a sonority peak, the intuition behind 2) is more experimental, relying on a global feature such as 'rounding'. The results for the probing classifiers are found in Table 3. Generally, both LSTM and transformer embeddings outperform both the most-frequent and random baselines, with the transformer beating the LSTM by a small margin. This should, however, be taken with a grain of salt considering the limited number of examples. Considering the global features, vowel and consonant , classifiers are able to learn this distinction using both LSTM and transformer character embeddings. In particular, consonants are identified with high certainty. This is, however, the majority group (ref. the most frequent strategy). The F1 scores for vowel prediction are considerably lower. However, in this case they cannot be explained by neither a most-frequent strategy nor a random baseline, which indicates that a global vowel/consonant distinction is captured in the embeddings. The findings for the voiced/voiceless consonant distinction are similar. But here the groups are more balanced, which provides the most-frequent strategy with less of an advantage and in turn the F1 scores are generally lower. For Korean, the scores are lower compared to the other languages. As the feature of consonant voicing correlates with manner in Korean (with all plosives, affricates and fricatives being voiceless, and plosives being the majority class), the task captured by the classifier may be distorted. The fact that the classifier may not be able to pick up features of voicing from the Korean embeddings are reflected in the zero-shot experiment. The results for the first zero-shot experiment for predicting vowels using the classifier for identifying voiced consonants are found in Table 4. Here, the results for Korean are worse than the random baseline. While the results for English and Dutch can be explained by the most-frequent strategy, the result for Spanish indicates that features of voicing or sonority are encoded in the embeddings, amplifying the initial results from the probing classifier experiment. Turning from consonant to vowel features, the inventory of vowels is considerably smaller, leaving a small number of training examples with few positive examples. Thus, the results of the probing classifiers are associated with uncertainty. For the zero-shot task of retrieving consonants with labial features from a classifier trained to predict vowel rounding, we focus our analysis on Spanish LSTM embeddings as they showed the most promising results for predicting rounding in the regular probing 6826 Language F1 True positive False positive False negative es 0.47 b f v w g k q x y z  p m Table 5: Results from the zero-shot task to predict 'rounded' consonants using the Spanish LSTM embeddings. task. However, as can be seen in Table 5, while the classifier for Spanish has a high recall its precision lacks behind and retrieves many false positives. Overall, we believe that the results are promising and a good indication on how character representations can capture features related to phonology. This especially in light of the results from the first zero-shot task, that suggested that classifiers are able to transfer knowledge of sonority from embeddings of consonants to unseen vowels. In this work, we attempted to understand the information encoded in character-level representations. We obtained two main types of embeddings: text-based embeddings and perceptual embeddings. While the first type of representations (PPMI, LSTM, and transformer) were trained from raw text data, perceptual representations were obtained from sources mimicking human language, i.e., pronunciation dictionaries, synesthesia data and shape visualizations. We have performed representational similarity analyses between these types of embeddings for five different languages. Besides, we defined and trained models to predict certain phonological distinctive features in order to interpret the embeddings. We found interesting patterns in the representational similarity analysis as a simple first approach for intrinsic character embedding evaluation. While clearly outperforming a random baseline in most cases, the strength of the correlations vary between scripts. For instance, the strong correlation between Korean character embeddings and shape representations provides positive evidence of the suitability of this approach. Further research is required to dissect the differences between character LMs: While the LSTM embeddings showed stronger correlation with color, the transformer embeddings were supe-rior when compared to sound representations. The inclusion of additional languages and scripts will be helpful to identify more generalizable insights. that they contribute differently for different tasks. For instance, sound representations would be expected to be useful for tasks revolving around phonology, such as grapheme-to-phoneme conversion, or shape representations could be relevant for predicting orthographic errors. The phonological probing tasks show promising results, especially with respect to interpretability. Besides, this methodology is applicable to any language with sufficient raw data and a pronunciation dictionary, and could potentially shed light in measuring the phonological difficulty of certain languages. In future work, we will focus on the development of more sophisticated probes, for instance, multitask networks with shared layers across tasks. Moreover, the labels of the probing task were given from using the sound embeddings retrieved from the most frequent phoneme mapping. Had we focused the analysis on contextual character embeddings instead, that would allow us to distance ourselves from this paradigm as we would be able to analyse character and sound embeddings in the context they occur in. Finally, we stress the need for further intrinsic evaluation methods for character representations. The high impact of orthography on human language learning is an adamant argument to consider the cross-linguistic diversity of writing systems more carefully in the development of NLP models. We thank the anonymous reviewers from ACL Rolling Review for their valuable comments. The first author is supported by the project Script and Text in Time and Space funded by the Velux Foundations.
abstain method abstain objective result abstain objective abstain abstain abstain abstain abstain abstain abstain objective abstain objective method objective method objective method abstain method result abstain other other other other other other other other other other other other other other other other method abstain other other other other other objective other other other other other other other abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain method abstain other abstain method abstain abstain abstain method abstain method abstain abstain abstain method abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain method result abstain method method objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain other other