This work explores techniques to predict Part-of-Speech (PoS) tags from neural signals measured at millisecond resolution with electroencephalography (EEG) during text reading. We show that information about word length, frequency and word class is encoded by the brain at different post-stimulus latencies. We then demonstrate that pretraining on averaged EEG data and data augmentation techniques boost PoS single-trial EEG decoding accuracy for Transformers (but not linear SVMs). Applying optimised temporally-resolved decoding techniques we show that Transformers outperform linear SVMs on PoS tagging of unigram and bigram data more strongly when information requires integration across longer time windows. Electro-/Magnetoencephalography (EEG/MEG), which measures neural activity at millisecond resolution, is a key neuroscientific method to assess how neural representations unfold dynamically in language processing. Early event related potential (ERP) studies that rely on averaging EEG activity across multiple trials have shown that EEG signal magnitude and topography depend on word length, frequency and open vs. closed class. Word length effects arose in EEG at about 150 ms, frequency effects at 200 ms and word class effects from 400-700 ms (Osterhout et al., 1997; Brown et al., 1999; Neville et al., 1992; Mnte et al., 1998; Segalowitz and Lane, 2000; Mnte et al., 2001; Dufau et al., 2015). Recent studies were able to predict these and other (e.g. semantic) aspects based on single trial multi-channel EEG/MEG activity (Ling et al., 2019; Chan et al., 2011; King et al., 2020). Importantly, the aim of cognitive neuroscience studies is to dissociate when (i.e. latency) and where (i.e. brain region) specific linguistic information is explicitly encoded in neural activity. Neuroscience studies therefore typically use linear decoders and try to disentangle neural activity for linguistic and non-linguistic dimensions that covary in natural language statistics (e.g. word class vs. length). By contrast, engineering applications mainly aim to maximise performance accuracy, utilising all available information and more powerful nonlinear classifiers. Intriguingly, recent studies have shown that adding human eye tracking data (Barrett et al., 2016) or morphosyntactic information extracted from human functional magnetic resonance imaging (fMRI) signals during sentence reading, can substantially improve PoS induction (Bingel et al., 2016). Yet, morphosynactic information obtained from fMRI is limited, because fMRI measures only the slow changes in blood oxygenation, peaking 5-6 s after stimulus onset, rather than the rapid neural activity during language processing. Contributions. This interdisciplinary paper decodes PoS tags from EEG signals with linear SVMs and Transformers, pursuing several aims relevant for neuroscience and/or engineering. Neuroscience-focused Section 3 uses linear SVMs to define the distinct neural representations of word length, frequency and class based on a new EEG data set, in which a single subject reads an extensive syntactically annotated corpus. To dissociate these linguistic and non-linguistic aspects, typically correlated in natural language statistics, Section 3 matched the stimulus distributions for each classification task with respect to the confounding dimensions of no interest. Consistent with previous reports, we show that word length, frequency and class can be decoded at different post-stimulus latencies based on single trial and trial-averaged data. This replication part serves to validate the new EEG data set. Methods-focused Section 4 moves beyond open 2201 vs. closed word class decoding that were the focus of previous MEG/EEG studies and decodes 6 PoS tags from EEG activity with linear SVMs and Transformers. We show that pretraining on trial-averaged data with subsequent fine-tuning on single-trial data, alongside data augmentation, boosts PoS decoding accuracy from single-trial EEG data selectively for Transformers (but not for linear SVMs). Engineering-focused Section 5 finally uses linear SVMs and Transformers together with pretraining and augmentation techniques from Section 4 to assess how PoS information about unigrams and bigrams becomes progressively available in EEG activity across post-stimulus time. Comparing EEG decoding from sliding and incremental time windows suggests that Transformers outperform linear SVMs particularly when information needs being integrated across longer time windows. Our results raise the possibility of combining PoS-tagging based on EEG decoding with corpora and dependency tree annotation to obtain more reliable morphosyntactic information for low-resource languages. Our experiments used a new corpus annotated with EEG data, previously acquired at the University of Birmingham following ethical approval and par-ticipant's informed consent. The EEG annotated corpus is available 1 under a public license (CC BY-SA 4.0). Data set . The stimulus set includes 4,479 sentences (74,953 tokens) selected from the English Web Treebank (Bies et al., 2012), covering the genres weblogs, newsgroups, reviews and Yahoo Answers . The mean sentence length is 16.7 words (standard deviation: 12.23). 75 sessions of EEG data are included over 20 days, each lasting 20-25 minutes, from a single subject who read approximately five and a half iterations of the stimulus set (i.e. 24,323 sentences and 404,205 tokens in total, thereby substantially exceeding current freely accessible data sets, e.g. (Bhattasali et al., 2020). Three sessions were excluded because of data corruption. The EEG data for separate text passages were divided into training, dev and test sets to avoid any temporal overlap. Further, dev and test sets were matched for length of text passages, recording 1 https://edata.bham.ac.uk/617 Figure 1: Example trial and EEG recording. dates and sentence position (initial, mid & end). The training set contains 83% of the data (19,156 sentences; 317,753 tokens), dev set 8.5% (2,704 sentences; 45,822 tokens) and test set 8.5% (2,463 sentences; 40,630 tokens). In a Rapid Serial Visual Presentation (RSVP) paradigm, sentences were presented one word at a time, on average every  240 ms, in a white monospace font on a grey background approximately in the centre of the screen, at the optimal viewing position (Rayner et al., 2016). Each word subtended a horizontal angle 0.76  to the left and 11.81  to the right from the centre. Sentences were separated by 500 ms of a white central fixation cross (see Figure 1). On approximately 20% of the sentences in each session, the participant was prompted to verbalise the previous sentence back to the experimenter. An accuracy score of 93% across all sessions confirmed that the participant successfully attended the sentences. Stimuli were presented using PsychoPy (Peirce et al., 2019) on an LCD monitor with a resolution of 1920x1080 pixels and 60 Hz refresh rate. The subject's head was stabilised with a chin-rest. Continuous EEG signals were recorded in 'reference-free' mode at a sampling rate of 1 kHz via BrainVision's PyCorder software using 64 Ag/AgCl active actiCAP slim electrodes arranged in a 1020 layout (ActiCAP, Brain Products GmbH, kept below 15 k . Data were preprocessed using MNE-Python (Gramfort et al., 2013). Individual EEG sessions were band-pass filtered between 1-40 Hz, down-sampled to 250 Hz and re-referenced to average reference. Noisy channels were determined based on visual inspection and interpolated. Non-neuronal components (e.g. ocular, muscular, electrical) were removed via Independent Component Analysis (ICA) individually for each recording session (an average of 4 components were removed per EEG session). EEG signals were extracted from -100 to 700 ms relative to word onset. For baseline correction, we subtracted the channel-wise mean from -100 ms to 0 ms from the evoked post-stimulus EEG response ([0 700] ms separately for each word; see Figure 1). EEG data were spatially multivariate noise normalised using the noise covariance matrix estimated separately for each target class (Guggenmos et al., 2018). Each EEG trial was annotated with the gold part of speech tags of the current and subsequent words, their word lengths, and Zipf-logarithmic frequency scores from the Python package WordFreq (Speer et al., 2018). The decoding analyses used linear support vector machines (SVM) (Chang and Lin, 2011) and Transformers, which can capture complex interactions of EEG data across time points. All classifiers were trained on the EEG training data, assessed on the dev set and scored on the independent test set. Hyperparameters and early stopping were selected based on the dev set. We assessed linear SVMs and Transformers on the dev set using 10 different random seed points. We show mean classification accuracy with 68% confidence intervals (CI) over those 10 replications on the dev (Table 4, Figure 3) resp. test set (Figure 2, 4, 5). We compute statistics on test set classification responses from the model that scored the highest on the dev set (e.g. binomial or Wilcoxon signed rank tests). For linear SVM, we used an online learning implementation of SCIKIT-LEARN (Pedregosa et al., 2011; Zhang, 2004), based on LIBSVM (Chang and Lin, 2011), with hinge loss and Stochastic Gradient Descent (SGD) optimiser. Hyperparameters were set to default except for the SGD regularisation parameter that was increased to  = 0 . 75 , which provided better classification accuracy on the dev set. The parameter  is inversely proportional to the C parameter in the standard SVM implementation. The online implementation also allowed us to select the best model using early stopping. The SVM was provided with EEG activity vectors as inputs, i.e. 1 x (EEG channels  time points). For the Transformer (Vaswani et al., 2017), we conducted a model architecture and hyperparame-ter search (layers, learning rate, MLP dimensions, dropout rate, Encoder vs. Encoder-Decoder) on the dev set. The selected model was composed of four encoder-blocks and a final dense layer that projects the output of the last encoder-block onto the PoS tags via a softmax function. We used the Adam optimiser and early stopping. The implementation is based on the WMT example 2 of Google's novel ML frameworks Flax/Jax . Table 2 lists the selected hyperparameters. The Transformer received EEG channels x time points as inputs and provided a classification response for the entire time window. We performed decoding based on (i) EEG for single-trials (i.e. no averaging), (ii) EEG averaged across three and (iii) ten trials. Averaging EEG 2 https://github.com/google/flax/tree/master/examples/wmt 2203 signals across trials increases the signal to noise ratio of the 'samples' (Grootswagers et al., 2017; Guggenmos et al., 2018; Roy et al., 2019; Tuckute et al., 2019), but ignores true variability across EEG data from different words of the same category (Mnte et al., 2001). For training (resp. dev) set we generated the same number of samples for 3 and 10 trial averages as for the single trial test (resp. dev) sets via boostrapping. For the test set, we averaged data without replacement, so that examples can be entered as independent data points in statistical tests. Hence, the number of samples in the test set (but not in the training or dev sets) is smaller for 3 and 10 trial averages than single trials (Table 3). For comparison with previous research (Osterhout et al., 1997; Mnte et al., 2001), we decoded word length, frequency and class with linear SVMs in a temporally-resolved fashion from 0 to 700 ms post-stimulus post-stimulus EEG, recorded during sentence reading. Data set . We decoded word class from EEG via binary classification between open class words (i.e. NOUN , VERB , ADJ , and PROPN ) vs. closed class words (i.e. DET , ADP , AUX , PRON , SCONJ and CCONJ ). Likewise, for decoding word frequency and length, words were assigned to two classes based on the median values in the data set (i.e. Zipf-frequency > 5.91 = HIGH else LOW ; word length > 4 characters = LONG , else SHORT ). EEG decoding analyses were performed for single-trials, averages across 3 and 10 trials. To dissociate the distinct contributions word length, frequency and class that are highly correlated in natural language statistics, we decoded one variable by controlling for the other two variables. For instance, when decoding open vs. closed class words, we selected a subset of trials such that the joint distributions over the three confounding variables of word frequency (discretised to the nearest 0.25), length (number of characters) and sentence position (i.e. sentence initial, mid, end) were equated for the categories of open and closed class words. To minimise confounds arising from the preceding word in the sentence, we balanced the test set with respect to the open/closed class status of the previous word. Similarly, we controlled the decoding of word frequency for word length, and the analysis of word length for word frequency, and both analyses for open/closed class and sentence position. Table 3 gives the number of examples for length frequency class train 82,424 51,364 45,502 dev 12,402 7,590 5,670 test (single) 10,810 6,590 5,670 test (avg. 3) 3,603 2,196 1,890 test (avg. 10) 1,081 659 567 Table 3: Number of samples across train, dev and test set in the confound-controlled data set. each analysis across training, dev and test sets. Methods . To temporally resolve how the brain encodes word length, frequency and class, we trained and tested linear SVMs on EEG signals separately for sliding windows of 64 ms (i.e. 16 time points) that shift in increments of 4 ms (i.e. one time sample). Figure 2 shows the mean accuracy values (averaged across 10 seed points) from the test set (centred on the last bin of each time window (Grootswagers et al., 2017)) with  68% CI. The classification responses for the test set from the model that performed best on the dev set were entered into a two-sided binomial test, separately for each time window. Solid lines in Figure 2 above the decoding accuracy time courses indicate time points that were significant at ( p < 0 . 05 ) False Discovery Rate (FDR) corrected for multiple comparisons (Rouam, 2013) across time (i.e. 160 tests). Results . Figure 2 (top rows of A, B, C) shows butterfly plots for the effects of word length, frequency and class across 64 electrodes. Our linear SVM decoding analysis replicates the temporal cascade of word length, frequency and class effects previously reported for EEG responses averaged across a large number of trials. The word length effect arises early at about 100 ms, previously associated with visual word processing in occipitotemporal cortices (Hauk and Pulvermller, 2004; Pul-vermller et al., 2009; Schuster et al., 2016). Word frequency influenced neural processing later from 200 ms onwards with a slight left-hemispheric predominance (Griffiths et al., 2012). The word class effect emerged in early and late time windows with the effect at about 550 ms in line with the wellknown P600 as an ERP indicator for syntactic processing (Osterhout and Holcomb, 1992; Hagoort et al., 1993; ter Keurs et al., 1999). Word length and frequency effects were stronger than the word class effect; see King et al. (2020). As expected, decoding accuracy increased when EEG signals were averaged across trials. Thus, carefully controlling each comparison of interest (e.g. word class) for the effects of no interest (e.g. word length and 2204 Figure 2: Butterfly plots for difference waves across all 64 channels (top rows, left), topographies (top rows, right) and time courses of decoding accuracy (bottom rows). frequency) enabled us to dissociate word length, frequency and class effects, despite their high correlation in natural language, thereby validating our new annotated EEG data set and analysis procedure. Moving beyond open/closed word class decoding, we assessed whether multi-class PoS decoding with SVMs and/or Transformers can be improved by (i) data augmentation, i.e. increasing the number of samples in the training set via bootstrapping and re-averaging (only applicable to 3 and 10 trial averages) and (ii) pretraining on trial-averages followed by fine-tuning of the model parameters on single-trial EEG data. Data set . We focused on decoding of 3 open class ( NOUN , VERB , PROPN ) and 3 closed class ( ADP , DET , PRON ) PoS tags. From the word class dataset that was controlled for length and frequency effects (section 3), we selected an equal number of examples per PoS class (i.e. train: 3,470, dev: 335, test: 335, i.e. in total  20k data points; word frequency of each word > median frequency Zipf value (5.91). The samples for dev and test sets were matched for distribution of word lengths. Methods . Using this 6-class unigram dataset, we assessed whether data augmentation via bootstrapping and re-averaging increases decoding performance for the 3 and 10 trial averages. We sampled 3 (resp. 10) individual trials with replacement from a particular PoS class and averaged them in 3 (resp. 10) trial averages. We thus trained SVMs and Transformers (over 20 random seeds) on 4 training set sizes: N size = {20 k , 100 k , 250 k , 500 k }  2 levels of trial averaging (3 vs. 10) resulting in 8 training sets. The baseline training (resp. dev) set included as many 3 (resp. 10) trial averages as the initial single-trial training (resp. dev) set. Results . Data augmentation systematically boosted the decoding accuracy of the Transformer but not of the SVM most likely because of the former's greater model complexity. For both 3 and 10 trial averages the Transformer's decoding accuracy on the dev set increased from a training set size of 20k to 100k, peaking at 250k. It then declined for an even larger training size of 500k potentially because continued bootstrapping pro-2205 Figure 3: Dev set decoding accuracy (mean across seeds) for SVM (blue) and Transformer (red) separately for 3 and 10 trial averages and four levels of data augmentation: 20k (original), 100k, 250k and 500k. gressively generates dependencies amongst training samples thereby limiting their additional benefit beyond 250k. We formally assessed whether the Transformer that scored best on the dev set obtained better decoding accuracy for 250K than for the original 20k training set (n.b. we performed this statistical test on the test set, because the 3 and 10 trial averages within the dev set were not independent from one another as a result of boostrapping). Indeed, for both 3 and 10 trial averages the Transformer's (but not the SVM's) decoding accuracy was significantly better for 250k than the original 20k training set ( p < 0 . 01 ; Wilcoxon signed-rank test). Methods . The ultimate goal is to decode PoS from single-trial EEG data (rather than trial av-erages). We therefore assessed whether pretraining the SVM and/or Transformer on trial averages with subsequent fine-tuning on single-trial data increases decoding accuracy. Pretraining may be beneficial because trial averages have a greater signal to noise ratio. Specifically, we assessed the impact of pretraining in a 2  2 factorial design manipulating i) pretraining scheme: training in three steps (10-3-1) from 10 trial averages to 3 trial averages to single-trials vs. training in two steps (3-1) from 3 trial averages to single-trials and ii) data augmentation: training only on the original 20k vs. SVM Transformer single-trials 31.93 (  0 . 62) 37.15 (  0 . 32) 10-3-1 31.74 (  0 . 51) 38.5 (  0 . 28) 10-3-1 (250k) 31.89 (  0 . 67) 39.17 (  0 . 33) 3-1 32.03 (  0 . 52) 37.83 (  0 . 24) 3-1 (250k) 31.79 (  0 . 58) 39.41 (  0 . 41) Table 4: Single-trial decoding accuracies (%, mean across seeds  68% CI) on dev set for the SVM and Transformer: without pretraining, with 10-3-1 pretraining, with 10-3-1 and 250k data augmentation, with 3-1 pretraining, with 3-1 and 250k data augmentation the 250k data set, which obtained highest dev set performance in section 4.1. We trained both SVMs and Transformers on the 2  2 training conditions using 20 random seeds and report mean accuracy (  68% CI, across those 20 seeds) in Table 4. Results . For the SVM, the 3-1 pretraining without data augmentation resulted in the highest dev set accuracy (32.03%), though accuracy was only slightly better than for direct single-trial training (31.93%). For the Transformer, the 3-1 pretraining scheme with 250k data augmentation obtained the highest single-trial decoding accuracy (39.41%) on the dev set. Indeed, Wilcoxon signed-rank test (Pereira et al., 2009) confirmed that the best dev set Transformer performed significantly better on the test set after 3-1 pretraining than after direct single-trial training ( p < 0 . 01 ). Sections 3 and 4 were driven by the neuroscience goal of dissociating neural representations associated with PoS from confounding factors such as word length or frequency, which are typically correlated with PoS in natural language statistics. To control for these confounds sections 3 and 4 generated data sets, in which e.g. PoS classes were equated with respect to word length. By contrast, Section 5 pursues the engineering goal of maximising PoS decoding accuracy. Here, correlations between word frequency, length and PoS class are no longer considered a confound but a useful source of information. Capitalising on the optimised 3-1 training scheme with 250k data augmentation from Section 4, section 5 will assess whether PoS information about unigrams and bigrams can be decoded from EEG signals (without any confound controls). For both unigrams and bigrams, we will first investigate how PoS information becomes available dynamically across post-stimulus time by training SVMs and Transformers on EEG signals from 64 2206 ms sliding time windows. Second, we will assess how SVMs and Transformers integrate PoS information across post-stimulus time by training them on EEG signals from incremental time windows. Data Set . We selected an equal number of examples for the 6 most frequent PoS tags (i.e. NOUN , VERB , ADP , DET , PRON & PROPN ) from the data set matched for length of text passage, recording dates and sentence position, but not for word length or frequency. Each PoS class included the following number of samples train: 28,265 , dev: 2,948, test: 3,183 examples. Methods .We implemented the 3-1 pretraining with 250k data augmentation (section 4). For the sliding window analysis, we trained and tested SVMs and Transformers on EEG signals from 64 ms windows (i.e. 16 time points) that shifted in increments of 16 ms from 0 ms to 700 ms (i.e. resulting in a sequence of 41 decoding accuracies). For the incremental window analysis, we successively increased the initial [0 16] ms time window (i.e. 4 samples) by 4 additional sampling points resulting in a temporal sequence of 44 decoding accuracies. We computed decoding accuracies (mean across seeds,  68% CI) from the test set. Across time windows we compared the decoding accuracies on the test set of the best dev set SVM and Transformer using the Wilcoxon signed rank-test (reported at p < 0 . 05 , FDR-corrected for multiple comparisons across time i.e. 41 resp. 44 tests). Results . In the sliding window analysis, the decoding accuracies of SVMs and Transformers show two prominent peaks at 200 ms and 400 ms suggesting that PoS decoding relies on several aspects of information encoded in the EEG. Based on our confound-controlled analysis (section 3) the first peak reflects word length and frequency information, while the second peak is more closely related to semantic and syntactic aspects of the word. The incremental window analysis showed an accuracy benefit of 4.5% for the Transformer starting in the very first [0 16] ms time window. This difference in performance between the two models further widened, reaching a maximum advantage of 11.6% around 360 ms. Transformers thus benefit from integrating information about word frequency, length and class that arise at different post-stimulus latencies. Moreover, because PoS classes of subsequent words are correlated in natural language, the TransFigure 4: Unigram results: Test set decoding accuracies (mean across seeds  68% CI), aligned with last bin of each time window. Top: Incremental window analysis. Middle: Sliding window analysis. Bottom: ERP for NOUN , i.e. EEG averaged across all examples from the training set. Vertical lines indicate word onset times. All time windows are significant at p < 0 . 05 FDR-corrected. former may also benefit via self-attention from information about the next word that is presented and progressively encoded in EEG activity about 240 ms after the current (i.e. to be decoded) word. Statistical testing confirmed that the Transformer significantly outperformed the SVM for all sliding and incremental windows (Wilcoxon-signed rank test, FDR-corrected at p < 0 . 05 ). To define the contributions of successive words to EEG PoS decoding in naturalistic text reading, we designed a bigram data set that artificially removes the correlations between PoS classes of subsequent words, though we note that this does not fully remove correlations between specific word tokens. Data set . We selected 6 bigrams, in which each first word's PoS is combined equally often with two different PoS classes from the second word: NOUN-PRON , NOUN-VERB , PRON-NOUN , PRON-VERB , VERB-NOUN and VERB-PRON . As a result, the PoS class of word 1 is uninformative about the PoS class of word 2 and vice versa. Hence, prior to the presentation of word 2, the maximal possible decoding accuracy for a particular bigram is 50%. Each bigram class included the 2207 Figure 5: Bigram results: Test set decoding accuracies (mean across seeds  68% CI), aligned with last bin of each time window. following number of samples train: 3,470, dev: 322, test: 349 examples. Results Similar to the unigram results, the sliding window analysis revealed two prominent accuracy peaks at 200 ms and 500 ms. Yet, the 2nd peak was slightly later than in the unigram analysis and it was higher than the first peak only in the bigram, but not the unigram analysis. These differences between unigram and bigram decoding profiles arise, because EEG at 500 ms encodes semantic and syntactic aspects of word 1 and crucial information about word 2 of the bigram. As shown in Figure 5, the Transformer significantly outperformed the SVM in the sliding and incremental window analyses. Yet, in contrast to the unigram results, the Transformer outperformed the SVM in the incremental analysis only for windows 0-208 ms and 0-336 ms. The Transformer's smaller benefit arises mainly because our balanced design radically reduced the number of examples and thereby the Transformer's generalisation ability. It also removed the natural correlations between subsequent words on which the Transformer may have additionally capitalised in the unigram data. The confound-controlled analysis dissociated word length, frequency and class effects in EEG. This replication of earlier ERP (Osterhout et al., 1997; Mnte et al., 1998) and MEG decoding work (King et al., 2020) validates a new EEG data set for an extensive morphosyntactically gold annotated corpus; c.f. Bhattasali et al. (2020). Transformers successfully decoded 6 PoS tags from single trial EEG with data augmentation and 3-1 pretraining (  40% accuracy), raising the possibility to boost PoS induction with EEG-decoded PoS tags. While we acknowledge that our results are limited to EEG data from a single subject, given the spatial smoothness of EEG scalp topographies, we envision pretraining on EEG obtained from different participants. Further, because human brains generate similar neural signatures for word classes across different languages (c.f. Yudes (2016); Mnte et al. (2001); Hagoort et al. (2003)), pretraining PoS-EEG decoders on large morphosyntactically annotated EEG datasets for English followed by fine-tuning on a smaller annotated EEG data set for a low-resource language may enable successful generalisation to EEG obtained from reading non-annotated texts in this low-resource language. PoS-induction jointly based on annotated texts and EEG signals could thus be transformative for corpus generation of low-resource languages. Combining neural signals measured at millisecond resolution with EEG and a linguistically annotated corpus, this work shows to the best of our knowledge the first time that unigram and bigram PoS tags can be decoded successfully from single-trial EEG data. Temporally-resolved EEG decoding unraveled how information about linguistic and nonlinguistic aspects evolved dynamically across time. Unsurprisingly, Transformers with self-attention mechanisms outperformed SVMs across all experiments. In particular, they benefited from integrating information across time, data augmentation and pretraining methods. Our work paves the way for future applications that incorporate human brain signals in traditional NLP methods.
objective result objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain result method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain method abstain method abstain method abstain abstain abstain method abstain abstain abstain other method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other other other abstain other other objective abstain abstain abstain abstain