ER-TEST: Evaluating Explanation Regularization Methods for NLP Models

Neural language models' (NLMs') reasoning processes are notoriously hard to explain. Recently, there has been much progress in automatically generating machine rationales of NLM behavior, but less in utilizing the rationales to improve NLM behavior. For the latter, explanation regularization (ER) aims to improve NLM generalization by pushing the machine rationales to align with human rationales. Whereas prior works primarily evaluate such ER models via in-distribution (ID) generalization, ER's impact on out-of-distribution (OOD) is largely underexplored. Plus, little is understood about how ER model performance is affected by the choice of ER criteria or by the number/choice of training instances with human rationales. In light of this, we propose ER-TEST, a protocol for evaluating ER models' OOD generalization along three dimensions: (1) unseen datasets, (2) contrast set tests, and (3) functional tests. Using ER-TEST, we study three key questions: (A) Which ER criteria are most effective for the given OOD setting? (B) How is ER affected by the number/choice of training instances with human rationales? (C) Is ER effective with distantly supervised human rationales? ER-TEST enables comprehensive analysis of these questions by considering a diverse range of tasks and datasets. Through ER-TEST, we show that ER has little impact on ID performance, but can yield large gains on OOD performance w.r.t. (1)-(3). Also, we find that the best ER criterion is task-dependent, while ER can improve OOD performance even with limited and distantly-supervised human rationales.

Introduction
Neural language models (NLMs) have achieved state-of-the-art performance on a broad array of natural language processing (NLP) tasks (Devlin * Equal contribution. Figure 1: Explanation Regularization: Given an instance and a target label, we can use rationale extractors (See Section 2) to generate machine rationales from a model F. Furthermore, human rationales are collected from annotators. Explanation Regularization (ER) aligns machine rationales to human rationales with a loss term, L ER , which is then used to refine F. et al., 2018;Liu et al., 2019). Even so, NLMs' reasoning processes are notoriously opaque (Rudin, 2019;Doshi-Velez and Kim, 2017;Lipton, 2018), which has spurred significant interest in designing algorithms to automatically explain NLM behavior (Denil et al., 2014;Sundararajan et al., 2017;Camburu et al., 2018;Rajani et al., 2019;Luo et al., 2021). The majority of this work has focused on rationale extraction, which explains a NLM's output on a given task instance by highlighting the input tokens that most influenced the output (Denil et al., 2014;Sundararajan et al., 2017;Li et al., 2016;Jin et al., 2019;Lundberg and Lee, 2017).
Recently, a number of works have investigated how machine rationales produced by rationale extraction algorithms can be operationalized to improve NLM decision-making (Hase and Bansal, 2021) (See Figure 1). Almost all prior works are based on explanation regularization (ER), which aims to improve NLM generalization by regularizing the NLM to yield machine rationales that align with human rationales (Ross et al., 2017;Huang et al., 2021;Ghaeini et al., 2019;Zaidan and Eis-ner, 2008;Kennedy et al., 2020;Rieger et al., 2020;Liu and Avci, 2019). Human rationales can be created by annotating each training instance individually (Lin et al., 2020;Camburu et al., 2018;Rajani et al., 2019) or by applying task-level human priors across all training instances (Kennedy et al., 2020;Rieger et al., 2020;Ross et al., 2017;Liu and Avci, 2019).
Although prior works primarily evaluate such ER models via in-distribution (ID) generalization (Zaidan and Eisner, 2008;Lin et al., 2020;Huang et al., 2021), out-of-distribution (OOD) generalization is more crucial in many real-world settings (Chrysostomou and Aletras, 2022;Ruder, 2021), yet ER's impact on OOD generalization is largely underexplored (Ross et al., 2017;Kennedy et al., 2020). Plus, despite them being major factors in ER, little is understood about how ER model performance is affected by the choice of ER criterion or by the number/choice of training instances with human rationale supervision. In light of this, we propose ER-TEST, a protocol for evaluating ER models' OOD generalization along three dimensions: (1) unseen datasets, (2) contrast set tests, and (3) functional tests. For (1), ER-TEST assesses ER models' task performance on datasets beyond their training distribution. For (2), ER-TEST assesses ER models' sensitivity to counterfactual instances created by perturbing existing datasets. For (3), ER-TEST assesses ER models' basic linguistic capabilities (e.g., perception of word/phrase sentiment, robustness to typos) expected for the given task.
Using ER-TEST, we study three key questions: (A) Which ER criterion are most effective for the given OOD setting? (B) How is ER affected by the number/choice of training instances with human rationales? (C) Is ER effective with distantly supervised human rationales? ER-TEST enables comprehensive analysis of these questions by considering a diverse range of text classification tasks and datasets. Through ER-TEST, we show that ER has little impact on ID performance (Sec. 5.3.1, 5.4.1), but can yield large gains on OOD performance (Sec. 5.3.2,5.4.2) w.r.t. (1)-(3). Also, we find that the best ER criterion is task-dependent (Sec. 5.3), while ER can improve OOD performance even with limited human rationale supervision (Sec. 5.4). Furthermore, we show that ER can improve OOD performance using distantly supervised human rationales (Sec. 5.5).

Background
Text Classification Let F be a NLM task model for M -class text classification. In modern NLP systems, F usually has a BERT-style architecture (Devlin et al., 2018), consisting of a Transformer encoder (Vaswani et al., 2017) followed by a linear layer with softmax classifier. Let x i = [x t i ] n t=1 be the n-token input sequence (e.g., a sentence) for task instance i. For sequence classification, F predicts a class for sequence x i , so let F(x i ) ∈ R M be the logits for x i . Let y i = arg max c F(x i ) c denote F's predicted class for x i . For token classification, F predicts a class for each token x t i , so let F(x i ) ∈ R n×M be the logits for the n tokens in x i . Let y i,t = arg max c F(x i ) t,c denote F's predicted class for x t i . Let y i = [y i,t ] n t=1 collectively denote all of F's predicted token classes for x i .
Rationale Extraction Given F, x i , and y i , the goal of rationale extraction is to output machine rationale r i = [r t i ] n t=1 , such that each r t i ∈ [0, 1] is an importance score indicating how strongly token x t i influenced F to predict class y i . Let G denote a rationale extractor, such that r i = G(F, x i , y i ). G first computes raw importance scores s i ∈ R n , then normalizes s i as probabilities r i using the sigmoid function. In general, G can be a heuristic or learned function, but we focus on heuristic G in this work, since they are more common (Luo et al., 2021). Broadly, heuristic G's can either be a gradient-based, that assign importance scores based on gradient changes in F (Sundararajan et al., 2017;Sanyal and Ren, 2021;Shrikumar et al., 2017), sampling-based, that assign important scores based on the neighbours/context of a given token (Zeiler and Fergus, 2013;Jin et al., 2019), or attentionbased, that use the attention-scores or a function of them to assign importance scores. (Ding and Koehn, 2021) Explanation Regularization (ER) However, G can also be used to compute machine rationales w.r.t. other classes besides y i , e.g., target classẏ i . Letr i denote the machine rationale for x i w.r.t.ẏ i . Givenr i obtained via G and F, many works have explored ER, in which F is regularized such thatr i aligns with human rationaleṙ i (Zaidan and Eisner, 2008;Lin et al., 2020;Rieger et al., 2020;Ross et al., 2017). Typically,ṙ i is a binary vector, where ones and zeros indicate positive (important) and negative (unimportant) tokens, respectively. ER's inductive bias pushes F to solve the task in a way that follows the human reasoning process given bẏ r i , which ideally provides denser learning signal for improving F's generalization.
We formalize the ER loss as: L ER = Φ(r i ,ṙ i ), where Φ is an ER criterion measuring the alignment betweenr i andṙ i . Thus, the full learning objective is: L = L task + λ ER L ER , where L task is the task loss (e.g., cross-entropy loss) λ ER ∈ R is the ER strength (i.e., loss weight) for L ER . Let γ ER > 0 be the rationale scaling factor, used to scaleŝ i prior to sigmoid normalization. If the magnitudes of theŝ i scores are lower, then ther i scores will be closer to 0.5 (i.e., lower confidence). However, scalingŝ i by γ ER > 1 will increase the magnitude ofŝ i , yieldinĝ r i scores closer to 0 or 1 (i.e., higher confidence). Though there are many possible choices for Φ, it is presently unclear how different Φ impact training and when certain Φ should be preferred. This limits our ability to use ER in real-world settings.

ER-TEST
Existing works primarily evaluate ER models via ID generalization (Zaidan and Eisner, 2008;Lin et al., 2020;Huang et al., 2021), though a small number of works have done auxiliary evaluations of OOD generalization (Ross et al., 2017;Kennedy et al., 2020;Rieger et al., 2020). However, these OOD evaluations have been relatively small-scale, only covering a narrow range of OOD generalization aspects, ER criteria, training settings, tasks, and datasets. As a result, little is understood about ER's impact on OOD generalization. To address this gap, we propose ER-TEST, a unified benchmark for evaluating ER models' OOD generalization along three dimensions: (1) unseen datasets; (2) contrast set tests; and (3) functional tests.

ID Generalization
While ER-TEST's main focus is on evaluating OOD generalization, ER-TEST also considers ID generalization as a baseline evaluation. Let are the target classes, and N is the number of instances (x i ,ẏ i ) in D. We call D the ID dataset. Assume D can be partitioned into train set D train , dev set D dev , and test set D test , where D test is an ID test set for D. After using ER to train F on D train , we measure F's task performance on the ID test set D test . Note that this is a standard protocol used by existing works to evaluate ER models (Zaidan and Eisner, 2008;Rieger et al., 2020;Liu and Avci, 2019;Ross et al., 2017;Huang et al., 2021;Ghaeini et al., 2019;Kennedy et al., 2020).

OOD Generalization
To assess F's generalization ability when using ER, we consider various OOD settings. Given D, let D test denote an OOD test set, with a different distribution from D. While F is expected to perform well on D test (ID), F should also perform well oñ D test (OOD). For each dimension of OOD generalization, we obtainD test in a different manner.

Unseen Datasets
First, we evaluate OOD generalization w.r.t. unseen datasets. Besides D, suppose we also have a set of datasets {D (1) ,D (2) , ...} of the same task as D. Each of these datasetsD (i) has its own train/dev/test sets and a distribution shift from D. After using ER to train F on D train , we measure F's task performance on each OOD test setD In other words,D (i) test is obtained by simply taking the test set of existing OOD datasetD (i) . This evaluation is designed to assess whether ER helps F learn general (i.e., task-level) knowledge representations that can (zero-shot) transfer across datasets.

Contrast Set Tests
Second, we evaluate OOD generalization under meaningful dataset perturbations. Annotation artifacts (Gururangan et al., 2018) are gaps present in a dataset that can lead to misleading interpretations of a model's performance on that dataset. To mitigate this, we evaluate F on contrast sets (Gardner et al., 2020), which are (mostly) labelchanging small perturbations on instances to understand the true local boundary of the dataset. Essentially, they help us understand if F has learnt any dataset-specific shortcuts. GivenD p is applied to that instance, where p denotes the kind of perturbation taking effect, and it often changes the target label for that instance. For example, p can signify semantic (e.g., changing tall to short), numeral (e.g., changing one dog to three dogs), or entities (e.g., changing dogs to cats). Each perturbation type is specific to the dataset it is being created for, so that instance labels are changed in a meaningful manner. The resulting set of instances test (j))∀j, p are termed as a contrast set for that dataset. Based on the way they are created, contrast sets are a property of the dataset, and are not created to explicitly challenge F (unlike adversarial examples (Gao and Oates, 2019)).

Functional Tests
Third, we evaluate OOD generalization w.r.t. functional tests (Ribeiro et al., 2020;Li et al., 2020). Unlike contrast sets which are designed to test artifacts present in a dataset, functional tests are used to provide 'zoomed-in' insights about specific linguistic capabilities (like changes in the vocabulary, adding negations to instances, etc). Furthermore, contrast sets are created by perturbing a reference real-world dataset, whereas, functional tests evaluate specific capabilities with the help of templategenerated synthetic instances.
If ER consistently improves F's performance on such tests, then we can have higher confidence that ER is a useful inductive bias for OOD generalization for that given capability Across all tasks, ER-TEST considers four categories of stress tests, which are adopted from CheckList (Ribeiro et al., 2020). Each test is described below.
Vocabulary Tests Vocabulary tests are used to evaluate F's capability to address changes in the vocabulary of the text, and is particularly diverse w.r.t the parts-of-speech it caters to. For example, certain vocabulary tests evaluate the relationship (taxonomy) between different nouns in a sentence, whereas some swap the modifiers or the verbs present in a sentence in a meaningful manner based on the task at hand, to capture F's targeted performance towards such changes (Ribeiro et al., 2020).
Robustness Tests Robustness tests evaluate F's behavior under character-level edits to words in a sentence, keeping the rest of the context same so as to not change the overall prediction. They include testing against typos as well as contractions in words, as well as addition of tokens that are irrelevant for the downstream task (like URLs or gibberish like Twitter handles). (Jones et al., 2020;Wang et al., 2020) Logic Tests Testing F's reasoning capabilities towards logical changes in a sentence is also important to evaluate its reliance on shortcut-patterns. These tests perturb sentences in a logical manner (by adding or removing negations, or purposefully inducing contradictions) that also change the target label in the same manner. (Talman and Chatzikyriakidis, 2018;McCoy et al., 2019) Entity Tests For certain tasks, named entities like numbers, locations and proper nouns are not relevant for predicted a target label, and are often a source of gender or demographic biases (Mishra et al.;Mehrabi et al., 2020). Entity tests measure F's sensitivity towards changed in named entities such that the overall context as well as the task label remains the same (Ribeiro et al., 2020).

Tasks and Datasets
To evaluate ER models, ER-TEST considers a diverse set of sequence and token classification tasks. For each, task ER-TEST provides one ID dataset (annotated with human rationales) and multiple OOD datasets. Compared to prior works, ER-TEST's task/dataset diversity enables more extensive analysis of ER model generalization.
First, we have sentiment analysis, using SST (movie reviews) ( Figure 3: Analysis Setup: We use ER-TEST to answer three RQs in this work -RQ1 addresses which ER criteria work for which task, and evaluates their improvements, RQ2 evaluates whether a particular method of selecting a fixed budget of samples for annotating human rationales yields performance improvements, and RQ3 studies ER w.r.t. another dimension of resource constraints, where we can only provide distant supervision to generate human rationales. We lay out more details in Section 4. 2020) as the ID dataset. For OOD datasets, we use Yelp (restaurant reviews) (Zhang et al., 2015), Amazon (product reviews) (McAuley and Leskovec, 2013), and Movies (movie reviews) (Zaidan and Eisner, 2008;DeYoung et al., 2019). Movies' inputs are much longer than the other three datasets'. For contrast set tests, we use an OOD contrast set for sentiment analysis released by the authors of the original paper (Gardner et al., 2020), which are created for the Movies dataset. Furthermore, for functional tests, we use an OOD test suite (flight reviews) from the CheckList (Ribeiro et al., 2020) which contains both template-based instances to test linguistic capabilities, as well as real-world data (tweets).

Distantly Supervised Human Rationales
Typically, human rationales are created by annotating each training instance individually (Lin et al., 2020;Camburu et al., 2018;Rajani et al., 2019). For each training instance, humans are asked to mark tokens that support the gold label as positive, while the remaining tokens are marked as negative.
Here, each human rationale is specifically conditioned on the input and gold label for the given instance. However, such instance-level human rationales are very expensive to obtain, given the high manual effort per instance.
Alternatively, some works have constructed distantly supervised human rationales by applying task-level human priors across all training instances (Kennedy et al., 2020;Rieger et al., 2020;Ross et al., 2017;Liu and Avci, 2019). For example, Kennedy et al. (2020) used a "blacklist" lexicon to distantly supervise human rationales for the hate speech detection task. In the past, hate speech detection models were largely oversensitive to certain group identifier words (e.g., "black", "Muslim", "gay"), almost always predicting hate speech for text containing these words. To address this, they first manually annotated a lexicon of group identifiers that should be ignored for hate speech detection. Then, for all training instances, they automatically marked only tokens belonging to the lexicon as negative (and the rest as positive). By using these human rationales for ER, they trained the NLM to be less biased w.r.t. these group identifiers. For the purpose of our study, we use the lexicons as used by (Jin et al., 2021) to generate distantly-supervised rationales for the Stormfromt (Stf) dataset (de Gibert et al., 2018). Each instance in the Stf dataset is matched to one or more lexicons by simple character-level matching, and the rationales are generated as described above. We evaluate ER methods on OOD hate speech detection datasets like HatEval (Barbieri et al., 2020) and Gab Hate Corpus (GHC) (Kennedy et al., 2018). All of the datasets contain binary labels for hateful and non-hateful content. The Stf dataset is collected from a white-supremacist forum, whereas HatEval instances are tweets and GHC instances are taken from the Gab forum.

Analysis Setup
After introducing the ER-TEST framework, we conduct a systematic study of ER through three primary research questions (described below) using ER-TEST ( Figure 3). First, we aim to study which ER criteria are effective for a given task at hand (Section 4.1). Second, we study ER from a resource constraint perspective, where only a handful of instances can have human rationales annotated. What is key here is to define how to select these instances for rationale annotation (Section 4.2). Lastly, we look at human rationales that are distantly-supervised and how ER can help improve performance when dense, instance-level human rationale annotations are not available (Section 4.3).
4.1 RQ1: Which ER criteria are most effective?
Compared to existing works, ER-TEST uses a wider range of ER criteria to evaluate ER model generalization. This provides a more comprehensive picture of ER's impact on both ID and OOD generalization. Also, this can help us understand why certain criteria work well and under what settings they work best. To demonstrate the utility of ER-TEST, we consider five representative ER criteria (i.e., choices of Φ).

Mean Absolute Error (MAE) MAE is used in
Binary Cross Entropy (BCE) BCE loss is used in Chan et al. (2021).
Huber Loss Huber loss (Huber, 1992) is a hybrid of MSE and MAE. ΦHuber(ri,ṙi) Order Loss Recall that the human rationaleṙ i labels each token as positive (one) or negative (zero). Whereas other criteria generally push positive/negative tokens' importance scores to be as high/low as possible, order loss (Huang et al., 2021) relaxes MSE to merely enforce that all positive tokens' importance scores are higher than all negative tokens' importance scores. This is especially useful ifṙ i is somewhat noisy, e.g., some positivelylabeled tokens should not really be positive.
4.2 RQ2: How is ER affected by the number/choice of train instances with human rationales?
In real-world applications, it is infeasible to obtain human rationalesṙ i for all of the instances in the training set, asṙ i requires dense annotation (Chiang and Lee, 2022;Kaushik et al., 2019). Let S be a subset of train instances for which we have human rationale annotations,ṙ S i . Therefore, the ER loss L S ER = Φ(r S i ,ṙ S i ) and the full learning objective L = L task + λ S ER L S ER , where L task is computed on the full dataset as it would normally.
In designing such a system, one needs to carefully select S that leads to highest performance gains, meanwhile maintaining resource constraints. To select relevant samples to annotate, existing methods use to active-learning based approaches (Schröder and Niekler, 2020). We use ER-TEST to compare three such approaches approaches to select S: Random Sampling Given a k, we uniformly select k% of samples from D to construct S.
Lower Confidence (LC) Sampling Given a k, we select top k% of samples ordered on the basis of the Lower Confidence criterion. (Zheng and Padmanabhan, 2002) where (x i ,ẏ i ) ∈ D train , and θ are the parameters of a model trained on D train without ER. In other words, these are the top k% examples that a model trained without ER is the least confident on.
Higher Confidence (HC) Sampling Given a k, we sample the top k% of samples ordered in the reverse order of lower confidence prioritisation as described above. In other words, these are the top k% examples that a model trained without ER is the most confident on.

RQ3: Is ER effective with distantly supervised human rationales?
In Section 4.2, we enforce constraints in the number of instances to annotate with human rationales, and use ER-TEST to compare different strategies to select such instances. However, annotating each token within each instance is also an expensive task, as described in Section 3.4. One way to overcome this issue is generate human rationales with distant supervision. Let L D be a list of lexicons curated by human annotators, specific to a given dataset D. Let l(·) be an indicator function that searches for a given lexicon list in all the tokens of an instance, and returns a binary representation of the same size as the instance with 1s in places with lexicon matches (0 otherwise). Therefore, we can obtain distantly- We can then study the effectiveness of ER methods as detailed in RQ1 (Section 4.1) in this setting.

Implementation Details
For the NLM architecture, we use BigBird-Base (Zaheer et al., 2020), in order to handle input sequences of up to 4096 tokens. For all results, we report the mean over three seeds, as well as the standard deviation. By default, we use a learning rate of 2e−5 and effective batch size of 32. For ER, there are many possible choices of rationale extractor G, but evaluating all of these choices would be prohibitive. Also, evaluating G is orthogonal to ER-TEST's goal of evaluating ER criteria Φ. Thus, as a proof of concept, we use the Input*Grad algorithm (Denil et al., 2014) as G in all experiments, given its popularity and computational efficiency (Bastings and Filippova, 2020;Luo et al., 2021). We leave investigation of other G for future work.

Intrinsic Evaluation of ER
ER in general is sensitive to certain hyperparameters for yielding meaningful training curves and actually attaining alignment between machine and human rationales. Due to a large set of tunable hyperparameters, running all configurations of ER are not feasible. Therefore, we intrinsically evaluate hyperparameter configurations by assessing the loss curves (which model alignment between machine and human rationales) w.r.t different hyperparameters values. We observe that the acceptable band of learning rates for ER is very narrow, and we use 2e−5 in all of our experiments. Furthermore, we also observe that setting λ ER = 1 and γ ER = 100 yields the most drop in the loss curves while training, so we use these hyperparameters for the rest of our experiments. We detail these experiments in Appendix A.1.
5.3 RQ1: Which ER criteria are most effective?

ID Generalization
In Table 1 (In-Distribution), we display the ID task performance results for sentiment analysis (SST), NLI (e-SNLI), and NER (CoNLL-2003). For SST, we find that all of the ER criteria yield about the same task performance as the None baseline, whereas, all ER criteria also perform similarly for NLI (yielding higher performance than None). For NER, we see more variance in task performance among ER criteria, although the variance is still quite small among the best methods (MSE, MAE, Huber). Here, MAE yields the highest task performance, while BCE yields the lowest by far. Overall, using ID task performance, it is difficult to distinguish between ER criteria and underplays its overall benefits. This motivates us to consider other evaluation metrics.

OOD Generalization Unseen Datasets
In Table 1  and compares them to a setting without ER (None). All models (with or without ER) are trained on the ID dataset and evaluated on the ID and OOD datasets without the need of machine or human rationales. Metrics displayed here (higher the better) for sentiment analysis is Accuracy and Macro F1 for NLI and NER. and correspond to cases where the ER criterion in bold are significantly similar and greater than None respectively (p < 0.05).
Meanwhile, despite performing best on SST, Order performs much worse than all other ER criteria here. For NER, MAE still performs best, while MSE and Huber are competitive. Overall, OOD task performance is much better than ID at distinguishing between ER criteria, especially showing ER's improvement over None.

Contrast Set Tests
In Table 2 (Contrast Set Analysis), we observe the drop in performance (denoted by ∆) for sentiment analysis (Movies) and NLI (MNLI) when using a contrast set designed for the given dataset. We observe that for both of the tasks, MAE leads to the least drop in performance. All of the methods apart from Order yield lower drops than None. All of them also have a higher performance on the original and contrast sets. For sentiment analysis, we observe that Order has the highest variance, and for NLI, it has the highest drop in performance. We believe that some of it can be attributed to the soft-ranking that is imposed by Order, which may be indifferent towards minor label-changing edits, that is observed by the contrast sets. Figure 4 demonstrates the failure rates on functional tests (as listed in Section 3.2.3) of our ID models trained on the sentiment analysis and NLI tasks. We also present the overall aggregated (over each individual tests within the categories mentioned) failure rates.

Functional Tests
We observe that apart from the entity-based tests, ER criteria generally have a lower failure rate than None for all of the other tests. For entity-based tests, ER criteria either perform comparably (sentiment analysis) or worse (NLI) than None. Generally, all methods perform well on robustness-based tests, as they have lower failure rates, with order loss having the least. What is important to note is the significant improvement by order loss in vocabulary-based tests than None, even though all of the methods are exposed to the same training set instances. We hypothesize that the biases induced by ER alleviates the shortcuts learnt by None. This is also validated by the overall performance on all of these stress tests, where all of the ER criteria (apart from Huber in sentiment analysis) have lower failure rates than None.  Table 1 on sentiment analysis. Furthermore, all hyperparameters are same as that detailed in Section 5.2. We also choose the best performing ER criterion for sentiment analysis from Table 1 (MAE) and base the following results on the same criterion.

ID Generalization
Consistent with the results we observed in Table  1, Table 3 shows us that there are little to no improvements in ID performance with prioritisation methods, however performance for all of them (except LC) is maintained at par with that of None and 100% sampling. This shows that doing ER on selective samples does not degrade ID performance.

OOD Generalization (Unseen Datasets)
Interestingly, we observe distinctions within various prioritisation methods as we look at OOD evaluations in  original and contrast sets. ∆ is the difference in performance of F between the contrast and original set, and lower the value, better the generalization power of F. A value farther from 0 suggests that F has learnt shortcuts specific a dataset, which are not generalizable to task that the dataset captures as a whole. outperform None. This implies that doing ER on a smaller subset of instances would instantly yield small improvements over None. As we increase the annotation budget, we observe that model performance declines as we select lower confidence samples, but is maintained or even improves over random while selecting instances with greater confidence. It is important to reiterate here, that samples are prioritized based on the confidence yielded by the None model. This implies that models in general require inductive biases on samples they are already confident on, vs. samples they are less confident on to avoid confusion. We plan to study this phenomenon in more detail for other tasks and more constrained annotation budgets as a part of our Future Work (as detailed in Section 6).
5.5 RQ3: Is ER effective with distantly supervised human rationales?

ID Generalization
For the task of hate speech detection, we train F with the Stf dataset. We report all accuracies in Table 4. As it was observed in Section 5.3.1, ER does not lead to a significant improvement in performance for the Stf test set. However, it is important to note that "blacklisting" group identifier lexicons does not lead to a drop in ID performance either. Benefits of "blacklisting" are then observed in OOD generalization.

OOD Generalization
Unseen Datasets We evaluate F on two OOD datasets, HatEval and GHC. Table 4 shows that while the improvements in HatEval are not significant, there are significant accuracy improvements for the GHC test set, which are due to the Order ER criterion.
Fairness Tests In addition to generic performance metrics like accuracy, we also measure group identifier bias (against the groups detailed by group identifier lexicons) by evaluating the False Positive Rate Difference (FPRD) as shown by (Jin et al., 2021). FPRD is computed as z |FPR z − FPR overall |, where FPR z is the false positive rate of all of the test instances mentioning group identifier z, and FPR overall is the false positive rate of all the test instance. Essentially, FPRD evaluates if F is more biased against a given group identifier z, than all of the groups. A lower FPRD value indicates less biased against the listed group identifiers by F.   ria suffer with higher bias than None, we observe that Order criterion consistently leads to the least bias, both in-distribution and out-of-distribution. Furthermore, the reduction in bias is significant when compared to None. Interestingly, Order ER criterion was initially conceived for distantlysupervised rationales (Huang et al., 2021), and the authors of the original paper also demonstrated experiments with rationales generated from lexicons where Order criterion leads to improvements. Our observations are in-line with theirs, and we also show its benefit in reducing bias in F.

A Appendix
A.1 Intrinsic Evaluation: evaluating ER's sensitivity to hyperparameters When using ER to train F, it is important to assess whether ER exhibits expected training behavior, orthogonally to task performance. If ER improves task performance, this kind of analysis can help us better understand ER's effectiveness. Conversely, if ER does not improve task performance, such analysis can help us identify the problem. Motivated by this, ER-TEST's intrinsic evaluation is based on machine-human rationale alignment, captured by the ER loss L ER = Φ(r i ,ṙ i ).
When using ER, we should generally expect the ER loss to decrease as F is trained. In practice, this may not always be the case, even when ER leads to slightly higher task performance (which is likely a mirage caused by lucky random seeds)! That is, by definition, non-decreasing ER loss signals ineffective ER usage, since the machine rationales are not becoming more similar to the human rationales. This can stem from a number of issues: e.g., poor choice of ER criteria Φ, improper ER strength λ ER , improper rationale scaling factor γ ER , noisy human rationaleṙ i , insufficient F capacity. Thus, we measure machine-human rationale alignment as the first step in diagnosing such issues.
Let ER loss curve denote a chart which plots L ER vs. the number of train epochs. For each combination of ER criteria Φ and some training configuration, we plot ER loss curves for the training set. Each component of our intrinsic evaluation varies a different hyperparameter in the training configuration: (A) ER strength λ ER ; (B) rationale scaling factor γ ER ; and (C) learning rate α. In contrast, prior works do not explore the relationship between L ER and these training variables (Huang et al., 2021;Ghaeini et al., 2019).
For intrinsic evaluation, we use ER strength λ ER = 1, rationale scaling factor γ ER = 1, and learning rate α = 2e−5, unless otherwise specified. As a proof of concept, we focus on SST here, but plan to add other datasets in future work.

A.1.1 ER Strength
Fig. 6 displays the ER loss curves for different ER strengths λ ER = [0.5, 1, 10, 100, 300], on SST using MAE. Among the λ ER values, we see that λ ER = 1 yields ER loss curves with the greatest decrease (Table 6), signaling good ER optimization.     ER-trained model, evaluated on either the dev set or test set (yielding two point sets). Fitting each point set with linear regression, we find that there is an inverse relationship between task performance and ER loss. In other words, higher machine-human rationale alignment (i.e., low ER loss) corresponds to higher task performance, which validates the usage of ER to improve generalization. Table 7 displays the slopes and R 2 scores of the lines in Fig. 9. The slope indicates the strength of the relationship between machine-human rationale alignment and task performance (lower is better), while the R 2 score indicates how accurately each line fits its corresponding data points. Among the five ER criteria, across dev and test, we find that MAE has the lowest slopes and highest R 2 scores overall, suggesting that using ER with MAE is most effective.

A.2.3 Change in Target Class Confidence
We consider ER with the MAE criterion, trained/evaluated on SST (via dev ID task performance). Fig. 10 Fig. 9 (ER Loss vs. Task Performance), using slope and R 2 score (Sec. A.2.2). Ideally, Fig. 9's lines would have low slope and high R 2 , indicating that ER helps improve task performance. We see that MAE yields the best ER results.  where FNo-ER's target class confidence is low, there is a higher percentage of instances that are predicted incorrectly/correctly without/with ER. This suggests that instances with low target class confidence are more likely to benefit from ER.
Among instances for which F No-ER 's target class confidence is low, there is a higher percentage of instances that are predicted incorrectly/correctly without/with ER (i.e., incor→cor). This suggests that, for F No-ER , instances with low target class confidence are more likely to benefit from ER (Table  8). Also, based on the T-test, target class confidence scores are significantly higher (p < 0.005) with ER than without.

A.2.4 ER Opportunity Cost
An ER-trained NLM F task, ER and a non-ER-trained NLM F task, No-ER are likely to yield different outputs given the same inputs. Let D + ER ⊆ D and D + No-ER ⊆ D denote the sets of instances predicted correctly by F task, ER and F task, No-ER , respectively. Ideally, we would have D + No-ER ⊂ D + ER . This means there is no opportunity cost in us- ing ER, as ER increases the number of correct instances without turning any previously-correct incorrect. However, this may not necessarily be the case, so we measure ER's opportunity cost as follows. Let n + ER = |D + ER \(D + ER ∩ D + No-ER )| be the number of instances predicted correctly by F task, ER , but not by F task, No-ER . Let n + No-ER = |D + No-ER \(D + No-ER ∩ D + ER )| be the number of instances predicted correctly by F task, No-ER , but not by F task, ER . Then, the opportunity cost of using ER is defined as: In practice, instead of defining o ER for all of D, we only consider test sets D test andD test . Table 9 displays the opportunity cost results for sentiment analysis. Generally, the opportunity cost results mirror the task performance results in Table  1, such that the methods with highest task performance tend to have the lowest opportunity cost. However, using opportunity cost, the variance is very high for OOD datasets, making it difficult to compare methods. In future work, we plan to modify the opportunity cost metrics to better accommodate OOD settings.

A.3 Efficient hyperparameter tuning with ER-TEST
In intrinsic evaluation (Sec. A.1), we used ER loss curves as priors for selecting three key ER hyperparameters (i.e., ER strength λ ER , rationale scaling factor γ ER , learning rate α). In Sec. 5, we assumed  a tuning budget that allows only one value for each of λ ER , γ ER , and α. By not tuning these hyperparameters, we greatly reduced our hyperparameter search space. Since ER has little effect on ID task performance, tuning based on ID task performance is unlikely to have helped anyway. ER works better on OOD data, but it also does not make sense to tune based on OOD task performance (otherwise, it would not be OOD). Though the ER hyperparameters chosen via intrinsic evaluation generally improved OOD task performance, we seek to verify their effectiveness compared to other possible hyperparameter values.
In Table 10 (in the appendix), we report sentiment analysis OOD (Amazon, Yelp, Movies) task performance, while varying each of the three hyperparameters. We include a Mean column, which averages the Amazon/Yelp/Movies columns. Our hyperparameters chosen via ER loss curves are highlighted in blue . For λ ER , 1 (ours) and 100 yield very similar Mean results, while considerably beating the other three values. For γ ER , we see the same trend for 100 (ours) and 10. For α, 2e−5 (ours) vastly outperforms other values in all columns. These results validate the utility of ER-TEST's intrinsic evaluation for low-resource ER hyperparameter tuning.

A.4 Details for Functional Tests
In this section, we provide details for different functional tests listen in Section 3.2.3. We breakdown each subcategory of functional tests and show performances of different ER criteria on those individual tests. For functional tests on the sentiment  analysis task, refer to Table 11. NLI functional tests are listed in Table 12.

A.5 Details for Instance Prioritisation Experiments
In this section, we provide further implementation details for confidence-based instance prioritisation experiments as described in Section 4.2. Given that we have 3-seed runs for the None model in Table 1, we extract the confidence scores based on the given metric (LC or HC), and then average these confidence scores across the 3 seed runs to obtain a single score for every instance. This process is done for training set instances only. This is followed by ranking each instance by the aggregated confidence metric and selecting the top k% of samples from this ranking. For experiments with random sampling based prioritisation, we generate 3 random subsets selected in a uniform manner.
While training in this setting, we ensure that within each batch, certain (one third to be specific) set of instances have available rationales. For these instances, we calculate the ER loss L ER , whereas, for the rest of the instances in the batch, we compute the task loss L task . All prioritisation settings are trained with 3 different model seeds and the aggregated results for ID and OOD datasets are shown in Table 3


s2fieldsofstudy:
Computer Science
Engineering

text:
Addressing Resiliency of In-Memory Floating Point Computation

In-memory computing (IMC) can eliminate the data movement between processor and memory which is a barrier to the energy-efficiency and performance in Von-Neumann computing. Resistive RAM (RRAM) is one of the promising devices for IMC applications (e.g. integer and Floating Point (FP) operations and random logic implementation) due to low power consumption, fast operation, and small footprint in crossbar architecture. In this paper, we propose FAME, a pipelined FP arithmetic (adder/subtractor) using RRAM crossbar based IMC. A novel shift circuitry is proposed to lower the shift overhead during FP operations. Since 96% of the RRAMs used in our architecture are in High Resistance State (HRS), we propose two approaches namely Shift-At-The-Output (SATO) and Force To VDD (FTV) (ground (FTG)) to mitigate Stuck-at-1 (SA1) failures. In both techniques, the fault-free RRAMs are exploited to perform the computation by using an extra clock cycle. Although performance degrades by 50%, SATO can handle 50% of the faults whereas FTV can handle 99% of the faults in the RRAM-based compute array at low power and area overhead. Simulation results show that the proposed single precision FP adder consumes 335 pJ and 322 pJ for NAND-NAND and NOR-NOR based implementations, respectively. The area overheads of SATO and FTV are 28.5% and 9.5%, respectively.

I. INTRODUCTION
In the big data era, conventional CMOS-based Von-Neumann architecture platforms are unable to face real-time data processing requirements [1]. Memory and computing elements are decoupled from each other in Von-Neumann architecture [2] which apply frequent communication between memory and computing cores [3]. The compute energy has been scaled asymmetrically compared to data transport energy with transistor scaling. Data movement in modern computing systems dominates energy-efficiency and performance [4].
In Memory Computing (IMC) is one of the promising compute models to fully or partially eliminate the need to transport data between processors and memory. The main concept of IMC is to infuse compute capability into the memory cells [5]. IMC is achievable by using emerging Non-Volatile Memories (NVM) e.g., RRAM, Spin Transfer Torque (STT) RAM and Phase Change Memory (PCM) [5], [6], [7], [8]. Near memory processing [9] and logic-in-memory, which employ NVMs in the logic space [10], [11] to preserve states between powering sequence have been proposed in the literature. However, they cannot solve the problem of separation between logic and memory.
Due to immature fabrication technology limitations, manufacturing yield is still a serious concern for NVMs such as, RRAM crossbar. Faults in RRAM crossbar arrays are categorized into hard and soft faults [1]. Previous studies have been predominantly focused on soft faults [19] whereas few attempts are made to recover crossbar arrays from hard faults. The soft faults (e.g., read disturb) can be recovered by calibrating the resistance [19] [20]. However, hard faults are recovered through mapping algorithms (i.e., by assigning inputs of faulty RRAMs to the redundant rows or columns) [1], [21], [22].
Stuck-at fault is defined as a situation when the RRAM is permanently stuck at High Resistance State (HRS) or Low Resistance State (LRS). It has been reported [23] that only 63% of Hf O 2 − based RRAM devices for 4Mb crossbar array are fault-free and about 10% of RRAM devices contain stuckat faults. Retention failure which is similar to the resistive switching due to the generation or recovery of oxygen vacancy is another type of hard faults in RRAMs. In the proposed IMC architecture, only 4% of the RRAMs are in LRS and the other 96% are in HRS. Therefore in this paper, we focused on the HRS retention failure and stuck-at-1 (i.e., stuck-at HRS) faults. If the yield of a single RRAM device is 99%, there is only 10 −9 probability for a column of 64*32 array to be fault free. The stuck-at failures and HRS to LRS switching [24] can be fixed by employing few redundant rows/columns when RRAM array is considered working as a memory. However, the whole array is needed for IMC application. Consequently, computations will fail due to errors in the absence of fault tolerance schemes.
We have considered Floating Point (FP) operations to evaluate the proposed resilience techniques. This is motivated by the fact that emerging applications e.g., mission-critical systems like autonomous cars require huge amount of data processing in real-time at low-power (to make timely decisions). The autonomous cars make complex decisions in a tight deadline using algorithms e.g., Kalman filters for data fusion, ray tracing for path planning and, edge detection and deep neural networks for classification. Most of these algorithms require FP vector operations involving transpose, inverse, addition/multiplication. Therefore, the capability to perform these tasks, quickly and accurately can be of utmost importance to enable the safe and energy-efficient autonomous systems. Conventionally, FP architectures are implemented as full custom VLSI or in FPGA. Although fast and power efficient, these custom designs impose cost and complexity. In this paper, we propose FAME (Single Precision Floating Point Arithmetic using In-Memory Computing) implemented on crossbar RRAM. We employ a modified version (Section ??) of Dynamic Computing In Memory (DCIM) [7] based architecture as our baseline compute substrate for FAME. Additionally, two approaches namely, Shift-At-The-Output (SATO) and Force To V DD (GN D) (FTV(G)) are proposed to enable in-memory computing in presence of HRS to LRS retention failures. We focus on this failure mechanism due to two reasons: (i) HRS to LRS switching is more common in RRAM [25]; (ii) majority of the RRAMs (96%) are in HRS for both N AN D-N AN D and N OR-N OR arrays. Carry Select Adder (CSA) based on DCIM implementation is used for the demonstration. We add extra peripheral circuits on each array to implement the proposed techniques.
In particular, we make the following contributions in this paper: 1) Alternative low-overhead realization of DCIM for FP computation; 2) In-memory shift circuit embedded in the peripherals e.g., sense amplifier (SA); 3) Enabling pipeline architecture using the latch embedded in the SA; 4) Propose fault mitigation approaches such as, SATO and FTV/FTG for DCIM architecture; 5) Conduct PV analysis of the RRAM array to check the integrity of SATO and FTV/FTG. Rest of the paper is organized as follows. Section II introduces related work on IMC. Section ?? explains the proposed FAME circuit and architecture. Section III presents the simulation results of FAME and comparison with other IMC logic implementation. Section IV explains proposed approaches to overcome SA1 faults in IMC architecture. Section V presents the proposed fault tolerance approaches and simulation results. Section VI draws the conclusion.

A. Memristor Aided Logic (MAGIC)
MAGIC [26] (shown in Fig. 1) is an IMC architecture in which logic state of the gates are represented by the memristor (RRAM in this paper) resistance where high (low) resistance is considered as logic '1' ('0'). The inputs to a MAGIC gate are the logic states stored in the input memristors and the output is the final state of the output memristor. MAGIC executes operations in two steps: 1) setting the output memristor to a known logic state (e.g., for N OR operation the output

Out Gateway
In-2

In-1
Step # Opertaion Application of Voltage In DCIM, wordlines (WL) serve as the inputs and the bitlines (BL) serve as the outputs of the arrays. Separate preprogrammed AN D and OR arrays are dedicated to implement the desired function. For instance, in order to implement in 0 .in 1 , the bitcells connected to in 0 and in 1 are programmed to LRS while the bitcells connected to in 0 and in 1 are programmed to HRS ( Fig. 2a. All bitcells which are not part of AN D gate inputs are programmed to HRS (e.g., the bitcells connected to input in n and in n ). Fig. 2 shows the implementation of XOR function using DCIM. Initially, Pre signal is activated to pre-charge BLs of the AN D array. Next, inputs (in 0 and in 1 ) are applied by asserting EN AN D . As shown in Fig. 2b, both BL 0 and BL 1 drop below the reference voltage (V Ref −AN D ) when in 0 = in 1 = 1. As a result, SA output which determines the results of in 0 .in 1 and in 0 .in 1 functions are pulled down to '0' at the edge of SE AN D . Next, AN D array SA outputs are provided as inputs to the OR array. Since inputs of the OR array are '0', the BL (BL0 OR ) remains discharged which results in in 0 ⊕ in 1 = 0. If in 0 = 0, in 1 = 1 (in 0 .in 1 = 0 and in 0 .in 1 = 1), BL 0 discharges while BL 1 remains pre-charged. Therefore, BL0 OR starts charging at the edge of EN OR . Finally, the voltage of BL0 OR is compared against V Ref −OR at the edge of SE OR which produces '1' at the output of SA.

C. FP Addition/Subtraction
In IEEE 754 standard, a single precision FP number is represented by 1 Sign bit, 8 Exponent bits, and 23 Fraction bits. A negative (positive) number is represented with a sign bit  [27] equal to '1'('0'). In order to demonstrate negative exponents, IEEE 754 uses a bias of 127 for single precision (e.g., -1 is represented by -1+127=126). The general representation of a FP number is given by: The flowchart for FP addition/subtraction as per IEEE 754 standard is shown in Fig. 3.

III. FAME SIMULATION RESULTS
The simulations are carried out in 65nm PTM [28] technology by employing ASU RRAM model [29] and bi-directional selector diode model [30]. Worst-case Sense Margin (SM), BL-delay, average delay, average power, and energy consumption (Table III) are calculated to evaluate FAME architecture. Key parameters of devices for simulations are listed in Table   TABLE I Table II to mimic process variations. The worst-case SM is obtained under process variation @25 o C for worst case compute array (i.e., fraction addition array). The BL-delay is the time when 100 mV SM is achieved. The proposed FP adder/subtractor implementation with both N AN D-N AN D and N OR-N OR architecture are compared against MAGIC and ASIC design.
The write latency is obtained by performing 1000 points MC simulation. The worst-case write latency for low-to-high and high-to-low switching under process variation is 20ns. FAME achieves 828X, 3.2X and 3.7X improvement in latency, power and energy, respectively compared to MAGIC. The higher energy associated with MAGIC is attributed to the need to write into the RRAMs when an operation is done. Furthermore, compared to the power, energy consumption, and   (Table III). V N AN D0 (NAND array BL voltage when input is '0'), V N AN D1 , V N OR0 , and V N OR1 distributions at worst-case temperature are shown in Fig. 4. In order to achieve the read access pass yield (RAP Y ) [31] [7] we have performed SA offset voltage analysis. The SA offset voltage can be modeled by a Gaussian distribution with σ = 16mV and µ = 8mV . To obtain RAP Y we assume that V Ref is produced by a voltage regulator with negligible variation (5mV ). We assigned V Ref in such a way to maximize RAP Y . Based on the Monte-Carlo simulation, the RAPY of N AN D and N OR operations are found to be 4.6σ and 4.5σ respectively.

IV. RESILIENCE TO STUCK-AT FAULT
In this section, we describe SATO and FTV, two fault mitigation techniques proposed for DCIM architecture. In the following we use, (i) faulty BL to denote each BL with an  undesired stuck-at-1 (SA1) RRAM; (ii) faulty WL (BL) to denote each WL (BL) with an undesired SA1 RRAM.
Computations are performed in two cycles when the proposed fault mitigation techniques are applied (Fig. 5). The computations of fault-free BLs (BL 1 , BL 2 and BL 3 in Fig.  5 (a)) are performed in the first cycle and the computations of the faulty BLs (BL 0 in Fig. 5 (a)) are performed in the second cycle. In FTV, the WLs corresponding to faulty RRAMs (In 2 in Fig. 5 (a)) for N AN D (AN D) array are forced to V DD to mask faulty bits. In a dual Force-to-Ground (FTG) technique, the faulty BLs are forced to 0V for NOR (OR) arrays. FTV/FTG tolerates 99% of stuck-at faults (SAF) while reducing power consumption of the array. In SATO approach, operations of fault-free BLs are executed in the first cycle and then the outputs are shifted in the SAs. Then, the operations of faulty BLs are computed using fault-free BLs (operation of BL 0 is done in BL 1 ). SATO covers 50% of SAFs without affecting power consumption. The high level timings of FTV and SATO are illustrated in Fig. 5

(b) and (c), respectively.
A. Shifting-At-The-Output (SATO) As described before, in this technique the normal operation for fault-free BLs are performed in the first cycle and computation of faulty BLs are performed in the second cycle. SATO does not use faulty BLs for performing an operation and executes all the operations on the fault-free BLs. SATO shifts the data stored in SAs' latch of fault-free BLs to prevent overwriting. When computation of first cycle is completed, the data are shifted in SAs (three shifts are needed if an adder/subtractor is implemented). As shown in Fig. 6, inputs of the WLs should get shifted too, so computation is performed using fault-free BLs. Peripherals of SATO incurs 29.5% area overhead.
1) Non-fixable Faults: SATO cannot handle faults that appear on two consecutive sets of BLs (each three consecutive BLs are a set if an adder/subtractor is implemented). More multiplexers are needed for each WL to handle faults on consecutive sets of BLs. The number of multiplexers per WL increases linearly with the number of consecutive faulty sets of BLs to be handled by SATO. For example, if faults occur on two consecutive sets of BLs (e.g., if BL 3 in Fig. 6 also contains a fault) SATO cannot handle it unless two or more multiplexers are dedicated to each WL. The probability of two faults occurring on two consecutive BL is less than 3% for a 64*32 crossbar array. However, SATO is able to handle less than 50% of the faults if a yield of 99.5% is considered on a crossbar array.
2) Handling multiple faults: SATO's efficiency degrades for increasing number of faults. In this paper, we considered a yield of 99.5% in a 64*32 crossbar array for SATO simulations. This corresponds to 11 randomly distributed faults throughout the array. SATO is able to mitigate around 50% of the faults in the array. Faults have been distributed on the memory cells using rand function provided by C + + programming language. FTV uses a multiplexer for the enable signal of SAs to ensure that the array is capable of working in two cycles. F · CS and F · CS are inputs of the multiplexer, where F is '0' if the BL is fault-free and is '1' if the BL is faulty. CS is the clock sequence initialized to '0' in the first cycle and '1' in the second cycle. Enable signal of SAs connected to fault-free BLs are asserted in the first cycle while the enable signal of SAs connected to faulty BLs is asserted in the second cycle to save power and maintain the correct logic. Furthermore, FTV uses 4 additional transistors compared to DCIM at the WL input to enable the test procedure (explained in Section IV-C) to find faulty RRAMs.
As shown in Fig. 7a, FTV uses two transmission gates to connect input and input to WLs. Also, one PMOS transistor is added to each WL to force the WL to V DD when is needed in the second cycle. In the second cycle, SC signal of faulty WLs gets activated to force faulty WLs to V DD . FTV employs a fault signal (F W ) for each WL to track the faulty WLs and set them to V DD in the second cycle. FTV also defines a fault signal (F B ) for each BL to keep track of faulty BLs. Additional circuitry and peripherals needed to apply FTV to the DCIM increase the area by 9.5%.
1) Forcing-to-Ground (FTG): The basic concept of FTG is similar to FTV but it applies to N OR (OR) arrays. FTG follows simple logic that number of '0's is not important in N OR (OR) operation. Therefore, FTG forces inputs of faulty   RRAMs to the ground. Peripherals and the rest of the FTG's operation are the same as FTV.
2) Non-fixable Faults : Although FTV can fix most of the faults in a crossbar array, it is unable to handle some rare situations. For example, if there are two faulty BLs and the faulty RRAM on one of the BLs is the operand of the other BL. As shown in Fig. 8, BL 0 and BL m are faulty and their operation must be done in the second cycle. In this case, the logic of BL m gets lost if FTV forces input of the faulty RRAM (RRAM1) on BL 0 to V DD since one of its inputs is set to V DD . The N AN D operation for BL m is incorrectly performed between in n and '1' instead of between in n and in m . The probability of occurrence of such a fault for fabrication yields of more than 99% is less than 1%. We randomly distributed the faults for 100 times using C + + language rand function in order to achieve the percentage of faults occurring in an array.
3) Handling multiple faults: As long as faulty RRAMs in the crossbar array are independent of each other, FTV can handle as many as possible faults. For our simulations we inserted 30 faults in a 64*32 crossbar array and FTV was able to solve more than 99% of fault distribution over the array.

C. Finding Faults using FTV Peripherals
It is required to find the faulty RRAMs to set fault signals of the BLs and WLs. Faults can be found by the peripherals that are included in the FTV. However, the BLs must be tested one at a time. To find the faults in a N AN D array, input of each RRAM, which is set to LRS in a BL is forced to V DD and the rest are forced to '0'. The output of the SA indicates whether a BL is faulty ('1') or fault-free ('0'). If the BL is faulty, we need to find out which RRAM is faulty on that BL. A divide-and-conquer approach cannot be used in this architecture since there might be more than one faulty RRAM per BL, so we use brute force algorithm to find faulty RRAM. The input of the RRAM-under-test is set to '0' while inputs of all other RRAMs are set to V DD . If SA output is '0', the RRAM-under-test is deemed faulty and its flag is set to '1'. All faulty RRAMs can be found by repeating this operation for each RRAM in each BL sequentially.

D. Usage and Limitations of SATO/FTV/FTG
SATO/FTV/FTG should be enabled only when a fault has been detected in the test process. Therefore, the fault-free array will only incur area overhead but no performance loss. The faulty array will be salvaged at the cost of performance overhead. Note that SATO/FTV/FTG are only applicable to DCIM-based IMC. They cannot be applied to MAGIC or RRAM-based static IMC in the current form.

V. SIMULATION RESULTS
To evaluate SATO and FTV, we compute performance metrics that include worst-case SM, BL-delay, average delay, average power, and energy consumption of a 64*32 DCIM RRAM crossbar array (VI). Based on the simulation results, FTV is more efficient than SATO.

A. SATO Simulation Results
Applying SATO to DCIM increases power and energy consumption by 12% and 127%, respectively (the worst case) and also performance is reduced by more than 50%. However, SATO is able to handle ∼50% of the SA1 faults. SAs are very costly and occupy large area, which using SAs to shift data, increases power consumption and leads to higher energy consumption. Power and Energy consumption of SATO with different number of SA1 faults is reported in Table VII.

B. FTV/FTG Simulation Results
SM is the most important parameter when FTV is applied to DCIM. Increased number of LRS RRAMs connected to V DD (ground) on a BL worsens the SM when '0' ('1') is the output. Considering N AN D − 2, the worst case '0' occurs when one of the operands is '0' and the other operand is '1'. In this case, there is a voltage division is between one LRS RRAM connected to '0' and one LRS RRAM which is connected to V DD . When there is a faulty LRS RRAM on the BL and it is forced to V DD , the worst case '0' is when two LRS RRAMs are connected to V DD and one LRS RRAM is connected to '0' which lead to increased output '0' voltage on the BL. Increased BL voltage for the worst case '0' degrades the SM as shown in Fig. 11 (a).
The degradation in worst case SM happens when the bitcell is made of only a RRAM (i.e., no selector diode). However, DCIM employs a bidirectional diode in series with the RRAM. This series-connected bidirectional diode is included to reduce power consumption by dropping 0.5V across the 2 terminals. When the voltage difference between a BL and a WL is less then the selector diode threshold voltage there is no current between the BL and the WL. So, increased number of LRS inputs connected to V DD (ground) does not affect the SM of DCIM ( Fig. 11 (b)). The current of HRS RRAMs increases with temperature which results in higher sneak path currents. In an AN D (OR) array, the higher sneak path currents pull up (down) BL voltage to degrade the SM. However, when number of faults increases, sneak paths currents become negligible compared LRS RRAMs which are connected to V DD (ground). Simulation results (Fig. 11 (a)) show that the SMs in different temperatures become equal when the number of faults is more than 20.
Compared to the fault-free situation, FTV reduces power and energy consumption by >54% and >7% respectively (since, SAs consume a lot of power and in the case of FTV, SAs connected to faulty BLs are deactivate in the first cycle and SAs connected to fault-free BLs are deactivated in the second cycle). This is due to inactive BLs and SAs and a longer time of operation. However, the performance reduces by 50% due to two cycle operations. Average power and energy for the four consecutive AN D operations in the 64*64 array are reported in Table VIII.

C. Process Variation Simulations
The most important parameter to consider in a crossbar array under process variation is SM. We ran 1000-point MC simulations at −10°C, and 90°C on DCIM by considering the bitcell consisting of only a RRAM and RRAM and a selector diode with different number of SA1 faults. Simulation results for RRAM bitcell and RRAM and selector diode bitcell are shown in Fig. 9 and 10, respectively.
As shown in Fig. 9, variations do not affect SM significantly due to the presence of selector diode which stabilizes BL voltage. However, as demonstrated in Fig. 10, variations affect the SM when only RRAM is used in the crossbar. This is due to large changes in the RRAM resistance for a small change in RRAM gap when 1.2V is applied across it. Worst case SM with the number of failures for both w/ and w/o selector diode is reported in Table. IX.

VI. CONCLUSIONS
We proposed FAME for in-memory FP arithmetic computation. FAME implements single precision FP adder/subtractor using RRAM crossbar and evaluated two flavors with N AN D − N AN D and N OR − N OR compute arrays. We also proposed a novel SA based shift circuit for frequent shifting needed in FP operation. Compared to MAGIC-based implementation, FAME achieves 828X and 3.7X latency and energy improvement over MAGIC and compared to processing units (e.g. CPU, FPGA, GPU) it also reduces energy consumption and delay by 93% and 70%, respectively. FAME achieves lower power and energy consumption compared to MAGIC and processing units at low area overhead to the memory arrays. FAME uses 3KB memory to implement single precision FP operations (V). Furthermore, two approaches to mitigate HRS to LRS retention and stuck-at-1 failures in RRAM-based compute memories are proposed along with a test approach to identify faulty RRAMs. Forcing-to-V DD (FTV) can mitigate 99% of the faults while reducing the power consumption by >50% and energy consumption by >7%.
Shifting-at-the-Output (SATO) technique increases power consumption slightly but increases energy consumption by >50%.


UMUTextStats: A linguistic feature extraction tool for Spanish

Feature Engineering consists in the application of domain knowledge to select and transform relevant features to build efficient machine learning models. In the Natural Language Processing field, the state of the art concerning automatic document classification tasks relies on word and sentence embeddings built upon deep learning models based on transformers that have outperformed the competition in several tasks. However, the models built from these embeddings are usually difficult to interpret. On the contrary, linguistic features are easy to understand, they result in simpler models, and they usually achieve encouraging results. Moreover, both linguistic features and embeddings can be combined with different strategies which result in more reliable machine-learning models. The de facto tool for extracting linguistic features in Spanish is LIWC. However, this software does not consider specific linguistic phenomena of Spanish such as grammatical gender and lacks certain verb tenses. In order to solve these drawbacks, we have developed UMUTextStats, a linguistic extraction tool designed from scratch for Spanish. Furthermore, this tool has been validated to conduct different experiments in areas such as infodemiology, hate-speech detection, author profiling, authorship verification, humour or irony detection, among others. The results indicate that the combination of linguistic features and embeddings based on transformers are beneficial in automatic document classification.

Introduction
Online communication has opened a new way of exploring how people communicate and interact with each other. Recent trends in Natural Language Processing, Artificial Intelligence and Information Retrieval have eased the analysis of large amounts of data that can be used for conducting automatic document classification tasks, such as Sentiment Analysis (SA), with applications in marketing in order to gain faster insights about brands, products and services. Other popular applications of automatic document classification are authorship attribution (AA), which can help uncover anonymous threats, plagiarism detection, and cyberbullying and hate-speech (HS) detection, building safer social environments. The state of the art concerning automatic document classification is focused on pre-trained embeddings that have been learned with large corpora. These models learn to represent words as vectors of fixed size and arrange them within the latent space, clustering words that have a close relationship. These models are learned based on unsupervised tasks, such as Masked Language Modelling (MLM). Moreover, transformers and attention mechanisms have enabled the learning of contextual embeddings, in which words are aware of surrounding ones, solving problems partially related to polysemy and word disambiguation. Word embeddings can be used as input of other Machine Learning models to perform downstream tasks of automatic document classification. These embeddings can capture rich semantic and lexical variety; however, there are linguistic phenomena that can help to classify documents that can go unnoticed with word embeddings. For example, using uppercase words or vocabulary range may indicate relevant findings that may not be detected easily with embeddings.
Linguistic Features (LF) are features that characterise uses of language in a text. LF can refer to grammatical aspects of a text, analysing how words and sentences are related. Besides, LF can show prosodic features related to stress and intonation or searching for specific lexicons that can indicate different demographic or psychographic features of the authors. Although the performance of the LF is, generally, poorer than state-ofthe-art word embeddings, we argue that the LF are easier to interpret and result in simpler models and showing competitive results. Furthermore, LF and word embeddings can be easily combined using different strategies such as Knowledge Integration (KI) or Ensemble Learning (EL), improving the results achieved separately.
We present UMUTextStats, a linguistic feature extraction tool designed for Spanish. This tool can extract a vector made up of the percentages of words and expressions that fit into a series of linguistic categories and features. This tool is based on Linguistic Inquiry and Word Count (LIWC) (Tausczik and Pennebaker, 2010) in an attempt to resolve the deficiencies found during the translation of LIWC to Spanish (Ramírez-Esparza et al., 2007). In addition, we show an extensive evaluation of these features in different tasks and domains such as infodemiology, hate-speech identification, and authorship attribution, among others. The rest of the manuscript is organised as follows. First, Section 2 describes LIWC, as it is the base of our proposal, and other linguistic feature extraction tools and their applications. Following this, Section 3 describes the architecture of UMUTextStats and how it classifies the LF into categories and dimensions. The reader can find a detailed list of the experimentation conducted with UMUTextStats in Section 4 and, finally, the conclusions and promising further research lines in Section 5.

State of the art
LIWC is the de facto standard for extracting LF (Tausczik and Pennebaker, 2010). It is widely used in text analysis and automatic document classification tasks. Its behaviour is quite straightforward, that is, it reads a collection of texts and counts the percentage of words according to certain psychologically relevant categories that are arranged hierarchically. The way in which LIWC works is based on lexicons. That is to say, each dimension is composed by a dictionary of terms. The dictionaries of the previous versions of LIWC were built manually and validated by several human annotators. The last known version of LIWC was released in 2015. It contains almost 6,400 words in its master dictionary (1), (2) word stems, and (3) emoticons. It is worth mentioning that one word can belong to multiple categories. Saying more, the current version of LIWC extends these dictionaries automatically using distant methods with large corpora. For this, the authors of LIWC test if each of the words in the dictionaries are related to the rest of the words within the same dictionary in a statistically significant way. This process allowed to evaluate if each word in a particular category is indeed related to the rest of the words in the same dictionary. Besides, this step helps identify other words that passed unnoticed in the manual compilation process. LIWC distinguishes mainly between two types of words: content words and function words. Content words are nouns, verbs, adjectives, and adverbs. Content words convey the content of the communication. Style or function words are pronouns, prepositions, articles, and conjunctions. Besides, some forms of auxiliary verbs can be considered as style words. Most of the words in a language are content words; however, style words make up over half of the words in a text. Under a psychological point of view, style words reflect how people communicate, whereas content words convey the intention to communicate. LIWC has a Spanish version validated in (Ramírez-Esparza et al., 2007). It is worth mentioning that, ac-cording to the LIWC manual 1 , the dictionaries of the Spanish version of LIWC are based on the versions of LIWC 2001 and 2007, but not on the 2015 version. The Spanish version of LIWC has been used in several domains, such as opinion mining López-López et al., 2014;García-Díaz et al., 2018), depression assessment (Ramirez-Esparza et al., 2008), deceit detection (Almela et al., 2013), and satire identification (del Pilar Salas-Zárate et al., 2017). However, during the translation of LIWC, the present authors identified some drawbacks concerning some linguistic issues between the English and the Spanish version. For instance, the fact that some grammatical phenomena of Spanish were not considered, which results in the loss of grammatical gender identification or the lack of many verb conjugations.

System architecture
UMUTextStats is a linguistic feature extraction system designed for Spanish. Like LIWC, this system can extract a vector made up of the percentages of words and expressions that fit into a series of linguistic features. However, an attempt is being made to resolve deficiencies found in the Spanish translation of LIWC (Ramírez-Esparza et al., 2007). Apart from these issues, there are other inconveniences that are shared by the Spanish and the English version of LIWC: (1) its arbitrary design of the linguistic categories and features, in which the words that belong to certain categories were selected by a limited number of human annotators; and (2) the fact that LIWC is based principally on simple term-count, so it does not consider the context of a word (that is, the surrounding words that can alter the meaning of a single word). To solve the aforementioned drawbacks, for the design of the UMUTextStats tool we have created a tree-based structure for defining and arranging the LF within categories and dimensions. Besides, we have included several mechanisms to extend UMUTextStats with new dimensions based on custom dictionaries, regular expressions, and a wide variety of performance errors or specific argot used in social networks. We have also developed a new system for extracting all Spanish verb tenses, including compound verbs and periphrases. Figure 1 depicts the system architecture of UMU-TextStats. Below, we describe each module in detail.

Source Resolver
This module is responsible for obtaining the source documents to extract the LF. The current version of UMUTextStats accepts the insertion of text directly using the console, plain text, zip files with several single files, and datasets extracted using the UMUCorpus-Classifier tool (García-Díaz et al., 2020a).

Text Cleaning
The source documents are encapsulated in objects of type TextAsset. These objects store the text of the document but also caches cleaned and normalised versions of the texts. We find an optional step at this point, which is that the source files can already have a cleaned version. This is useful, for instance, when a custom cleaned version of the text is needed. If a cleaned version of the text is not provided, UMU-TextStats performs a cleaning process that involves: (1) removing blank lines, (2) striping HTML tags, (3) removing URLs, mentions and emojis, (4) removing letter elongations, and (5) converting the text to their lowercase form. Optionally, it is possible to use the PSPELL library to fix misspellings. For this purpose, UMUTextStats analyses each word in isolation and replaces the misspellings with the best suggestion only if the text similarity between the two words is higher than a certain threshold.

Configuration
The Configuration module is the core of UMU-TextStats. It is an XML file that defines the LF and categories in a tree-based structure. It is worth mentioning that the design of UMUTextStats considers maintainability and extensibility as first-citizen requirements. UMUTextStats comes with a series of predefined classes that include the usage of lexicons or regular expressions as typical use cases. Below, we describe the available classes: • Dictionary-based features. This class allows defining new dimensions based on keyword lists. The keywords can be regular expressions. It is possible, however, to configure this dimension disabling regular expressions to speed up the process. Besides, dictionary-based features have other options. For example, it is possible to define counterexamples. The benefit of using counterex-amples is that it is easier to define few general regular expressions, and then, to list the exceptions.
• Verb-based features. This class is similar to the dictionary-based features class, as verbs are stored in plain text files. However, the large number of verbs make the usage of dictionary-based dimensions impractical. Verb-based features are optimised to identify verbs in O(1). In addition, it is possible to use a custom word separator to consider auxiliary verbs as part of a matching.
• Sentence per dictionary-based features. This class calculates the number of sentences that match a regular expression. This class is useful, for example, to obtain the number of sentences that use passive voice of verbs.
• Features based on enclitic personal pronouns. This class captures personal pronouns with clitics. In Spanish, the pronouns la, lo, le, los, las, and les are clitic. They indicate direct or indirect thirdperson pronominal object.
• Grammatical-gender-based features. This class captures Spanish grammatical gender. It extends the dictionary-based features class but relies on a list of basic rules for obtaining Spanish grammatical gender combined with a list of counterexamples. In addition, this class considers only certain words based on their Part-of-Speech (PoS) category. This way, it is easier to discard rare and made-up words.
• STTR-based features. This class allows to calculate the Standardised Type/Token Ratio (STTR), which is the ratio between the total unique words between the total of the words of a text. For long pieces of text, it is possible to calculate this ratio in chunks of N words. It is possible to use the standard deviation of this metric as a linguistic feature.
• Features based on misspellings. This class counts the number of misspellings. This class relies on the PSPELL library to capture misspellings, next it checks if the first suggestion of PSPELL is the same word.
• Features based on wrong accentuation usage. This class is like the Error-misspelling-based features class, but it is focused on counting misspellings based on wrong usage of the written accents. For this purpose, this class also relies on the PSPELL library to capture misspellings, next it checks if the first suggestion of PSPELL is the same word.
• PoS-Tagging-based features. This class counts the number of words that match a specific PoS category. PoS categories are calculated using the Stanza Library (Qi et al., 2020).
• NER-based features. This class counts the number of words that match certain NER (Named Entity Recognition) category. Like the PoS-Tagging-based features class, UMUTextStats relies on Stanza (Qi et al., 2020). However, its Spanish model only considers four categories: Person, Location, Organisation, and Miscellaneous.
• Features based on two or more equal words. This class detects two or more similar words together. Although this does not have to be an error, it can be an indication of a lack of attention when reviewing a text.
• Features based on incorrect capitalisation. This class accounts for the number of sentences that starts in lowercase.
• Pattern-based features. This class includes all the occurrences matching certain regular expressions. For example, using the following regular expression (.) 1{3,}<, it is possible to capture expressive lengthening, that is, emphasising a verbalised word.
• Typography-based features. This class allows detecting the number of words written in lower or uppercase. For example, detecting the number of words that are completely written using capital letters could indicate loud volume of the voice.
• Composite-based features. This class allows obtaining certain LF using the Composite Pattern. The aim of this pattern is to define a new dimension based on averaging, adding, subtracting, calculating the maximum or the minimum.
• Features based on word length. This class includes the number of words that match or exceed a specific threshold. For this purpose, it is possible to configure word length and equality.
• Features based on word average length. This class calculates the average length of all the words in the input.
• Features based on words per sentence. This class calculates the number of words per sentence.
• Features based on unique words. This class calculates the number of unique words.
• Features based on syllables per word. This class calculates the number of syllables per word.
• Features based on character count. This class counts the number of a list of specific characters. This class is useful, for example, to capture characters that can be represented with different symbols, such as quotes, currencies, prime symbols, or brackets, among others.
• Features based on sentences starting with the same word. This class counts the number of sentences that starts with the same word. This is a custom class to capture certain stylistic errors.
• Features based on sentences starting with numbers. This class counts the number of sentences that start with a number, which is considered as a poor writing style.
• Features based on Twitter's Reply to. This class is specific for the Social Network category. It determines if a certain text (usually, a Tweet from Twitter) is a response to a specific user based on a list of names.
For example, in Listing 1, we show how to create a LF that captures how many exclamatory sentences are in a document. As can be observed, this dimension relies on the uses the Pattern-based features class, so it is easy to define it using a regular expression that matches sentences that end with two or more exclamation marks. Besides, some classes have common options. For example, dictionary and pattern-based features allow for defining the separator. Usually, the texts are divided into words, but it is also possible to separate texts into sentences or to apply the regular expression to the whole text. Sentence splitting is useful, for example, to count exclamatory sentences. Another option is to indicate if the dimension must obtain the raw count or the percentage. It is worth noting that the percentage is related to the separator employed. For example, in the aforementioned example (Listing 1), the feature divides the texts into sentences, so the results are reported as the percentage over the number of sentences. An advantage of UMUTextStats in comparison to other applications is that it allows operating simultaneously with different versions of the same text. Therefore, some dimensions can operate on a filtered version that makes it easier to look up terms in the dictionary, while the original version can be used to measure characteristics such as the percentage of words in capital letters.
• Phonetics (PHO). It is the part of linguistics that analyses how humans produce and perceive sounds. The current version of UMU-TextStats, which is focused on writing, includes only one suprasegmental feature concerning expressive lengthening, a linguistic device that consists in repeating some of the letters of a word for emphasis (Fersini et al., 2016).
• Morphosyntax (MOR). Also known as grammar, it is the part of linguistics focused on morphology, which studies words and how they are composed, and syntax, which studies phrases and sentences and how they are related. Spanish is a highly inflected language. Inflections can denote multiple syntax and semantic meanings that can be used to track stylometric features in authorship attribution tasks. UMUTextStats classifies morphosyntactic features into: (1) PoS-based features, which include adverbs, adjectives, determiners, and pronouns, to name but a few; and (2) sub-word level, which includes features that capture subcomponents of words, such as stems and affixes. This includes features that capture grammatical gender and number of words.
• Correction and style (CAS). This linguistic category covers linguistic and stylistic errors. Linguistic errors deviate from the valid rules of language. They include, for instance, misspellings or the wrong use of accentuation. Regarding stylistic errors, even though it is possible to understand a text containing them and the meaning conveyed by the author, the text may not sound natural to the recipient.
UMUTextStats divides correction and style features into three major groups: orthographic, stylistic, and performance errors. Orthographic errors capture wrong uses in accentuation and misspellings. Besides, other writing mistakes, such as starting sentences in lowercase, are also identified as orthographic errors. Stylistic errors capture bad writing habits, such as starting sentences with cardinal numbers or repeating several sentences with the same word. Finally, performance errors detect if an author makes a mistake despite knowing well the rules of the language they are using.
• Semantics (SEM). Semantics is the branch of linguistics that studies meaning out of context. Semantics can consider meaning at different units, such as words or sentences.
The current version of UMUTextStats capture four linguistic categories, including (1) onomatopoeia, which is the formation of a word from a sound associated with what is named; (2) euphemism, which is a mild expression replacing one considered as too rude; (3) dysphemism, which is a derogatory expression used instead of a pleasant one; and (4) synecdoche, which is a figure of speech in which a part is used to represent the whole.
• Pragmatics (PRA). It is the branch of linguistics that deals with how language is used and its context. UMUTextStats captures the use of figurative language, including hyperboles, several idiomatic expressions, verbal irony, understatements, metaphors, and similes. Besides, it captures rhetorical questions. Pragmatics also includes several discourse markers, used for structuring the conversation regarding connectors, reformers, argumentative clauses, and conversational bookmarks. Finally, it also includes several typical courtesy forms for greetings or con-dolences.
• Stylometry (STY). It is the automated study of linguistic style, mainly in written language. This category contains features concerning the length of the text, lexical diversity applying the standard type-token ratio (STTR) (Chipere et al., 2004). This metrics has been applied to calculate the lexical richness of texts as, for instance, in the clinical domain (López Hernández and Almela, 2021). Besides, we use different statistics to measure the number of words and syllables, number of sentences, number of words in uppercase, readability formulas or punctuation symbols.
• Lexis (LEX). It refers to the total set of all possible words in a language, or a particular subset of words for a particular domain. It is worth mentioning that lexis provides a functional perspective, so an element of the lexis can be composed of multiple words, such as New York City.
The current version of UMUTextStats includes lexicons of several general domains that capture the intention and the topic of the message. It includes lexicons of terms such as animals, weapons, jobs, crime, money, health, and ingesting, as well as abstract concepts such as achievement, risk, or cognitive processes.
• Psycholinguistic processes (PLP). Psycholinguistics is the branch of linguistics concerning the usage of language from a cognitive point of view. It includes psychological processes involved in language comprehension, production, and first and second language acquisition. UMU-TextStats captures LF based on lexicons, regarding emotions and positive, neutral, and negative attitudes.
• Register (REG). The register helps to define the way in which the speakers and writers use the language under different circumstances. It is related to the selection of the words and other prosodic features related to tone, pitch and body language. UMUTextStats identifies features related to offensive language, informal speech, and the usage of learned words.
• Social media (SOC). Although this category is not exclusive to linguistics, we added a set of features that capture the degree in which users of Twitter make use of specific terminology related to that network. They include the usage of hashtags, mentions, and hyperlinks. This category was added with the purpose of testing the extensibility of UMUTextStats and because we consider that it captures valuable features in certain domains such as cyberbullying or authorship attribution.

Output Resolver
This module is responsible for generating the output of the system. The current version of UMUTextStats has several outputs already predefined, including CSV, ARFF files (for the WEKA platform), HTML (for online visualisation), JSON, and Table format for outputting the results directly to the console. UMUTextStats is ready to work from a terminal. However, it is also possible to use it from a web browser thanks to a graphic user interface.

Experimentation
UMUTextStats has been validated on different domains within the document classification task.
In Table 1 we include a summary of the results achieved in different shared tasks in which we evaluate the LF combined with embeddings. Concerning SA (Eysenbach, 2002), the LF are applied to build an Ontology-guided Aspect-based Sentiment Analysis system from a dataset related to infectious diseases in Latin America (García-Díaz et al., 2020c). For this purpose, we build a custom ontology of the infectious disease domain to represent the aspects of the system. Next, we train a machine-learning classifier using ten-fold cross-validation and evaluating the LF and pretrained word embeddings. The neural networks evaluated include recurrent and convolutional neural networks C(NN). We observed that the LF in isolation achieved the best overall accuracy (55.3%). We also observed that all neural networks architectures except CNN achieved better accuracy when combined with the LF. Finally, we link the sentiment of each document with the aspects of the ontology by using a custom TF-IDF formula , in which the sentiment of a document influences the concepts that appears both explicitly and implicitly in the text. In addition, we calculate the correlation between the LF and the sentiments. The results of this analysis is depicted in Figure 2 (top). We found a negative correlation between numerals and sentiments, and a positive correlation of colloquialism with positive documents. Concerning HS, the LF are applied to conduct a misogyny detection system in (García-Díaz et al., 2021a), in which we also observed that LF also outperform features based on word and non-contextual sentence embeddings. We build several machine-learning classifier that evaluates all feature sets in isolation or combined. We obtained our best result with a model based on Support Vector Machine (SVM), achieving an accuracy of 85.175%. This result was achieved with the combination of the LF and sentence embeddings from fastText. Besides, we obtained the Information Gain for the 20 best LF. These features are depicted in Figure 2 (middle). We found positive correlations with the misogynous label with the usage of offensive language, grammatical gender, and several features concerning correction and style.  (García-Díaz et al., 2021a) Mexican, binary 90.52 AMI 2018 (Fersini et al., 2018) Spanish, binary 84.72 HaterNET (Pereira-Kohatsu et al., 2019) Spanish, binary 84.08 HatEval 2019 (Basile et al., 2019) Spanish, binary 77.27 EXISTS 2021 (Rodríguez-Sánchez et al., 2021) Spanish and English, binary 75.14 EXISTS 2021 (Rodríguez-Sánchez et al., 2021) Spanish and English, multi-class 53.62 AA PoliCorpus 2020 (García-Díaz et al., 2022a) Spanish, binary, gender 72.02 PoliCorpus 2020 (García-Díaz et al., 2022a) Spanish, multi-class, age range 46.69 PoliCorpus 2020 (García-Díaz et al., 2022a) Spanish, binary, ideology 98.04 PoliCorpus 2020 (García-Díaz et al., 2022a) Spanish, multi-class, ideology 91.05 AI-SOCO 2020 (Fadel et al., 2020) Multi-language, multi-class 91.16 FL SatiCorpus 2021 (García-Díaz and Valencia-García, 2022) Spanish, binary 97.32 Satire (Barbieri et al., 2015) Spanish   English, binary 91.60 HaHackathon  English, multiclass, controversial 46.50 HaHa 2021  Spanish, binary 85.44 HaHa 2021  Spanish, multiclass, mechanism 20.87 HaHa 2021  Spanish  (1) an author profiling task regarding psychographic and demographic traits, and (2) an authorship attribution task with tweets from Spanish politicians compiled in 2020. The results in this task were promising. In fact, LF in isolation outperformed BERT-based models in gender prediction. For the rest of the traits, we observed that adding LF to embedding-based features outperformed the results achieved separately. Besides, concerning interpretability of the results, we found that features related to lexis and morphosyntactic were more effective for conducting author profiling tasks whereas the stylometric features were more reliable in authorship attribution. The features of the multi-class political ideology are depicted in Figure 2 (bottom). Besides, we evaluated the LF in other shared tasks proposed in IberEval 2021: one regarding Humour identification (Garcıa-Dıaz and Valencia-Garcıa, 2021a), in which we achieved the 1st position in the subtask of Funniness Score Prediction and the 3rd position for the subtask of target classification. The other task in which we evaluated the LF was related to emotion analysis , in which we achieved the 6th position. It is worth noting that the current version of the LF can also be applied to other languages as some of the linguistic categories, like stylometric features, are language independent. We participated in the AISOCO'2020 shared task (Fadel et al., 2020), focused on authorship identification of source-code. In this challenge, we proposed a system that mixes character n-grams with stylometric features that capture certain author traits. We achieved the 6th position (accuracy of 91.16%), outperforming baselines based on RoBERTa. Other languages in which the LF have been tested are English, Tamil, and Arabic. In English we have participated in HaHackathon 2021 (García-Díaz and Valencia-García, 2021b), concerning humor detection, achieving an F1-score of 91.60% concerning humour identification, and an F1-score of 57.22% in humour controversy detection. In Tamil we have participated in several shared tasks concerning equality, diversity, and inclusion. These tasks were focused on detecting depression signs, abusive comment detection, homophobic and transphobic comments, and emotion detection. In Arabic we have participated in a SemEval shared task concerning sarcasm detection.

Conclusions
In this paper we have described the development and the evaluation of the UMUTextStats tool, a tool for extracting linguistic features designed for the Spanish language. The evaluation is performed in several document classification tasks: Sentiment Analysis, Author Analysis, Satire identification, and Hate speech. There is a demo of the tool available 2 . All things considered, UMUTextStats still has some limitations. The main one is that dictionary-based dimensions consider words in isolation; so it is weak for polysemy and figurative language, especially in short texts such as the ones compiled from micro-blogging social networks. In order to solve this drawback, we are evaluating the reliability to obtain a small subset of LF per token, and train the models using Recurrent Neural Networks. In addition, we are adapting this tool to English and different Spanish dialects. Besides, we are extending and validating the hierarchical structure of the linguistic categories and features.