English Natural Language Understanding (NLU) systems have achieved great performances and even outperformed humans on benchmarks like GLUE and SuperGLUE. However, these benchmarks contain only textbook Standard American English (SAE). Other dialects have been largely overlooked in the NLP community. This leads to biased and inequitable NLU systems that serve only a sub-population of speakers. To understand disparities in current models and to facilitate more dialect-competent NLU systems, we introduce the VernAcular Language Understanding Evaluation (VALUE) benchmark, a challenging variant of GLUE that we created with a set of lexical and morphosyntactic transformation rules. In this initial release (V.1), we construct rules for 11 features of African American Vernacular English (AAVE), and we recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. Experiments show that these new dialectal features can lead to a drop in model performance. Most of today's research in NLP mainly focuses on 10 to 20 high-resource languages with a special focus on English, though there are thousands of languages and dialects with billions of speakers in the world. NLU systems that are trained on polished or textbook Standard American English (SAE) are not as robust to linguistic variation (Belinkov and Bisk, 2018; Ebrahimi et al., 2018). While some recent works have challenged leading systems with adversarial examples like typos (Jones et al., 2020), syntactic rearrangements (Iyyer et al., 2018), and sentence/word substitutions (Alzantot et al., 2018; Jia and Liang, 2017; Ribeiro et al., 2018), fewer have considered the effects of dialectal differences on performance. When language technologies are not built to handle dialectal differences, the benefits of these technologies may not be equitably distributed among different demographic groups (Hovy and Spruit, 2016). Specifically, models tested on African American Vernacular English (AAVE) have been found to struggle with language identification (Jurgens et al., 2017), sentiment analysis (Kiritchenko and Mohammad, 2018), POS tagging (Jrgensen et al., 2016) and dependency parsing (Blodgett et al., 2018), and led to severe racial disparities in the resulting language technologies such as the automated speech recognition used by virtual assistants (Koenecke et al., 2020) the hate speech detection used by online media platforms (Rios, 2020; Halevy et al., 2021). However, no prior work has systematically investigated these dialect-specific shortcomings across a broad set of NLU tasks, and the effectiveness of low-resource NLP methods for dialectal Natural Language Understanding (NLU) remains largely unexplored. The first barrier to progress is that a standard benchmark for dialectal NLU has not yet been constructed. The second is that no systematic error analyses have yet revealed causal insights about the specific challenges that models face with domain adaptation to different language varieties. To both understand dialect disparity and facilitate ongoing work on dialect-competent NLU, we introduce a new dialect-specific challenge dataset  the V ern A cular L anguage U nderstanding E valuation benchmark ( VALUE ). We specifically focus on African American Vernagular English (AAVE), a dialect spoken by nearly 33 million people, and approximately 80% of African Americans in the United States (Lippi-Green, 1997). To facilitate direct comparison with prior work, we build VALUE by directly transforming GLUE (Wang et al., 2019) into synthetic AAVE. Our AAVE transformation pipeline comes with two key advantages: it is flexible enough to facilitate an interpretable perturbation error analysis, and 3701 the transformation rules are meaning-preserving, which ensures the validity of the transformed NLU tasks. Our pipeline includes a set of linguistically-attested rules for syntax (sentence structure; e.g. negation rules), morphology (word structure; e.g., suffixes), orthography (writing and spelling conven-tions), and the lexicon (the list of available words and phrases). Because our system is rule-based, we can isolate and systematically test which features most significantly challenge models. While it is also possible to generate pseudo-dialects via end-to-end style transfer (Krishna et al., 2020), these systems often fail to disentangle style from content, and thus also fail to preserve meaning (Lam-ple et al., 2019). We confirm these shortcomings in this work, and affirm the validity of our own meaning-preserving transformation rules via the acceptability judgments of fluent AAVE speakers in a participatory design manner. To sum up, our work contributes the following: 1. Dialect Transformations: A set of 11 new linguistic rules for reliably transforming Standard American English (SAE) into African American Vernacular English (AAVE). 2. VALUE : An AAVE benchmark dataset with seven NLU tasks. 3. Synthetic + Gold Standard Data: Robust validation of synthetic transformations as well as gold standard dialectal data from native AAVE speakers via an iterative participatory design process. 4. Benchmark Evaluation: Experiments with RoBERTA baselines plus fine-tuning methods to improve model robustness on dialectal variants. 5. Dialect-Specific Analysis: Perturbation analysis that reveals the task-specific challenges of AAVE-specific grammatical features. Computational Sociolinguistics of Dialect Prior work on developing NLU models has often used dominant English varieties, Standard American English (SAE), owing to the availability of text datasets for training and testing (Blodgett et al., 2016). Models can marginalize certain groups when trained on datasets that lack linguistic diversity or contain biases against minority language speakers (Blodgett and O'Connor, 2017). Despite these shortcomings, there still has been relatively little attention paid to dialects in the language technologies research communities. Prior studies have mainly focused on distinguishing between English language varieties (Demszky et al., 2021a; Zampieri et al., 2014). Failure to account for dialects like AAVE can lead to performance degradation of the NLU tools such as Automatic Speech Recognition (ASR) (Dorn, 2019), Language Identification (LID) and dependency parsing tools (Blodgett et al., 2016). Hwang et al. (2020a) also demonstrated the inadequacy of WordNet and ConceptNet in reflecting AAVE and other varieties. Thus there have been several works highlighting the need for AAVE-inclusivity in NLU (Groenwold et al., 2020). Despite its large community of speakers, AAVE is under-represented in current technologies. Model Robustness and Challenge Datasets Language technologies are not inherently robust to linguistic variation. The performance of neural models is expected to degrade due to sparsity in the presence of non-canonical text (Zalmout et al., 2018; Belinkov and Bisk, 2018; Ebrahimi et al., 2018), as shown empirically for random character, word, and sentence-level permutations (Jones et al., 2020; Alzantot et al., 2018; Jia and Liang, 2017; Ribeiro et al., 2018; Iyyer et al., 2018). This has motivated growing interest in challenging datasets based on adversarial perturbations (Nie et al., 2020; Tan et al., 2020), spurious patterns or correlations (Zhang et al., 2019; McCoy et al., 2019), and counterfactual examples (Gardner et al., 2020; Kaushik et al., 2020). However, the same attention has not been shown to dialects, which vary systematically in their syntax, morphology, phonology, orthography, and lexicon (Jurgens et al., 2017). To this end, we introduce the evaluation set by adapting from the in-distribution examples (SAE) to out-of-distribution examples (AAVE) on GLUE benchmarks. Our goal is to develop robust models that have a good performance on test sets in different linguistic variations. We constructed VALUE from the widely-used GLUE benchmark (Wang et al., 2019), which contains NLU tasks such as natural language inference (e.g., MNLI; Bowman et al.), question answering (QNLI; Rajpurkar et al.), and linguistic acceptability (CoLA; Warstadt et al.). For each of the main 3702 tasks, we translated the Standard American English (SAE) into a synthetic form of AAVE  a form containing many of AAVE's distinguishing features with extremely high concentration. We implemented these transformations using a set of lexical and morphosyntactic rules derived from a broad survey of the linguistics literature (Collins et al., 2008; Green, 2002; Labov, 1972; Labov et al., 1998; Sidnell, 2002; Stewart, 2014; Thompson, 2016; Wolfram and Schilling, 2015). These features were specifically chosen for their high empirical attestation across regional and generational variants of AAVE. This work represents the first attempt to systematically catalogue and operationalize a set of computational rules for inserting AAVE-specific language structures into text. We distill field linguists's observations into procedural code, which operates on specific grammatical conditions from the SAE source. Each grammatical condition is specified by the part of speech tags and syntactic dependency relationships present in the text. Appendix A.1 lists all implementation details for each transformation rule, and we will now enumerate them briefly. Auxiliaries. AAVE allows copula deletion and other auxiliary dropping (Stewart, 2014; Green, 2002; Labov, 1972; Wolfram and Schilling, 2015). This means the SAE sentence  We are better than before  could be rendered in AAVE without the copula as  We better than before . We look for the present tense is and are as well as any tokens with AUX part of speech tag to drop (under special conditions listed in more detail in Appendix A.1). Completive done and remote time been . The phrase  I had written it.  can be rendered in AAVE as  I done wrote it  using the completive verbal marker done . The phrase  He ate a long time ago  can be rendered as  He been ate  using the remote time been (Green, 2002). Constructions involving the word ass . These constructions may be misclassified as obscenity, but they serve a distinct and consistent role in AAVE grammar (Spears et al., 1998). One common form is called the ass camouflage construction (Collins et al., 2008), and it can be seen in the phrase  I divorced his ass . Here, the word behaves as a metonymic pseudo-pronoun (Spears et al., 1998). Similarly, it can appear reflexively, as in  Get yo'ass inside . Ass constructions can also serve as discourse-level expressive markers or intensifiers, as in the compound  We was at some random-ass bar . Existential dey / it . AAVE speakers can indicate something exists by using what is known as an it or dey existential construction (Green, 2002). The existential construction in  It's some milk in the fridge  is used to indicate  There is some milk in the fridge . We identify existential dependencies for this transformation. Future gonna and immediate future finna . AAVE speakers can mark future tense with gon or gonna , as in  You gon understand  (Green, 2002; Sidnell, 2002). In the first person, this becomes I'ma . In the immediate future, speakers can use finna (or variants fixina, fixna and fitna ), as in  I'm finna leave.  Have / got. In the casual speech of AAVE and other dialects, both the modal and the verb form of have can be replaced by got (Trotta and Blyah-her, 2011). Have to can become got to or gotta , and similar for the verb of possession. We simply convert the present-tense have and has to got and ensure that the verb has an object. Inflection. In AAVE, speakers do not necessarily inflect simple present or past tense verbs differently for number or person (Green, 2002). This means the SAE sentence  She studies linguistics  could be rendered in AAVE as  She study linguistics . We use the pyinflect library to convert all present and simple past verbs into the first person. Negative concord. This widely-known feature of AAVE (and numerous other dialects) involves two negative morphemes to convey a single negation. (Martin et al., 1998). For example, the SAE sentence  He doesn't have a camera  could look more like  He don't have no camera  in AAVE. This transformation rule is sensitive to the verb-object dependency structure, and requires that the object is an indefinite noun (Green, 2002). Negative inversion. This feature is superficially similar to negative concord. Both an auxiliary and an indefinite noun phrase are negated at the beginning of a sentence or clause (Green, 2002; Martin et al., 1998). For example, the SAE assertion that  no suffering lasts forever  could be rendered in AAVE as  don't no suffering last forever.  3703 Null genitives. AAVE allows a null genitive marking (Stewart, 2014; Wolfram and Schilling, 2015), like the removal of the possessive 's in  Rolanda bed  (Green, 2002). We simply drop any possessive endings ( POS ) from the text. Relative clause structures. There is a grammatical option to drop the Wh-pronoun when it is serving as the complementizer to a relative clause, as in  It's a whole lot of people  don't wanna go to hell  (Green, 2002). In our transformation, we simply drop all lemmas who and that where the head is a relative clause modifier. Some of the most recognizable differences between SAE and AAVE are found in the lexicon and orthographic conventions. Because we are not aware of any comprehensive AAVE lexicons, we automatically learn our own SAE to AAVE dictionary from public data, and we will provide this resource in our public repository. This dictionary serves as a mapping between plausible synonyms (e.g., mash / press ; homie / friend ; paper / money ) and orthographic variants (e.g., da / the ; wit / with ; sista / sister ) In a method inspired by Shoemark et al. (2018), we trained a skip-gram word embedding model 1 (Mikolov et al., 2013) on the public TwitterAAE dataset of Blodgett et al. (2016). This dataset contained attested code-switching behavior, which allowed us to extract a linguistic code axis c in the embedding space, defined by the average c = (cid:88) ( x i , y i )  S x i  y i | S | where S was our seed list of known priors from Shoemark et al. (2018), given in Appendix A.2. Next, we ranked the candidate word pairs w i , w j by cos( c , w i  w j ) following Bolukbasi et al. (2016). In this ranking, we consider only the pairs whose cosine similarity passed a threshold  , where  was defined by the bottom quartile of the cosine similarities in our seed set S . After automatic filtering, we were left with 2,460 pairs. We hand-filtered this list to remove any semantically dissimilar words, like fishin/kayakin or mom/gramps . This left us with 1,988 pairs. AAVE variants. We provide a sample of this mapping in Table 1. In the final step of the translation, we chose uniformly at random between the AAVE variants to make our substitution. We simply scanned the GLUE dataset and swapped any known tokens from SAE to AAVE. Our transformed tasks are all derived from GLUE. We skip Diagnostics because it is not a benchmark, and we do not transform the Microsoft Research Paraphrase Corpus (Dolan and Brockett, 2005) because it is proprietary. However, we do transform the remaining seven benchmarks, which include the single-sentence tasks (i) Stanford Sentiment Treebank (SST-2) which involves classifying the sentiment of movie reviews as positive or negative, and (ii) Corpus of Linguistic Acceptability (CoLA) which involves deciding whether a sentence is linguistically acceptable or not; the similarity and paraphrase task called Semantic Textual Similarity Benchmark (STS-B), which involves predicting the similarity ratings between two sentences; and the inference tasks (i) Multi-Genre Natural Language Inference (MNLI) which involves classifying the relationships between two sentences as entailment, contradiction, or neutral, (ii) Question Natural Language Inference (QNLI) which involves predicting whether a given sentence is the correct answer to a given question; and finally (iii) Recognizing Textual Entailment (RTE) which involves predicting an entailment relation between two sentences. Ta-3704 Dataset # data ass aux been dey/it got lexical neg cncrd null gen null relcl uninflect CoLA 1,063 9% 15% 6% 2% 2% 51% 4% 3% 3% 17% MNLI 9,682 30% 20% 9% 4% 5% 69% 4% 11% 10% 23% QNLI 5,725 16% 42% 2% 1% 3% 50% 1% 10% 4% 17% QQP 390,690 16% 2% 3% 63% 3% 59% 1% 3% 3% 13% RTE 3,029 48% 40% 36% 3% 5% 81% 4% 28% 25 40% SST-2 1,821 31% 25% 5% 3% 4% 64% 4% 14% 15% 39% STS-B 1,894 1%  0 32% 2% 3% 2% 9% 4% 2% 5% WNLI 146 48% 36% 38% 3% 16% 90% 1% 37% 12% 33% Table 2: Dataset statistics reveal important differences between VALUE datasets, which come in markedly different sizes. ble 2 provides a set of summary statistics for these datasets. It is clear that they come in different sizes, and that the some tasks have been more heavily modified than others. However, most of the sentences in this benchmark have undergone at least one transformation. Since our morphosyntactic transformations are rule-based rather than data-driven, it is especially important to validate that these rules are aligned with real AAVE speakers' grammaticality judgments. User-Centered Validation Protocol. We opt for a participatory design process (Schuler and Namioka, 1993) to help ensure that these transformations are usable and meet the language practices of real speakers. We partnered with DataWorks, 2 an initiative started in Georgia Tech's College of Computing that seeks to involve members of underrepresented and economically disadvantaged groups in research and data annotation. All annotators were AAVE speakers and members of the Black community in Atlanta, and they were compensated for their time. Four volunteers from DataWorks partnered in the design of this rule-validation process. Specifically, we co-designed appropriate questions to measure the linguistic and social plausibility of our transformation system. The HIT questions were based on a pair of utterances: (1) the original SAE sentence from the GLUE benchmark, and (2) the transformed AAVE sentence using only the morphosyntactic rules. We highlighted and indexed the portions of utterance (1) that were transformed in utterance (2), and 2 https://dataworkforce.gatech.edu/ we asked annotators for a binary grammaticality judgment. Separately, we asked for the social acceptability using a scale that was co-designed by DataWorkers. Then, for text marked as ungrammatical, annotators provided us with the indices at which transformation errors occurred. The task was hosted on the Amazon Mechanical Turk sandbox platform, but we interfaced with the annotators throughout the entire annotation process to answer any questions. In early iterations of the task, DataWorkers discussed confusions and disagreements with the authors, and we discovered that the greatest variation in their judgments came not from differences in the speakers' underlying grammars, but rather from their different intuitions about what is socially acceptable (alternatively awkward and unnatural) to say in certain social settings. To disentangle these factors, DataWorkers helped us design a 10-point social acceptability Likert scale with the following vernacular: If someone said this in your community, would it be (1) not very cool, (5) a bit sensitive, (7) passing, or (10) cool? Separately, we discussed certain orthographic conventions that we had adopted from the linguistics literature. DataWorkers indicated that some of these conventions were disagreeable  especially the spelling for there are as dey from Green (2002). Some DataWorkers suggested we use the spelling dey're instead. Relatedly, the DataWorkers found the ass constructions sensitive, given its long history of mischaracterization as an expletive, as well as the broader relationship between such dialect misunderstandings and racial injustice (Rickford and King, 2016; Rickford, 2016). We simply excluded ass constructions from the validation. DataWorkers also reported sentences from the original GLUE task were highly offensive (e.g. mentions 3705 Accuracy Accuracy Size Transformation (Maj. Vote) (Unanimous) n Ass constructions -Auxiliaries 96.6 77.4 638 Been / done 95.4 72.7 670 Existential dey/it 91.4 57.9 304 Gonna / finna 95.4 78.7 197 Have / got 96.2 84.8 290 Inflection 97.1 82.3 761 Negative concord 95.9 73.6 584 Negative inversion 95.0 69.3 101 Null genitives 97.9 85.3 573 Relative clause structures 94.1 58.3 489 Table 3: Accuracy of SAE  AAVE transformations and n the number of instances present. of sexual violence). We used the Perspective API 3 and the offensive language classifier Davidson et al. (2017) to filter out such instances. Finally, we discussed the visual and interactive elements of the task itself. Workers preferred to see the synthetic AAVE text appear with visual priority above the SAE sentence. We also adjusted the color scheme to maximally distinguish concepts of social and grammatical acceptability. The word acceptability itself was triggering for the DataWorkers because it evoked the history of linguistic discrimination against AAVE speakers based on ignorant and prescriptive claims regarding correct or proper English. For this reason, we modified the prompt to read: Do the words and the order of the words make sense? With extensive follow-up meetings, we clarified that to make sense means a sentence follows expected and consistent language rules (i.e. a speaker's internal grammar). Results. In the end, we collected acceptability judgments from three DataWorks workers for each of 2,556 randomly sampled sentence transformation pairs. We observed fair inter-annotator agreement with Krippendorf's  = 0 . 26 . Table 3 presents the aggregate judgments for local transformations in each morphological category. Here, we report the transformation accuracy as the proportion of local transformations marked as acceptable by majority vote or unanimous consensus, and we find our transformation rules are strongly validated. Majority vote gives nearly 100% accuracy for all transformation types. Even under strict unanimous consensus, the accuracy exceeds 70% for seven of the 11 transformation types. Overall, this shows the 3 https://www.perspectiveapi.com/ quality of our linguistic transformations. Error Analysis. Although our transformation rules are generally valid, errors can stem from an overapplication of the rule in restricted contexts. For example, most rules do not apply to idioms or named entities, so if we see a brand name like Reese's Pieces , we should not remove the possessive s . Other observed challenge cases include the subjunctive mood and subject inversions in questions, the non-standard morphology of certain contractions, as well as co-reference and scoping issues in relative clauses, ellipsis, and long-range dependencies (See Appendix D for more details). These each may introduce their own special cases that could be coded in future iterations. For a more reliable test set, we next construct a gold standard in Section 4.2 4.2 Building a Gold Test Set Despite the advantages of controllable feature transformations for benchmarking with explainable error analysis, we cannot rely on the synthetic benchmark alone. Synthetic data may not fully capture the social and structural nuances of AAVE, nor speakers' dynamic and contextual use of dialect feature density. This motivates us to build a small test set of Gold Standard AAVE utterances. Here, annotators considered GLUE sentence transformations as before. The DataWorkers could either (1) confirm that synthetic transformation was natural, or alternatively (2) provide us with their own translation of the SAE text. Together, datapoints from (1) and (2) construct our Gold Test Set. 4 We provide the distribution of Gold Standard datapoints for each task in Table 4. In future iterations, we will expand the total size of the Gold Test sets for reliable benchmarking. In this section, we stress-test current systems on NLU tasks and reveal performance drops on dialect-variants. We investigate the effectiveness of standard training on VALUE and we ablate the dialect test set to understand which dialect features most significantly challenge models. We have two variants of synthetic AAVE data. In AAVE (VALUE) , we apply the full suite of Sec-4 Note: we did not build a Gold CoLA Test set. The nature of the annotation task would be ambiguous since CoLA itself contains intentionally ungrammatical utterances. It is not clear how annotators should translate ungrammatical SAE into ungrammatical AAVE. tion 3 transformations to the standard GLUE tasks. In AAVE Morph , we have an ablated variant of VALUE where only the morphosyntactic transformations (Section 3.1) are executed. By testing base SAE models on this data, we can disentangle the challenges associated with vocabulary shift from those associated with structural differences. If the challenges of VALUE were entirely lexical, we would anticipate that any performance disparity could be recovered with domain-specific word embeddings, since prior work has found such embeddings adequately represent the meanings of new words in AAVE corpora (Hwang et al., 2020b). The most direct way to prepare models for a particular language variety is to directly train them on a dialect-variant of the task. Using our transformation rules (Section 3), we first augment the GLUE training set with AAVE features and then re-train the models (125M-parameter RoBERTA-base) on the augmented data. Following Liu et al. (2019), the batch size was 16. The maximum learning rate was selected as 5 e  4 and the maximum number of training epochs was set to be either 5 or 10 . Table 5 compares the performance of RoBERTa models trained and tested on SAE or AAVE-variants of seven natural language understanding tasks in GLUE. Results are given as Matthew's Correlation for CoLA, Pearson-Spearman Correlation for STS-B, and Accuracy for all other tasks, averaged over three random seeds. In most cases, training jointly on GLUE and VALUE (SAE + AAVE) leads to best performance . With a single training set, there is an expected pattern: training with the corresponding train set typically leads to best performance on the corresponding test set. With the exception of RTE, 5 base models all suffer a drop in performance when tested on the full 5 RTE may be an outlier because of variance due to its small size: only 2.5k data points vs. QNLI with 100k AAVE (VALUE) test set compared with the models trained on AAVE or jointly on SAE + AAVE (e.g., a 1.5% drop on SST-2; a 0.9% drop on QNLI compared to SAE + AAVE). Performance gaps of a similar magnitude are observed when we test on the Gold Test set (e.g., a 1.2% drop on SST-2; a 0.8% drop on QNLI). Further effort is needed to make the current NLU models more robust to dialect variations. We also see that AAVE Morph challenges current models, which suggests that strategies for resolving any performance gap should take dialect morphology and syntax into consideration. Compared to the AAVE column, there is a less severe but still visible drop in AAVE Morph testing: from 94.3 to 93.2 in SST-2, and from 92.6 to 92.0 in QNLI, for instance. Thus we conclude that the challenge with dialects extends beyond a mere difference in the lexicon. Finally, we run a perturbation analysis (Alvarez-Melis and Jaakkola, 2017) to better understand the impact of each dialectal feature on model performance. For the sake of simplicity, we focus only on MNLI. Specifically, we are interested in cases where the introduction of a particular feature results in a model error. Therefore, we count, for each feature transformation function T , the number of sentence pairs ( x 0 i , x 1 i ) for which a GLUE-trained RoBERTA model f changes its prediction from a correct inference y i to an incorrect inference under the transformation. Not all sentence structures allow for new features, so we consider only the subset of pairs for which the transformation is effective in the hypothesis sentence, and where the original GLUE pair had been predicted correctly. Then the ratio r T is be defined as: r T = (cid:12)(cid:12)(cid:8) ( x 0 i , x 1 i )  XT : f ( x 0 i , T ( x 1 i )) (cid:54) = y i (cid:12)(cid:12)(cid:9) |X T | Here XT is: XT = { ( x 0 i , x 1 i ) : T ( x 1 i ) (cid:54) = x 1 i  f ( x 0 i , x 1 i ) = y i } 3707 Test Synth. and r T indicates the proportion of inferences that were flipped to an incorrect label in the presence of T . We report this ratio for each feature in Table 6. The first column in table shows that, when we introduce a negative inversion into a Hypothesis sentence for which the GLUE-trained RoBERTa model was originally correct, then in 9.09% of cases, that correct label would be flipped to an incorrect one. 6 The inflection rule and been / done constructions appear less challenging, but still result in 2.88% and 3.06% of new errors respectively. The remaining table columns indicate the contributions of different model mistakes to the overall r T ratio. For example, the single error due to negative inversion occurs here when the model mistakes a neutral relationship for entailment (n  e) in the following pair: PREMISE :  Still, commercial calculation isn't sufficient to explain his stand  and HYPOTHESIS :  Won't nothing be enough to explain his strong opinion . In negative concord environments, we most often see neutral pairs mistakenly labeled as contradictory (n  c), as with the PREMISE :  Each state is different...  and HYPOTHESIS :  You can go from one area of a state to another and not see no resemblance . For more examples, see Tables 8 and 9 in Appendix C. 6 Why Not Use Style Transfer? We qualitatively investigated the differences between our rule-based approach and a very well-performing unsupervised dialect style transfer model, STRAP Krishna et al. (2020). To train STRAP , we created a pseudo-parallel corpus using a diverse paraphrase model to paraphrase different styles of text, including SAE and the AAVE text from the TwitterAAE corpus Blodgett et al. (2018). Then we fine-tuned a GPT-2 model as the inverse paraphrase function, which learned to reconstruct the various styles. We used the SAE paraphrase model and the AAVE inverse paraphrase model to transfer from SAE to AAVE. In general, we found that STRAP is capable of much greater output diversity. However, in a systematic analysis of dialectal NLU, the first goal is to ensure that the underlying relationships like entailment are not distorted. STRAP can distort the meaning of the text with hallucinations and deletion of key details. Our transformation approach preserves the meaning of the text and thus better captures AAVE morphosyntax. See Appendix E for more details. 7 Conclusion This work introduces the English VernAcular Language Understanding Evaluation (VALUE) benchmark, a challenging variant of GLUE that we created with a set of lexical and morphosyntactic transformation rules. We constructed rules for 11 fea-6 This is the highest error rate for any transformation rule. Note that |X T | = 11 datapoints is a much smaller sample size so the r T estimate is more variable. 3708 Feature r T c  n c  e n  c n  e e  c e  n |X T | Auxiliaries 4.20 0.20 0.07 1.62 0.88 0.68 0.74 1,477 Been / done 3.06 0.22 0.00 1.31 0.44 0.22 0.88 457 Inflection 2.88 0.33 0.20 0.59 0.46 0.39 0.92 1,526 Lexical 5.92 0.67 0.27 1.35 0.57 0.88 2.18 4,902 Negative concord 6.88 0.64 0.16 2.56 0.16 2.08 1.28 625 Negative inversion 9.09 0.00 0.00 0.00 9.09 0.00 0.00 11 Relative clause structures 5.86 0.31 0.62 1.23 0.62 0.31 2.78 324 Table 6: Perturbation analysis. The first column r T gives the proportion of testing instances where the introduction of a particular dialect feature results in a new model error. This column indicates that negative inversions are the most challenging for MNLI. The final column gives the size of the set XT , which is the denominator in the ratio r T . The remaining columns indicate the contributions of different error types to the cumulative r T : the model flips the correct label on the left  into the incorrect label on the right side. c : contradiction; n : neutral; e : entailment. tures of AAVE, and recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. Experiments show that the introduction of new dialectal features can lead to a drop in performance. We also test methods for efficiently adapting models to different language varieties, and discuss dialect specific challenges that our current NLP models are struggling with. Our work sheds light on the disparities of language technologies and has key implications for facilitating more dialect-competent NLU systems. Our longer term goals are to expand VALUE to more NLP tasks such as CoQA (Reddy et al., 2019), and to include other dialects such as Indian English (Demszky et al., 2021b; Lange, 2012; Bhatt, 2008) and Singapore English (Wee, 2008). Limitations and Considerations. Researchers and practitioners should keep the following limitations and considerations in mind when using VALUE. Firstly, dialects are not the deterministic speech patterns that our transformation rules might suggest. While speakers of a dialect have linguistic competence over systematic and internalized grammar rules, speakers still posses an individual degree of control over which features they will employ (Coupland, 2007). The density of these features can vary, not only along demographic axes of geography, age, and gender (Nguyen et al., 2016), but also with different identity presentations in different social contexts (Bucholtz and Hall, 2005). We use VALUE to stress-test current systems by maximally modifying current resources with feature transformations. The high density of dialectal features may appear exaggerated here. Secondly, linguists have historically studied dialects through oral speech via live interviews (Rickford, 2002). The descriptions of academic references will not always map perfectly to the written domain (see Section 4.1 on the spelling of dey ). The orthographic conventions of language communities may vary as significantly as do speech patterns. A third and critical concern is the limitation of synthetic data. Synthetic transformations have the advantage of allowing carefully controlled perturbation analysis and scaling up this analysis without the expensive creation of new datasets. However, synthetic data will not fully capture the social and structural nuances of AAVE, nor speakers' dynamic and contextual use of dialect feature density. For this reason, it is important to ultimately test user-facing models on domain-specific and gold-standard dialectal data. We are continuing to expand our gold-standard test set for GLUE tasks. A fourth consideration is the history of linguistic discrimination and the broader relationship between such dialect misunderstandings and racial injustice (Rickford and King, 2016; Rickford, 2016). AAVE has been frequently appropriated and misused by non-Black individuals, especially in online contexts (Reyes, 2005; Ilbury, 2020). To mitigate deployment risks, we ask users to sign a Data Use Agreement (See Ethics Section). Acknowledgements The authors would like to thank the DataWorks team, as well as Elizabeth DiSalvo, Rahual Gupta, and Jwala Dhamala for their helpful feedback. CZ is supported by the NSF Graduate Research Fellowship under Grant No. DGE-2039655. DY is supported by the Microsoft Research Faculty Fellowship. This work is funded in part by Amazon Research Award under the Alexa Fairness in AI. 3709 Ethics Our task comes from the public version of GLUE (Wang et al., 2019). Our annotation efforts revealed non-normative and offensive language in these original datasets, and we caution practitioners to be aware of this. The rules for converting SAE to AAVE are linguistically informed, and are not designed to change the original meaning of the sentence. Due to the participatory design nature of this work, we involved AAVE speakers and volunteers in the task creation and rule validation process. We asked annotators to skip a specific task and take a break if they are overwhelmed with the task. Our annotators were compensated by DataWorks for their time, and volunteered to help build this linguistic resource for their dialects. Note that AAVE is spoken, and our work only involves speakers from Atlanta. We ask that all users sign the following online agreement before using this resource:  I will not use VALUE for malicious purposes including (but not limited to): deception, impersonation, mockery, discrimination, hate speech, targeted harassment and cultural appropriation. In my use of this resource, I will respect the dignity and privacy of all people.  References David Alvarez-Melis and Tommi Jaakkola.
abstain abstain abstain abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain objective method method method abstain abstain abstain method objective objective objective objective objective other other other other other other other other other other other other abstain objective method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method abstain method abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other method method abstain abstain method method abstain method abstain abstain other abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain result abstain abstain abstain method abstain method abstain other abstain abstain abstain abstain abstain other