  Dialect differences caused by regional, social, and economic factors cause
performance discrepancies for many groups of language technology users.
Inclusive and equitable language technology must critically be dialect
invariant, meaning that performance remains constant over dialectal shifts.
Current systems often fall short of this ideal since they are designed and
tested on a single dialect: Standard American English (SAE). We introduce a
suite of resources for evaluating and achieving English dialect invariance. The
resource is called Multi-VALUE, a controllable rule-based translation system
spanning 50 English dialects and 189 unique linguistic features. Multi-VALUE
maps SAE to synthetic forms of each dialect. First, we use this system to
stress tests question answering, machine translation, and semantic parsing.
Stress tests reveal significant performance disparities for leading models on
non-standard dialects. Second, we use this system as a data augmentation
technique to improve the dialect robustness of existing systems. Finally, we
partner with native speakers of Chicano and Indian English to release new
gold-standard variants of the popular CoQA task. To execute the transformation
code, run model checkpoints, and download both synthetic and gold-standard
dialectal benchmark datasets, see http://value-nlp.org.

Computation and Language