ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science
<|reference_start|>ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science: This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in interpreting the traditional EAT measurement by quantifying bias at three levels of increasing granularity: the differential association between two target concepts with two attribute concepts; the individual effect size of each target concept with two attribute concepts; and the association between each individual target concept and each individual attribute concept. Using the ML-EAT, this research defines a taxonomy of EAT patterns describing the nine possible outcomes of an embedding association test, each of which is associated with a unique EAT-Map, a novel four-quadrant visualization for interpreting the ML-EAT. Empirical analysis of static and diachronic word embeddings, GPT-2 language models, and a CLIP language-and-image model shows that EAT patterns add otherwise unobservable information about the component biases that make up an EAT; reveal the effects of prompting in zero-shot models; and can also identify situations when cosine similarity is an ineffective metric, rendering an EAT unreliable. Our work contributes a method for rendering bias more observable and interpretable, improving the transparency of computational investigations into human minds and societies.<|reference_end|>
@article{wolfe2024ml-eat:,
    title={ML-EAT: A Multilevel Embedding Association Test for Interpretable and
  Transparent Social Science},
    author={Robert Wolfe, Alexis Hiniker, Bill Howe},
    journal={arXiv preprint arXiv:2408.01966},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.01966},
    primaryClass={cs.CL cs.AI cs.CY}
}