A common factor in bias measurement methods is the use of hand-curated seed lexicons, but there remains little guidance for their selection. We gather seeds used in prior work, documenting their common sources and rationales, and in case studies of three English-language corpora, we enumerate the different types of social biases and linguistic features that, once encoded in the seeds, can affect subsequent bias measurements. Seeds developed in one context are often re-used in other contexts, but documentation and evaluation remain necessary precursors to relying on seeds for sensitive measurements. There has been increasing concern in the NLP community over bias and stereotypes contained in models and how these biases can trickle downstream to practical applications, such as serving job advertisements. In particular, there has been much recent scrutiny of word representations, with many studies finding harmful associations encoded in embedding models. Combating such biases requires measuring the bias encoded in a model so that researchers can establish improvements, and many variants of embedding-based measurement techniques have been proposed (Bolukbasi et al., 2016; Caliskan et al., 2017; Manzini et al., 2019). These measurements have had the additional upstream benefit of providing computational social science and digital humanities scholars with a new means of quantifying bias in datasets of social, political, or literary interest. Researchers increasingly use embeddings (Garg et al., 2018; Knoche et al., 2019a; Hoyle et al., 2019) and other lexicon-based methods (Saez-Trumper et al., 2013; Fast et al., 2016; Rudinger et al., 2017) to provide quantitative answers to otherwise elusive political and social Target Concept Highlighted Seeds Unpleasant divorce, jail, poverty, cancer, ... questions about the biases in a corpus and its authors. This work often involves comparing bias measurements across different corpora, which requires reliable, fine-grained measurements. While there is a wide range of bias measurement methods, every one of them relies on lexicons of seed terms to specify stereotypes or dimensions of interest. But the rationale for choosing specific seeds is often unclear; sometimes seeds are crowd-sourced, sometimes hand-selected by researchers, and sometimes drawn from prior work in the social sciences. The impact of the seeds is not well-understood, and many previous seed sets have serious limitations. As shown in Table 1, the seeds used for bias measurement can themselves exhibit cultural and cognitive biases (e.g., reductive definitions), and in addition, linguistic features of the seeds (e.g., frequency) can affect bias measurements (Ethayarajh et al., 2019). Though they are often re-used, the suitability of these seeds to novel corpora is uncertain, and while evaluations sometimes include permutation tests, distinct sets of seeds are rarely compared. We use a mixture of literature survey, qualitative analysis of seed terms, and analytic methods to explore the use of seed sets for bias measurement through two overarching research questions. (1) We explore how seeds are selected and from which sources they are drawn to better understand rationales and assumptions underlying common seed sets. (2) We explore which features of seeds can cause instability , including both social biases and linguistic dimensions in our analysis. Our work provides the following contributions. Documentation: We document and test 178 seed sets used in prior work, and we release this documentation as a resource for the research community. 1 Analysis: We provide a systematic framework for understanding the different sources of instability in seed sets that can affect bias measurements. We compare the gathered seeds to larger sets of artificially created seed sets, and we investigate the reliability of seed terms used for two popular embedding-based bias measurement methods in case studies on three datasets. Recommendations: With this larger perspective, we discuss how seed sets should be examined versus how these sets are popularly considered and what kind of documentation best practices should be followed. Seeds are a brittle but unavoidable element of current bias measurement algorithms, with weaknesses that need probing even for embedding-based measurements. The term bias has many definitions, from a value-neutral meaning in statistics to a more normative meaning in socio-cultural studies. In the bias measurement literature in NLP, lack of precise definitions and problem specifications (Blodgett et al., 2020) has led to many of the errors we explore in this paper. In general, bias in NLP most often represents harmful prejudices (Caliskan et al., 2017) whose spurious and undesirable influence can affect model outputs. While these downstream effects have inspired work on removing bias from embedding models (Bolukbasi et al., 2016), there have also been critiques of these efforts (Gonen and Goldberg, 2019), and we do not focus on this use case in our study. Instead, we focus on bias measurement as a tool used in diverse settings to make comparisons across specific corpora of interest. Unsupervised methods for bias measurement have included pointwise mutual information (Rudinger et al., 2017), normalized frequencies and cosine similarity of TF-IDF weighted word vectors (Saez-Trumper et al., 2013), generative models (Joseph et al., 2017; Hoyle et al., 2019), 1 Seeds and documentation are available at https://gi thub.com/maria-antoniak/bad-seeds and a combination of odds ratios, embeddings, and crowd-sourcing (Fast et al., 2016). All of these methods rely on sets of seed terms. While much recent NLP work has focused on contextual embeddings, most recent bias-detection work has focused on vocabulary-based embeddings and word representations. Researchers have increasingly used embedding-based methods to measure biases and draw comparisons in training corpora of social interest (Kim et al., 2014; Hamilton et al., 2016; Kulkarni et al., 2016; Phillips et al., 2017; Kozlowski et al., 2019). For example, Bhatia et al. (2018) train embedding models on news sources to compare trait associations for political candidates. We believe that our results should extend to contextual embedding methods (Zhao et al., 2019; Sedoc and Ungar, 2019), but vocabulary-based embeddings are easier to analyze. We discuss several recent studies that include analysis of seed sets (Kozlowski et al., 2019; Ethayarajh et al., 2019; Sedoc and Ungar, 2019) in 8. Training Corpora. Our dataset choices are guided by our focus on the upstream use case, where embeddings are trained on relatively small, special-purpose collections to answer social and humanist questions about the training corpus. The scope of these datasets fits the use case of a social scientist interested in measuring bias during a small time window, across specific genres, or in a particular set of authors. Table 2 shows an overview of the data, and more details are in the Appendix. articles from April 15th-June 30th, 2016; high quality WikiText articles, using the full WikiText-103 training set (Merity et al., 2016); and Goodreads book reviews for the romance and history and biography genres, sampled from the UCSD Book Graph (Wan and McAuley, 2018; Wan et al., 2019). For added validity, we also replicate existing studies, using a pre-trained model on a large Google News corpus (Mikolov et al., 2013). For each dataset, we lowercase all text, parse and obtain POS tags using spaCy (Honnibal et al., 2020), tokenize the text into unigrams, and filter words that occur fewer than 10 times in the training dataset. Lowercasing controls for the varying levels of capitalization used in the gathered seeds. We leave analysis of bigram seeds to future work and rely on unigrams as a simplifying assumption. Corpus-Derived 7/18 papers Re-Used 7/18 papers Borrowed from Social Sciences 6/18 papers Curated 5/18 papers Adapted from Lexical Resources 3/18 papers Crowd-Sourced 2/18 papers Population-Derived 2/18 papers Table 3: Overview of the surveyed seed sources. Gathered Seeds Sets. We gather 178 seed sets used in a representative sample of 18 highly-cited prior works on bias measurement. Seeds include both embedding-based and non-embedding-based bias detection methods as there is often crossover and re-use of seed sets. Because we use word embedding models trained on unigrams, we do not include bigram seeds in our analysis, and in each experiment, we omit words that were not present in our training set. While these choices could be seen as limitations, we see them as realistic applications of seeds to constrained datasets, reflecting the scenario in which biases are compared across specific corpora. Figure 1 overviews the seed sets, examples used in the paper are documented in the Appendix, and the full collection is shared in the supplementary materials and is available online. How do researchers select seeds, and from which sources are they popularly drawn? We explore this question using the gathered seed sets from prior works on unsupervised bias detection. The origins of these seeds and the rationales for using them are not always explained by researchers, but in cases where we were able to determine a source or rationale, we group them into the following categories. Table 3 overviews the source frequencies. We emphasize that each source comes with risks and benefits; there is no one correct method to selecting seeds, but awareness of pros and cons can help guide decisions and evaluation methods. Borrowed from Social Sciences. Seed sets are often borrowed from prior work in psychology and other social sciences, usually in an effort to either replicate results or build confidence from previously validated work. For example, Caliskan et al. (2017) validate prompts from the Implicit Association Test (Greenwald et al., 1998), while Garg et al. (2018) and Hoyle et al. (2019) use personality traits from Williams and Bennett (1975); Williams and Best (1977, 1990). Sometimes the seeds appeal for validity via highly cited resources, like LIWC (Pennebaker et al., 2001), despite critiques about unreliability (Panger, 2016; Forscher et al., 2017). Borrowing seeds does not absolve researchers from examining and validating seeds. Crowd-Sourced. Custom seed sets can be created through crowd-based annotation. Fast et al. (2016) use Mechanical Turk to validate the inclusion of terms in their seed sets; the final terms are then included in packaged code for researchers and practitioners. Kozlowski et al. (2019) use Mechanical Turk to gather ratings of items scaled along gender, race, and class. Crowd-sourcing can aid in gathering contemporary associations and stereotypes. However, controlling for crowd demographics can be difficult, and crowd-sourcing can result in alarming errors, in which popular stereotypes are hard-coded into the seeds (as in Table 1). Population-Derived. Some seed sets are derived from government-collected population datasets. Popular sources include U.S. census data (Boluk-basi et al., 2016; Caliskan et al., 2017), the U.S. Bureau of Labor Statistics (Caliskan et al., 2017), and the U.S. Social Security Administration (Garg et al., 2018). These sources are usually used to gather names and occupations common to certain demographic groups. These sources tend to be U.S.-centric, though the training data for the embedding does not always match (e.g., large Wikipedia datasets are not guaranteed to have U.S. authors). Reliance on these sources is particularly vulnerable to reductive definitions of the target conceptse.g., gender (Keyes, 2017)and assumes a level of trust and representation in the data collection that might not exist evenly across groups. Adapted from Lexical Resources. Some seed sets are drawn from existing dictionaries, lexicons, and other public resources, such as SemEval tasks (Zhao et al., 2018) and ConceptNet (Fast et al., 2016). Pre-packaged sentiment lexicons are a popular source (Saez-Trumper et al., 2013; Sweeney and Najafian, 2019); these lexicons include the Affective Norms for English Words (ANEW) (Bradley and Lang, 1999) and negative/positive sentiment words from Hu and Liu (2004). These seeds have the advantage of previous rounds of validation, but this does not guarantee validity for new domains. Corpus-Derived. Quantitative methods can be used to extract seed terms from a corpus of interest. For example, Saez-Trumper et al. (2013) use sorted lists of named entities extracted from a target dataset to create seed sets for personas of interest. Similarly, Sweeney and Najafian (2019) extract high frequency identity terms from a Wikipedia corpus. These methods have the advantage of ensuring high frequency terms in the target dataset, but they pose similar risks to crowd-sourcing; unless an extra round of cleaning and curation is completed by the researchers, terms with unintended effects can be included in the seed sets. Curated. Seed sets are sometimes hand-selected by the authors, usually after close reading of the corpus of interest. For example, Rudinger et al. (2017) hand-select a set of seed terms that correspond to a set of demographic categories of interest, and Joseph et al. (2017) hand-select a set of identity seeds based on their frequency in a Twitter dataset. Often, even when papers rely on other seed sources, manual curation is included as a step in the seed creation process. Hand-curation can result in high precision seeds, but this method relies on the au-thors' correction for their own social biases. Re-Used. Finally, many papers rely on prior bias measurement research for seed terms. The most popular sources in our survey include early papers on bias in embeddings such as Bolukbasi et al. (2016) and Caliskan et al. (2017). This repetition means that the seeds are tested on many different datasets, but they should not be trusted without validation; there can be mismatches in frequency and contextual meaning between datasets. In the upstream use case, locally trained word embeddings remain state of the art because fine-tuned pre-trained contextual models might introduce extrinsic information, and it is not feasible to pre-train contemporary contextual embeddings on such small collections. Here, we focus on two popular seed-based methods to detect bias in word embeddings. Bolukbasi et al. (2016) and Caliskan et al. (2017) both introduce embedding-based methods for bias detection that rely on sets of seed words. Each of these methods requires two sets of seed words, X and Y , and one additionally requires matched pairs of seed words { ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , ... } . WEAT. Given a set of embedding vectors w , the Word Embedding Association Test (WEAT) (Caliskan et al., 2017) defines a vector based on the difference between the mean vector of the two target sets, and then measures the cosine similarity of a set of attribute words to that vector. The strength of the association between the target sets X and Y , and the sets of attributes, A , and B , is given by s ( X , Y , A , B ) = (cid:88) x  X s ( x, A , B )  (cid:88) y Y s ( y, A , B ) where s ( w, A , B ) is equal to the difference in average cosine similarities between a query w and each term in A and w and each term in B . To test whether the resulting difference s ( X , Y , A , B ) is significant, this result is compared to the same function applied to randomly permuted sets drawn from X and Y . Caliskan et al. (2017) use WEAT to measure stereotypical associations between sets of targets and attributes, where, for example, the target terms might be arts and science terms, and the attribute terms might be male and female terms. PCA. The principal component analysis (PCA) method tests how much variability there is in the difference vectors between pairs of word vectors (Bolukbasi et al., 2016). If the vector difference between pairs of seed terms can be approximated well by a single constant vector c , then this vector represents a bias subspace . In this case, the subspace is simply a one dimensional vector, though this process could be extended to more dimensions. For each pair of embedding vectors corresponding to one seed word from set X and one from set Y , Bolukbasi et al. (2016) calculate the mean vector of those two vectors and then include the two resulting half vectors from that mean to the two seed vectors as columns in the input matrix. To quantify how large an effect seed features can have on bias measurements, we calculate a set of metrics for both PCA and WEAT methods that summarizes how well the bias subspace represents the target seeds. For each dataset, we use the popular skip-gram with negative sampling (SGNS) algorithm to train a word2vec model. We use the gensim package for training ( Rehurek and Sojka, 2010). We use a window size of 5, a minimum word count of 10, and a vector size of 100 for all experiments. We repeat this process across 20 bootstrapped samples of each dataset. For PCA, we calculate the difference vector between the embedding vectors for each pair of words in the two seed sets. For each set of paired seed sets, we run PCA and plot the percent of variance explained by each component. For the gathered seeds, we only use pairings documented in prior work. We perform a manual confirmation that the first component g indeed represents the bias subspace by ranking all the words in the vocabulary by their cosine similarity to g . For WEAT, we hold the attribute terms constant, where A = [ good ] and B = [ bad ] , while our generated seed sets take the place of the target terms X and Y . Holding the attribute terms constant is a simplifying assumption; our goal is not to test all possible attribute terms but to show that significant variation is possible. We then calculate the WEAT test statistic and significance. Coherence. In addition to the PCA explained variance and WEAT test statistic, we also measure the coherence of each pairing of seed sets after being mapped to the bias subspace. Ideally, when we project all the words in the vocabulary onto the subspace, the two sets would be drawn as far apart as possible. We rank all words by cosine similarity 0.0 0.2 0.4 Similarity to Unpleasantness Vector woman, women, she, her, her,... (Kozlowski et al 2019) sister, female, woman, girl, daughter,... (Caliskan et al 2017) woman, girl, she, mother, gal,... (Bolukbasi et al 2016) woman, girl, mother, daughter, sister,... (Hoyle et al 2019) lady, nun, heroine, actress, businesswoman,...(Zhao et al 2018) baker, counselor, nanny, librarians, socialite,... (Zhao et al 2018) S eed s romance history + biography Figure 2: Bias measurements depend on seeds. where X 1 and Y 2 are seed sets and R 1 and R 2 are their mean ranks in the bias subspace. Finally, we normalize the scores to a [0, 1] range. Higher coherence scores indicate that the seed sets have very different mean ranks, i.e., the seeds are separated by more of the vocabulary. For example, in Figure 4, ordered seeds (a) produce a subspace with greater coherence (sets are further apart in the bias subspace) than shuffled seeds (b). Generated Seed Sets. In order to control for frequency and POS when measuring instabilities due to semantic similarity and word order, we generate a large collection of artificial, randomized seed sets. We select a target term at random from the model's vocabulary, filtered by POS. Each seed set consists of this target term and its four nearest neighbors, ranked by cosine similarity. We repeat this process for each of the models trained on the bootstrapped samples of the corpus. We choose seed sets that are semantically similar (rather than randomly selecting seeds) because we expect that seed sets of realistic research interest would be coherent. We emphasize that researchers have used bias measurement methods for increasingly creative purposes, moving beyond gender and race, and similar bias measurement techniques can be used for aspect detection and other seed-based tasks. Example seeds are shown in Table 4. Before moving to specific seed features, we present some general results showing the instability of measurements using seeds. Figure 2 shows a motivating example, in which we imagine a digital humanities scholar interested in measuring whether women are portrayed more negatively in different genres of book reviews. As in the WEAT test, each seed is plotted according to its cosine similarity to an averaged unpleasantness vector (Caliskan et al., 2017). For some sets, no significant difference is visible, while for other sets, there are much larger differences, causing the researcher to draw different conclusions when comparing biases across datasets. Table 4 shows both the generated and gathered seed sets ordered by their coherence after using the WEAT method to discover a bias subspace. These examples highlight factors contributing to lower coherence (e.g., similarity of the seed sets) which we discuss in 8. They also highlight the general difficulty in constructing seed sets; e.g., as noted by Garg et al. (2018), the final row demonstrates that some U.S. racial categories are not distinguishable from available census data. Similar challenges arise when seeds do not occur in the target dataset, which is often true for names. The wide variation in coherence scores, especially for the generated seeds which are less likely to contain overlapping terms, indicates that different seed sets can have widely differing success for bias measurement. Sometimes seeds can reflect the curator's (or crowd's) personal biases. Instabilities can also arise from the organization of the seeds and seemingly innocuous linguistic features. We describe a series of distinct sources of instability that can be encoded in seed sets and discuss the implications of each. We rely on a combination of literature review, qualitative close reading of example seeds, and quantitative tests of seed features. We iterated through the seeds, flagging problematic sets, and then manually clustered and labeled the factors that could cause instability. Our identified factors can be categorized as definitional factors (reductive definitions, inclusion of confounding concepts), lexical factors (frequency, POS of individual seeds), and set factors (number and order of seeds, similarity of seed sets). Reductive Definitions. The seeds can be reductive and essentializing, codifying life experiences into traditional categories. Using names as place-holders for concepts like race (Nguyen et al., 2014; Sen and Wasow, 2016) or reducing gender to a binary with two extremes (Bolukbasi et al., 2016; Caliskan et al., 2017) can create a distorted view of the source data. Sometimes these are simplifying assumptions, made in an effort to measure biases that would otherwise go unexamined. However, these decisions run the risk of further entrenching these category definitionse.g., see discussions 1 2 3 4 5 6 7 8 9 10 Component 0.0 0.2 0.4 E x p l a i ned V a r i an c e Setting original order shuffled in Keyes (2017); Larson (2017) for the mistakes and harms that can be caused by mapping names to gendersand these trade-offs should be evaluated and documented. More broadly, recent work has critiqued NLP and ML bias research for not successfully connecting with the literature in sociology and critical race studies (Hanna et al., 2020; Blodgett et al., 2020). Engaging with this literature would provide a better foundation for decision-making about seed sets and provide context for future researchers. Imprecise Definitions. If the target concept is not well-defined, the resulting seed terms can be too broad and include multiple concepts, risking the creation of confounded or circular arguments. Similarly, the unexamined use of pre-existing sets and over-reliance on the category labels from prior work can result in a series of errors. The seeds can contain confounding terms (e.g., in Table 1, unpleasant contains cancer which in some datasets might be more prevalent for certain demographic groups) or terms from the target group (e.g., domestic work includes the gendered terms mom and mum). Similarly, the seeds can manifest cultural stigmas: for example, including fat and wrin-kled in an ugliness category (Fast et al., 2016) results in a seed set that itself contains stereotypes. These stigmas are harmful and can interact with other demographic features like gender or age (Puhl and Heuer, 2009), and unless their inclusion is intentional, they can accidentally inflate measurements towards certain groups. Predicting all such errors is impossible, and there can be cases where researchers intentionally include such terms (e.g., to capture a particular stereotype)but this should be a conscious decision by each researcher using the seeds, and at a minimum, researchers should clearly define their target concepts. Lexical Factors. Prior work examining seeds has shown that the frequency and part of speech of seeds can affect the resulting bias measurements. Ethayarajh et al. (2019) show that the WEAT test requires that the paired seeds occur at similar frequencies and that seed sets can be manipulated to produce certain measurements. Brunet et al. (2019) explore the effects of perturbing the training corpus, finding that (1) second-order neighbors to the seeds can have a strong impact on the bias measurement effect size and (2) effects are stronger for rarer words. Using contextual embeddings, Sedoc and Ungar (2019) show that different classes of words (e.g., names vs. pronouns) can result in different bias subspaces and that sometimes these subspaces represent an unintended dimension (e.g., age instead of gender). Set Size and Alignment. The number of seeds included in each set can affect the resulting bias subspace; Kozlowski et al. (2019) find small increases in performance when using more seed pairs. The alignment of the seeds in matched sets (i.e., the ordering or pairing of seeds in one set with seeds in another set) can also affect the bias subspace. In the PCA method, each term in one seed set is explicitly linked to a single term in the other seed set. The specific alignment between paired words matters; altering the pairing can result in dramatically different results, even for cases like gender, which is marked in English. However, we observe conscious pairings of seeds only for obvious cases, and sometimes obvious pairings produce subspaces that explain less variance. We replicate a study previously carried out on embeddings trained on internet-scale collections (Bolukbasi et al., 2016) using both a large, pretrained embedding and the relatively small NYT dataset. Figure 3 shows how much variance is explained by the first ten principal components of three difference matrices. When we use the original paired male-female seed words from Bolukbasi et al. (2016) (e.g., man woman , he she ), we see a single dominant first component, suggesting a strong male-female axis. As previously reported, the variances fall off gradually when the seeds are a set of random words. When we shuffle the order of the seed words, the drop off is steeper than for random pairs, but there is no longer a single dominant principal component. Similarly, Figure 4 shows that when we used the ordered gender pairs, the ranked words roughly divide into groups correlated with gender, while if we use shuffled pairs, the lists of high and low ranked words are not as easily distinguishable as masculine or feminine. We find an opposite effect social class pairs (Kozlowski et al., 2019); when we shuffle, we find a subspace that explains more variance than the explicitly ordered pairs (e.g., richest-poorest). We find similar differences when testing some seed sets that lack intuitive pairings, e.g., the matched pleasantness and unpleasantness seeds (Caliskan et al., 2017) and the matched Christianity and Islam seeds (Garg et al., 2018). Order does not always affect the subspacee.g, we found no significant difference when shuffling sets of namesbut we have shown that it can affect the subspace, and so to build confidence in measurements, testing is required. Set Similarity. By sampling random seed sets we find that it is more difficult to represent the variance of seed sets that are too close together. Figure 5 shows that set similarity (cosine similarity between the set mean vectors) is significantly correlated with explained variance for generated sets (Pearson r =  0 . 67 , p < 0 . 05 ). We highlight two comparisons between gathered sets intended to measure racial bias that explain different degrees of variance. Synthetic pairings generally explain more variance than pairings of gathered sets of equal similarity, although for gathered sets we cannot control for POS and frequency. Table 4 shows the generated seed sets ranked by coherence, where higher scores indicate that the bias subspace was able to separate the seed sets. Similar seed sets and sets with duplicates (e.g., the pairing in the table in which both generated sets contain food terms) have low coherence scores. Almost all recent work on bias measurement relies on sets of seed terms to ground cultural concepts in language. If we do not pay attention to the seeds, these methods will lack foundation and the claims they support will be left open to criticism and dis-missal. Seeds and their rationales need to be tested and documented, rather than hidden in code or copied without examination. Some of the risks discussed in this paper may seem obvious in retrospect, but our literature survey suggests there are widely varying levels of evaluation and documentation. Rationales for picking sources or seeds are not always explained, or the reader is left to assume that prior work has adequately validated the seeds. Tests for frequency, semantic similarity, and other features are rare or non-existent, and clear definitions and discussion of limitations are often missing. Permutation tests are sometimes used, but these do not account for seeds outside of those already selected. Significantly different results can be found using alternative seeds sets for the same target concept, and fine-grained comparisons require validation on multiple sets. We faced a number of challenges in gathering 178 seed sets from prior work. Sometimes seeds are shared online at an undocumented location and sometimes hard-coded into code repositories; this can significantly obscure the seeds from public view, which is troubling for tools intended for wide use on sensitive topics. Documentation is often scattered across locations, and in more than one case, we found contradictions between different sources for a single project. In one case, we were unable to find the full list of seeds used in the paper, and in several cases, it was unclear which seed sets were used for which experiments. While some authors went to commendable lengths to document their materials, there is a need for more consistent and transparent documentation. We recommend that researchers carefully trace the origins of seed sets , with attention to the risks associated with the origin type. We also recommend that researchers examine seed features . POS, frequency, semantic similarity, and pairing order can significantly affect the results of bias measurements. Seeds should be both examined manually and tested as shown in 8; importantly, they should be compared to alternative seeds with different attributes, as in 7. To assist this we release a compilation of 178 seed sets from prior work. These tests are particularly important when comparing biases across datasets. Finally, researchers should document all seeds and the rationales underlying their design, including concept definitions. We add to recent calls for better documentation and problem specification in machine learning (Bender and Friedman, 2018; Gebru et al., 2018; Mitchell et al., 2019; Blodgett et al., 2020) and in studies of social biases in technology (Olteanu et al., 2019). Specifically, when the seeds intentionally encode harmful stereotypes or slurs, it can be beneficial to include a trigger warning or not highlight the seeds in the paper; however, full seed lists should always be accessible, not hard-coded, with unique labels matched to experiments. Ultimately, our goal is not to eliminate a problem but to illuminate it: 2 to help practitioners think through the potential risks posed by seed sets used for bias detection. We encourage thoughtful, critical studies, but we observe a trend in which seed sets are used in new research and applications simply because they have been used in prior published work, without additional vetting. Research precedents can take on a life of their own and we have a responsibility to explore and document possible sources of error. We believe that seed sets can be useful and are probably unavoidable, but that no technical tool can absolve researchers from the duty to choose seeds carefully and intentionally. Thank you to our anonymous reviewers whose comments substantially influenced and improved this paper. Thank you to Rishi Bommasani, Forrest Davis, Os Keyes, Lauren Kilgour, Rosamund Thalken, Marten van Schijndel, Melanie Walsh, and Gregory Yauney for their many helpful suggestions. This work was funded through NSF grant #1652536.
abstain method abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain objective objective objective objective method method objective method abstain other abstain other method method other other other other other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method abstain method abstain method abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain method method abstain abstain abstain abstain abstain method abstain objective objective objective method other other other