Generating factual, long-form text such as Wikipedia articles raises three key challenges: how to gather relevant evidence, how to structure information into well-formed text, and how to ensure that the generated text is factually correct. We address these by developing a model for English text that uses a retrieval mechanism to identify relevant supporting information on the web and a cache-based pre-trained encoder-decoder to generate long-form biographies section by section, including citation information. To assess the impact of available web evidence on the output text, we compare the performance of our approach when generating biographies about women (for which less information is available on the web) vs. biographies generally. To this end, we curate a dataset of 1,500 biographies about women. We analyze our generated text to understand how differences in available web evidence data affect generation. We evaluate the factuality, fluency, and quality of the generated texts using automatic metrics and human evaluation. We hope that these techniques can be used as a starting point for human writers, to aid in reducing the complexity inherent in the creation of long-form, factual text. Wikipedia has become one of the major sources of dissemination of knowledge across the globe. However, the knowledge contained in Wikipedia is not neutral  it is biased in various ways (Hinnosaar, 2019; Schmahl et al., 2020). Many studies, including those from the Wikimedia Foundation itself, have emphasized that biographies in particular are overwhelmingly written about men. This leads to many subtle yet far-reaching effects, from students not writing their first book reports on a woman to bias in models trained on Wikipedia, as Wikipedia has long been used as a source of data. Many existing efforts, such as the Wikipedia Women in Red project, focus on encouraging article creation to mitigate this gender gap. However, Wikipedia articles remain painstakingly written and edited primarily by a network of human contributors. Despite advances in text generation and modeling architectures that retrieve information, the automatic creation of Wikipedia articles is incredibly challenging (Liu et al., 2018). Even the functionality of tools that aid human editors are limited. In this work, we strive to create a system that could write an entire Wikipedia article in English, focusing on the biography domain. We confront several major challenges. First, this is fundamentally a long-form generation task. Improvements driven by pretraining (Radford et al., 2019; Lewis et al., 2019) have improved generation fluency at the level of multiple sentences. However, Wikipedia biographies contain multiple paragraphs in a structured form with headings, as well as citations to indicate where the information originated from. Second, the task confronts obstacles around the factuality (Elazar et al., 2021) of generated content, as articles must be factually accurate. Third, Wikipedia articles are written using reference material, often found on the web (Piktus et al., 2021). Thus, models need to find and ingest web searches as a pre-requisite to writing accurate biographies. We develop a method for English Wikipedia that starts with the subject and occupation of the biography, then leverages web search to find relevant evidence. Given search results, we employ a retrieval-augmented generation architecture (Lewis et al., 2020; Guu et al., 2020) based on large-scale pretraining to identify relevant information and write the biography. We generate section by section, using a caching mechanism similar to Transformer-XL (Dai et al., 2019) to reference previous sections and achieve greater document-level context. Finally, after each section, we append a citation based on which web searches were retrieved. 2004), entailment, and named entity coverage. Further, we study the strong dependency of our method on accurate retrieval, and design a specific evaluation dataset that highlights this challenge. The dataset consists of 1,527 Wikipedia biographies about women, where information on the internet is not as easily retrieved. We use this dataset to analyze the gap between model quality when retrieval is challenging (our novel evaluation dataset with biographies about women) and model quality when retrieval is more accurate (a random set of evaluation biographies). Finally, we conduct a large-scale human evaluation to measure the factuality and coverage of our generated biographies. We hope that our techniques can eventually be used as a starting point for human Wikipedia writers, for biographies and beyond. A large body of work in generation utilizes Wikipedia, often for data-to-text tasks that use Wikidata or DBpedia RDF triples (Gardent et al., 2017; Castro Ferreira et al., 2020; Kaffee et al., 2018b; Vougiouklis et al., 2018; Sha et al., 2018; Puduppully et al., 2019; Chen et al., 2020b; Wang et al., 2020; Agarwal et al., 2020; Parikh et al., 2020), as well as graphs (Jin et al., 2020) as input. Some have focused on long text, such as writing summaries (Chen et al., 2020a) or sections of articles (Kaffee et al., 2020), expanding stubs (Baner-jee and Mitra, 2015), and writing full articles (Liu et al., 2018). Some of these works utilize structure to learn templates (Sauper and Barzilay, 2009), Markov logic networks (Liu et al., 2010), or word graphs (Banerjee and Mitra, 2015), but we anticipate that pretraining and large neural network based techniques will vastly improve upon this quality. Closest to our work, Liu et al. (2018) use web evidence to write full length articles, but do not focus on biographies and use extractive summarisation techniques rather than a retrieval mechanism to identify relevant information. Further, their work generates the entire Wikipedia article at once, whereas we demonstrate that breaking down the article to generate section by section is more effective. We also include a mechanism for the model to generate citations, which was not included in existing work. Thus, our model can produce a full-form Wikipedia article that would look like what a human editor wrote. Finally, our work (i) leverages recent advances in large-scale pretraining, which improves generation fluency and (ii) investigates the impact of available web evidence on the generated texts. Other work has focused on automatic creation of biographies, such as generation from infoboxes (Le-bret et al., 2016) or Wikidata (Chisholm et al., 2017), as well as extracting biographical sentences (Biadsy et al., 2008). The majority of existing research focused on short biographies. Retrieval mechanisms have been used to support a variety of tasks, including dialogue (Moghe et al., 2018; Dinan et al., 2018; Shuster et al., 2021), fact verification (Thorne et al., 2018), and sentence generation (Guu et al., 2018). Most notably, retrieval has been heavily used in question answering (Chen et al., 2017; Kwiatkowski et al., 2019; Seo et al., 2019; Karpukhin et al., 2020). Recent innovations in incorporating retrieval mechanisms have increased the quality and scale of retrieval-augmented generative methods (Guu et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020). Gender bias on Wikipedia is a well-known problem (Hinnosaar, 2019; Dinan et al., 2020; Schmahl et al., 2020), particularly in the case of biographies (Graells-Garrido et al., 2015; Stratigakos, 2016; Luo et al., 2018; Schmahl et al., 2020). This bias is compounded by geographical location, as information about certain areas of the world is far more prevalent (Kaffee et al., 2018a; Beyta, 2020). This bias exists not only in what articles are written, but also in articles targeted for deletion  articles about certain marginalized groups are removed at higher rates (Worku et al., 2020). Wikipedia reflects biases present in society (De-Arteaga et al., 2019; Young et al., 2020; Schmahl et al., 2020), though numerous initiatives exist to de-bias Wikipedia. These range from training programs (Iglesias, 2020) to projects such as Women in Red 1 and WikiProject Women 2 . The success of these initiatives has been studied (Langrock and Gonzlez-Bailn, 2020) and found to be effective, but not at addressing the systemic challenges that create bias in the first place. In the natural language processing community, work has focused on combating gender bias in co-reference resolution (Zhao et al., 2018), dialogue (Dinan et al., 2019; Lee et al., 2019; Liu et al., 2020), detection of abusive language (Park et al., 2018), machine translation (Stanovsky et al., 2019), and word embeddings (Gonen and Goldberg, 2019). These works present a variety of strategies, including data augmentation, additional data collection efforts, modified generation, and fair evaluation (Yeo and Chen, 2020). A comprehensive survey can be found in Blodgett et al. (2020). However, most of these efforts are focused on specific tasks or models  our work uniquely targets generation of full Wikipedia biographies to combat gender bias present on Wikipedia. Given a person's name, one or more occupation(s), and CommonCrawl as a source of evidence, the task is to generate a Wikipedia biography and to associate each generated section with adequate bibliographic references. We model this task by generating a biography section by section using section headers as additional information. A special section header called toplevel is used as the start of the article. The subsequent headers are automatically generated at the end of each section as input for the next. Thus for each section, the input includes a name, one or more occupations, a section header, and CommonCrawl as a retrieval corpus. Wikipedia biographies begin with an introductory paragraph followed by various subsections 3 . To account for this structure and generate long-form text based on retrieved web evidence, our system, illustrated in Figure 1, generates a biography section by section. Based on the subject, their occupation(s), and the section heading, the model first identifies a subset of relevant evidence from a set of web search results found using that triplet ( retrieval module ). It then conditions upon that evidence to generate the section, using a Sequence-to-Sequence model ( generation module ) which can access previous sections using a caching mechanism. Finally, the model indicates which evidence documents it used and outputs those as citations, mimicking a standard Wikipedia article ( citation module ). We focus 3 Many biographies contain infoboxes, which we do not generate. Given a query Q and a set of web documents D retrieved from the web based on this query, the task of the retrieval module is to retrieve the subset of D that is most relevant given Q . The challenge is sifting through the large quantity of potentially useful information. Query. The query Q consists of three parts: (1) the name of the person for which the biography is generated, (2) their , possibly multiple, occupation(s), and (3) a section heading. Including the occupation narrows the realm of potential relevant content, especially as proper names are often ambiguous (e.g. Jane Wang ). Similarly, the section header allows the model to retrieve different information for each section (e.g. Personal Life compared to Career ). Documents. The query Q is put through a search engine to retrieve web hits, which form the set of documents D that are candidates for retrieval. The web results are represented only as text, and all non-text information is discarded. Retrieval. To retrieve the relevant subset of D , each sentence in D is encoded with RoBERTa base trained with LayerDrop (Fan et al., 2019b; Liu et al., 2019; Devlin et al., 2018). The concatenation of the subject's name, occupation(s), and section header is also encoded. We then calculate the dot product to identify which encoded document sentences are most relevant given the currently encoded query Q , following the strategy used in other retrieval works (Karpukhin et al., 2020). The representation of the top k most relevant sentences are then passed onwards through the model. Note that compared to some other retrieval-augmented generation (Lewis et al., 2020), the RoBERTa encoder is not fixed, so the retrieval module learns based on the performance of the generation module. This is possible because our retrieval is far smaller scale, we limit the search to approximately 40 sentences (1,000 words) that could be used to generate each section. To generate the sections we use a Transformer-based Sequence-to-Sequence model initialized with BART-Large (Lewis et al., 2019). The input to BART is the concatenation of the subject's name, 8563 Figure 1: Model Architecture. occupation(s), the section header and the retrieved evidence. Note that the maximum number of input tokens for BART is 1024 words, which is why we cap the retrieval at 1000 words, as described in the previous section. The decoder conditions on the input information to generate the section. One challenge with this is that the sections would be generated completely independently, which might result in redundancy between generated sections. Thus, we equip the Sequence-to-Sequence model with a mechanism to refer to previous sections using the cache mechanism from Transformer-XL (Dai et al., 2019). This mechanism caches the previous section's hidden states at every layer, using it as memory to generate the current section. Recent work has focused on models that not only perform a task, but also produce an explanation (DeYoung et al., 2019). Much of this work has focused on question answering (Latcinnik and Be-rant, 2020; Lamm et al., 2020; Lakhotia et al., 2020; Gonzalez et al., 2020) and generating explanations in natural language (Camburu et al., 2019; Narang et al., 2020; Kumar and Talukdar, 2020; Hase et al., 2020). A similar requirement exists on Wikipedia  not only to collate the information into an article, but to provide the original references for users to verify. Thus, to complete the generation of a full Wikipedia biography, we cite the information used, as in any real article. On Wikipedia itself, each sentence could contain citations. We simplify this, citing at the end of each section. To do this, we track the original document the retrieved evidence originates from, and reference that document at the end of the generated section. To write a full biography, models must generate the introductory paragraph followed by each section. For a new article, the introductory paragraph is given as a section heading called toplevel . For each subsequent section, we follow the process outlined above to retrieve evidence, then write a section, then add citations. At the end of each section, the model generates the section heading of the next section. This allows the model to generate an entire article section by section. A possible failure point for our method is the retrieval step as good biography generation requires access to sufficient relevant information. To study the impact of accurate retrieval on generation quality, we design a specific evaluation dataset that pushes this problem to the forefront. Specifically, we create a novel evaluation dataset which consists exclusively of biographies about women. Ongoing efforts to write biographies about women in the Wikipedia editor community, such as the Women in Red project, have identified insufficient online evidence as a major challenge for writing Wikipedia biographies about women. To study the importance of retrieval on model quality, we therefore create an evaluation dataset where the target Wikipedia articles are women bios. We collate candidate biographies, retrieve information about their occupation, and gather web sources using web search. The resulting dataset, summarized in Table 2, consists of 1,527 biographies, each linked to a set of retrieved web articles. Identifying Biographical Subjects. We first source various notable women on Wikipedia using internet lists (e.g. Famous Women you should know ) and existing efforts by collective groups of Wikipedia editors, such as the Women in Red project. Several recent efforts focus on Women in Science 4 , and so we specifically include scientists as a category. Overall, we collate almost two thousand candidate Wikipedia women biographies. We then narrow down by selecting articles that have previously Featured Article or Good quality. The final evaluation dataset contains 1,527 biographies in four groups: Women, Women in Science, Women in Asia, and Women in Africa (see Table 2). 4 https://towardsdatascience.com/who-is-wikipedia-famous-within-natural- language-processing-fa0c8e91bdf6?gi=b910dd838c47 , https://www.newscientist. com/article/mg24532680-800-jess-wades-one-woman-mission-to-diversify-wikipedias-science-stories/ Biography Text and Occupation. After finalizing candidate Wikipedia biographies, we use the MediaWiki API 5 to query the text of the article. We use the Wikidata API 6 to retrieved the individuals, possibly multiple, occupations (e.g. Rachel Carson is an author and an environmental activist). As seen in Table 2, on average, articles have around 6 sections with 130 words each. The most common occupations include writers, teachers, and doctors (see Table 1), though the entire dataset contains almost 500 different occupations, with people having on average 2 occupations (see Table 2). Retrieving Web Evidence. Next, we identify web sources with reference evidence for each biography. We follow the construction of similar datasets, such as WikiSum (Liu et al., 2018) and ELI5 (Fan et al., 2019c), which searches through CommonCrawl. We query CommonCrawl based on the subject's name and occupation(s) and return the top 20 search results. We reject all CommonCrawl links from Wikipedia, to prevent querying the Wikipedia articles in our dataset. Statistics are presented in Table 2. Out of a maximum of 20 possible hits, on average each biography returns around 18. We describe our training data, baselines, and automatic and human evaluation metrics. Training Data. We utilize the WikiSum (Liu et al., 2018) dataset of Wikipedia articles paired with web references. We filter to biographies using a combination of querying for occupations in Wikidata and using Named Entity Recognition 7 to recognize names. We query each article title in the WikiSum dataset to attempt to find an occupation and see the title is recognized as a named entity, to identify the bibliographical subset of WikiSum. This produces 677,085 biographies, each associated with a set of web articles. Evaluation Data. We utilize the WikiSum (Liu et al., 2018) dataset, filtered to biographies, for evaluation. Similar to the training dataset, we query to identify occupational information. To study the impact of retrieval and available evidence on model 5 https://www.mediawiki.org/wiki/API 6 https://query.wikidata.org/ 7 https://spacy.io/usage/linguistic-features/ 8565 Most Common Section Headings Career, Personal Life, Early Life, Biography, History Most Common Occupations Writer, Politician, University Teacher, Physician, Researcher Table 1: Example Section Headings and Occupations in Wikipedia Biographies. quality, we also evaluate on our constructed evaluation dataset about women (which has substantially less web-based evidence). As shown in Table 2, these two datasets differ in the length and quality of both the Wikipedia articles and the web-based evidence. Baseline. We compare our method described in Section 4 to a pretraining and finetuning generation baseline. We use the BART model (Lewis et al., 2019) and finetune on the Biography subset of the WikiSum data. Note that BART has a token limit of 1024, thus the entirety of the web retrieval is not available to this model. We take the web search hits ordered by the search engine, and provide the first 1000 available tokens. To compare this baseline with our method equitably, the baseline is also trained to generate section by section. However, it does not use the retrieval module (all evidence is given), the caching mechanism, or the citation module (as described in Section 4), meaning citations are not added to the generated text. Additional training details are in the Appendix. Generation. We generate from all models with beam search, setting the beam size to 5. We allow the model to generate an output of any length, with no restrictions. For human evaluations, we set the minimum and maximum length such that it matches the length of the gold target to minimize the effect of length on human interpretations. Automatic Evaluation. We evaluate the quality of generated biographies with three automatic metrics. First, we measure the ROUGE-L between the generated text and the Wikipedia reference text to assess the similarity. ROUGE-L is commonly used in multi-sentence summarization and is a measure of longest common substring overlap. Next, we use Natural Language Entailment as a high level proxy for quantifying a form of factuality: if two sentences entail each other in both directions, then they are semantically equivalent. We use a model pretrained and finetuned on MNLI, open sourced by Liu et al. (2019). To evaluate entailment, we split the generated biography and reference biography into sentences, then for each sentence in the generated biography we calculate if it is semantically equivalent to a sentence in the reference. We then compute the percentage of generated sentences that are semantically equivalent to at least one sentence in the reference biography, where entailment is evaluated bidirectionally. Finally, we assess the Coverage of information in the generated biography, constraining this to analyzing mentions of named entities. We report the percentage of named entities detected in the reference which are also detected in the generated text. We extract entities with BLINK , a BERT-based entity linking system (Wu et al., 2019). Human Evaluation Long-form text generation is very difficult to assess automatically (Thomson and Reiter, 2020; Howcroft et al., 2020), particularly for factuality (Goodrich et al., 2019; Maynez et al., 2020; Peshterliev et al., 2021) and hallucination (Zhou et al., 2020; Duek and Kasner, 2020). We conduct a detailed, large-scale human evaluation with the goal to assess Coverage (How much of the information in the reference section is in the generated section?) and Factuality (How much of the generated section is in the reference and, for the information added in the generated text, how much of that information is verifiable based on the web evidence?). To reduce the challenge of evaluation, the text is compared section by section, and the generated text is the same length as the reference by constraining the max length of beam search (to remove length as an evaluation artifact). First, each sentence of the generated section is shown next to the full reference section and the entire document cited in the generated section (recall our generated biographies cite the retrieved evidence). Evaluators are asked to decide (1) if the information in the generated sentence is present in the reference section (ground truth) and (2) if the information in the generated sentence is present in the cited document (web evi-dence). This question assesses if the information from the generated section is factual with respect to either the reference Wikipedia text or the retrieved web documents. Then, the evaluation is flipped to assess coverage with respect to the Wikipedia reference. Each sentence of the reference is shown next to the generated section, and evaluators are asked to decide (3) if the information in the reference sentence is present in the generated section. In total, human annotators evaluated 100 sections with length between 200 to 500 words. Each section is reviewed by one annotator. Additional details are in the Appendix. Automatic Evaluation. We examine the model's overall performance. Results are summarized in Table 3. Compared to the pretraining+finetuning baseline, adding the retrieval module statistically significantly 8 increases results by 1.4 ROUGE-L. Adding a caching mechanism improves further by 0.5 ROUGE-L. This trend is reflected across the entailment and entity coverage metrics, indicating that retrieving the most relevant information to write a biography is critical. Next, we examine the impact of our modeling choices using ablation (Table 4). Compared to previous work on WikiSum (Liu et al., 2018; Fan et al., 2019a), we add an end-to-end retrieval mechanism based on RAG that substantially improves results. Further, instead of retrieving solely based on the 8 We use the confidence interval reported in the ROUGE package. subject name, as was previously done (Liu et al., 2018), we retrieve on a detailed query (the name, occupation, and section heading). Table 4 indicates that this enriched query improves the retrieval quality by almost 2 ROUGE-L. We conjecture it helps improve disambiguation and retrieve evidence that is relevant to the desired entity rather than to one of its homonyms. We also generate the biographical articles section by section, rather than an entire article at once. This allows the retrieval mechanism to be focused on the section information. As shown in Table 4, this also has a positive effect of +1.5 ROUGE-L. Human Evaluation. Next, we examine quality with human evaluation, as shown in Figure 3. Models generating nonfactual or hallucinated content is an ongoing area of study (Tian et al., 2019; Nie et al., 2019; Liu et al., 2021). Our goal is to understand how much information in the generated text is present in the reference text or the web evidence, as a proxy for factuality and coverage. Overall, 68% of the information in generated sections is not present in the reference text. Conversely, 71% of information in the reference text is not in the generated text. This indicates that the generated text has far from perfect coverage. However, we found that 17% of the added information can be validated by examining the web evidence, which shows that some information added by the generative model is valid biographical information. We examine why there is low information overlap between the generated and reference text. First, information in the reference biography may not be available on the web 9 or may not be retrieved. In a manually examined subset of 250 sentences taken from reference biographies, we found that about 50% of the information was not contained in the web evidence. The other 50% was partially present in the web evidence but were not retrieved by the model. Second, annotators must compare sentences, but sentences contain partial information. For example, if Person is was born in Chicago in 1968 was in the generated text and Person was born in Chicago was in the reference text, this would count as the generation having information not in the reference. Annotators were very precise in sticking to the requested standard that the entire sentence should be factual to count as fully factual, which is reflected by annotators marking partial 9 Note that search hits from the Wikipedia domain are removed from web search results. hyman is best known for her work on the classification of invertebrates. she was the author of a six-volume set of reference books titled the invertebrate treatise, which was published by mcgraw-hill in the united states and in germany. she also wrote a series of laboratory manuals for the teaching of zoology classes nationwide. hyman's work has had a lasting influence on scientific thinking about a number of animal groups, and the only works that can be compared with hers are of composite authorship. factuality as not factual. Our stringent standard for factuality produces a clearer understanding of hallucinations at the sentence-level. In summary, our investigation suggests two explanations for the low coverage reported by human annotators: lack of information in the web evidence and difficulty assessing whether two sentences contain the same core knowledge. One major challenge of accurate Wikipedia article generation is when information is not available on the web or not easily retrieved. For example, information could simply not exist on the internet. Writing a Wikipedia biography about any randomly chosen person on the street would likely manifest this scenario. Other situations could include having a large number of search results returned but difficulty identifying which are relevant, having too few search results to write a good biographic article, or even having only noise returned in the search results. We discuss these challenges and possible mitigations in this section. The Evidence Gap. We compare the results on our evaluation set about women with those on the WikiSum test set. Compared to WikiSum, the un-igram overlap of the web hits with the biographical article is substantially lower for our evaluation dataset (see Table 2). As shown in Table 5, across the board, the quality of generated biographies is higher for the WikiSum Test set. This is especially prominent for Women in Asia and Africa, which are more than 2.5 ROUGE-L worse on average. Reducing the Dependency on Retrieval. One challenge is that there is a disconnect between the training dataset, where retrieval information is readily available, and the women-focused evaluation dataset, where retrieval information is noisy or missing. We investigate the potential of a straightforward strategy to mitigate differences in training data: that of training on biographical articles with less reliable web evidence. We mimic this by finetuning our model on a subset of our evalu-8568 Model WikiSum Test Women Scientists Women in Asia Women in Africa BART Pretraining 19.0 17.4 18.2 16.7 16.4 + Retrieval 21.4 18.8 19.3 17.9 17.1 + Caching 21.8 19.3 19.7 18.4 17.3 Table 5: ROUGE-L Performance broken down by sub-categories. ation dataset, and then testing on Women in Asia and Africa, the two categories that perform most poorly. As shown in Table 6, finetuning statistically significantly improves performance, though the improvement is not large (+0.5 ROUGE-L). Another phenomenon that arises with noisy web evidence is that retrieving more is not necessarily better. Perhaps only one website has really relevant information. In the retrieval module, all available web documents are encoded at the sentence level, and the model can select sentences across all documents. We next explore an approach where the model first scores documents, then selects sentences from the most relevant document. We found this had very similar performance, and thus conclude that the challenge of identifying relevant documents and then sentences is probably similar in difficulty to identifying relevant sentences directly. We developed a novel retrieval and cache-augmented generative model to generate long-form biographies based on evidence from the web. Experimental evidence reveals that an enriched query including occupations, caching, and backpropagation through the retrieval module contributes to improved performance. We investigate the dependency on high-quality web evidence, which manifests strongly in our constructed evaluation dataset of biographies about women. We discuss this challenge and possible mitigations. We thank the anonymous reviewers for their feedback. We gratefully acknowledge the support of the French National Research Agency and of Facebook AI Research Paris (for Claire Gardent; award ANR-20-CHIA-0003, XNLG "Multi-lingual, Multi-Source Text Generation"). We thank Adina Williams, Emily Dinan, Ledell Wu, and Aleksandra Piktus for thoughtful discussions and feedback on this entire effort, as well as previous collaborations that influenced this work. We thank Sebastian Riedel, Douwe Kiela, Mona Diab, and Michael White for their suggestions to improve this work. We thank Mojtaba Komeili for developing the web query service we used to create the evaluation dataset. Finally, we thank all of the editors of Wikipiedia, particularly those in the Women in Red Project, for their hard work and dedication to creating, moderating, editing, and all that is necessary to keep Wikipedia running. We encourage readers to donate to Wikipedia to support this public project. In this section, we discuss several known limitations and ethical considerations of our work. We do not recommend any kind of text generation technology to be deployed on Wikipedia given this is an active area of research. Biographies, whether written as books or available online, reflect societal bias. While many Wikipedia editors rely on web-based references to create their articles, and we follow the same strategy in this work, relying on the web is flawed. The prominent reason is that the internet is full of bias in it of itself. For example, Donna Strickland, who received a Nobel Prize, did not have a Wikipedia article 10 10 https://wikimediafoundation.org/news/ 2018/10/04/donna-strickland-wikipedia/#:~:text=Donna%20Strickland%20is%20an% 8569 as there was not sufficient content about her on the web as a basis for her article. Thus, it is important to recognize that the availability of references is problematic, affecting the downstream ability to write accurate, comprehensive biographies. Further, information on the web can be contradictory, information can be affected by the passage of time, and not information on the web is necessarily factually correct. Our proposed modeling mechanism does not have a way to explicitly recognize or correct for these challenges, which also plagues text generation generally. Our work focuses on text generation in English only, which limits inclusivity purely on the basis of language. This is challenging as the content of the internet and Wikipedia itself is different in various languages. For example, articles about people from Germany may be more likely to be located on the German version of Wikipedia. Another factor is that the content of the references may be written in another language, and then used by a bilingual individual to write an article in English about that subject. This is often the case for many biographical subjects who may be more well known in a non-English speaking area. There are a very large number of marginalized groups in the world and numerous important intersectional aspects to consider. When discussing identity, a wide variety of factors and personal views influence individuals when thinking about how they describe themselves. Our evaluation dataset focuses on women alone, which leaves out many groups, including non-binary people. Further, Wikipedia may not reflect the up-to-date information  names and gender are both mutable, for example  and Wikipedia articles do not ask each subject to self-report their gender. Finally, we note that by grouping people into hard categories, there can potentially be harm  such as limiting people from opportunities because of their gender or race. However, we strongly believe that it is important to recognize bias in its various forms as it exists, particularly in popular, default online sources of information such as Wikipedia. 20optical,of%20a%20Sloan%20Research%20Fellowship. In this work, we focus on bias manifesting as unequal prevalence and length of biographical content on Wikipedia, focusing specifically on different intersectional groups of women. However, bias manifests in a number of other ways. Studies have indicated that the words used in biographies about women compared to biographies about men (Di-nan et al., 2019) also differs, and is reflective of gendered terminology. For example, many articles about women are actually written with a lot of information about men, such as their husband's careers, and articles about actresses describe more often their physical appearance. This is also a manifestation of bias, and we do not present any focused modeling techniques to address this type of bias explicitly. In the modern internet, a large number of events are recorded for the public record. These include events that people may personally prefer to forget, often termed right to be forgotten 11 . Automatically generating biographies about individuals may collate such information in an easily accessible public place, which can conflict with this personal right. This has a complex but important interaction with marginalized groups. For example, many celebrities who are women, transgender, or a part of another marginalized group are far more likely to have news articles written about intimate personal details such as plastic surgeries. Thus, it is important to consider the interaction of biographical data with individual privacy. This is a larger challenge of biographical information generally.
abstain objective method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain objective abstain result method abstain result abstain objective method method other other abstain method objective method method abstain other other other other other other other other other other other other other other other other method abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain method abstain abstain method abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain abstain abstain other other abstain method method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective abstain objective abstain other other other other other other other abstain method abstain method abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain