We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasetsfor example removing from C4 a single 61 word English sentence that is repeated over 60 , 000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer training steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. Code for deduplication is released at https://github.com/google-research/ deduplicate-text-datasets . A key factor behind the recent progress in natural language processing is the development of large-scale text corpora used to train increasingly large language models. These datasets have grown from single gigabytes to as much as a terabyte over the past few years (Chelba et al., 2013; Xue et al., 2020; Graff et al., 2003; Brown et al., 2020). Because it is so expensive to perform manual review and curation on massive datasets, they tend to suffer in quality compared to their smaller predecessors. This has implications far beyond metrics like perplexity and validation loss, as learned models re-flect the biases present in their training data (Ben-der et al., 2021; Wallace et al., 2019; Sheng et al., 2020). Quantitatively and qualitatively understanding these datasets is therefore a research challenge in its own right (Dodge et al., 2021a). We show that one particular source of bias, duplicated training examples, is pervasive: all four common NLP datasets we studied contained duplicates. Additionally, all four corresponding validation sets contained text duplicated in the training set. While naive deduplication is straightforward (and the datasets we consider already perform some naive form of deduplication), performing thorough deduplication at scale is both computationally challenging and requires sophisticated techniques. We propose two scalable techniques to detect and remove duplicated training data. Exact substring matching identifies verbatim strings that are repeated. This allows us to identify cases where only part of a training example is duplicated (4.1). Approximate full document matching uses hash-based techniques (Broder, 1997) to identify pairs of documents with high n -gram overlap (4.2). 1. Over 1% of tokens emitted unprompted from a model trained on standard datasets (e.g., C4) are part of a memorized sequence (See 6.2) even though the 1.5 billion parameter model is much smaller than the 350GB dataset it was trained on. By deduplicating the training dataset we reduce the rate of emitting memorized training data by a factor of 10  . 2. Train-test overlap is common in non-deduplicated datasets. For example, we find a 61-word sequence 1 in C4 (Raffel et al., 2020) that is repeated 61 , 036 times verbatim in the training dataset and 61 times in the validation set ( 0 . 02% of the samples in each dataset). 1 by combining fantastic ideas, interesting arrangements, and follow the current trends in the field of that make you more inspired and give artistic touches. We'd be honored if you can apply some or all of these design in your wedding. believe me, brilliant ideas would be perfect if it can be applied in real and make the people around you amazed! 1 8424 This train-test set overlap not only causes researchers to over-estimate model accuracy, but also biases model selection towards models and hyperparameters that intentionally overfit their training datasets. 3. Training models on deduplicated datasets is more efficient. Processing a dataset with our framework requires a CPU-only linear-time algorithm. And so because these datasets are up to 19% smaller, even including the deduplication runtime itself, training on deduplicated datasets directly reduces the training cost in terms of time, dollar, and the environment (Bender et al., 2021; Strubell et al., 2019; Patterson et al., 2021). 4. Deduplicating training data does not hurt perplexity: models trained on deduplicated datasets have no worse perplexity compared to baseline models trained on the original datasets. In some cases deduplication reduces perplexity by up to 10% . Further, because recent LMs are typically limited to training for just a few epochs (Radford et al., 2019; Raffel et al., 2020), by training on higher quality data the models can reach higher accuracy faster. To summarize, data duplication offers significant advantages and no observed disadvantages. In the remainder of this paper we present our text deduplication framework in 4, and study the extent of duplicate content in common NLP datasets (e.g., C4, Wiki-40B, and LM1B) in 5. We then examine the impact of deduplication on test perplexity (6.1) and on the frequency of emitting memorized content (6.2). Finally, we analyze to what extent perplexity on existing, released models are skewed as a result of overlap between the train and test/validation splits (6.3). Large language model datasets. While we believe our results are independent of model architecture, we perform our analysis on Transformer-based decoder-only language models (Vaswani et al., 2017) trained for open-ended text generation. These current state-of-the-art models are trained on internet text. For example, the GPT-2 family of models Radford et al. (2019) is trained on Web-Text, a dataset of web documents highly ranked on Reddithowever this dataset was not made available publicly. A common dataset starting point is CommonCrawl, an index of public webpages. Among the models trained on CommonCrawl include GPT-3 (Brown et al., 2020) with the addition of book datasets, GROVER (Zellers et al., 2019) on a restricted subset filtered to news domains called RealNews, and T5 (Raffel et al., 2020) on a cleaned version of common crawl called C4. Other models are trained on more curated Internet sourcesfor example Guo et al. (2020) used high quality processed Wikipedia text from 40 different languages to train monolingual 141.4M parameter language models. Non-English models necessarily use different datasets; Zeng et al. (2021) for instance introduced PANGU , a family of models with up to 200B parameters that were trained on a non-public corpus of cleaned and filtered Chinese-language documents from CommonCrawl and other sources. Since many of these datasets are not public, we deduplicate three that are: Wiki-40B, C4, and RealNewsas well as the One Billion Word Language Model Benchmark (Chelba et al., 2013), a smaller dataset commonly used for evaluation. Contamination of downstream tasks. When models are trained on datasets constructed by crawling the Internet, it is possible the model will train on the test set of downstream target tasks. For example, Radford et al. (2019, 4) performed a posthoc analysis to identify 8-gram overlaps between GPT-2's training set and datasets used for evaluation, and Dodge et al. (2021b) analyzed C4 and found that up to 14.4% of test examples for various standard tasks were found verbatim (normalizing for capitalization and punctuation) in the dataset. A more proactive approach removes contaminated data. Trinh and Le (2018, Appendix B) removed documents from their CommonCrawl-based train set that overlapped substantially with the commonsense reasoning used for evaluation. And GPT-3 (Brown et al., 2020, 5) did the reverse and removed downstream evaluation examples from their training data by conservatively filtering out any train set examples with a 13-gram overlap with any evaluation example. Up to 90% of tasks were flagged as potentially contaminated. In our research, we do not focus on the impact of duplicate text in pretrained models on downstream benchmark tasks; instead we address how duplicate text in the LM training and validation sets impacts model perplexity and the extent to which generated text included memorized content. Memorizing training data. The privacy risks of data memorization, for example the ability to extract sensitive data such as valid phone numbers and IRC usernames, are highlighted by Carlini et al. (2020). While their paper finds 604 samples that GPT-2 emitted from its training set, we show that over 1% of the data most models emit is memorized training data. In computer vision, memorization of training data has been studied from various angles for both discriminative and generative models (e.g. Arpit et al., 2017; Webster et al., 2019; Feldman and Zhang, 2020; Stephenson et al., 2021; Teter-wak et al., 2021). Duplicate text in training data. The Book Corpus (Zhu et al., 2015), which was used to train popular models such as BERT, has a substantial amount of exact-duplicate documents according to Bandy and Vincent (2021). Allamanis (2019) shows that duplicate examples in code datasets cause worsened performance on code understanding tasks. We analyze the presence of duplicate text in four datasets of varying sizes that have been used for training natural language generation systems, producing general-purpose pre-trained models, and for language model benchmarking. While this paper restricts itself to English datasets, we expect that non-English datasets suffer from similar issues and could likewise benefit from de-duplication. Wikipedia (Wiki-40B) consists of multi-lingual cleaned Wikipedia text (Guo et al., 2020). We take the English portion, which contains 2.9M Wikipedia pages with an average length of 768 BPE tokens. The dataset creators do not indicate any deduplication was performed aside from removing redirect-pages (e.g., sunflower to Helianthus). One-Billion Word benchmark (LM1B) contains 30M sentences of news commentary (Chelba et al., 2013). Unlike the other datasets we analyze, LM1B's examples are one sentence long rather than multi-sentence documents. The average example length is 32 BPE tokens. While this dataset is extremely standard for benchmarking language models, Radford et al. (2019, Sec 4) note it has 13.2% overlap of the test set with the train set. Colossal Cleaned Common Crawl (C4) is made up of 360M web documents, with an average length of 486 BPE tokens (Raffel et al., 2020). C4 was introduced as a pre-training dataset for T5, a set of encoder-decoder models which have been widely used in fine-tuned downstream tasks. The dataset was previously deduplicated in a more sophisticated process than the prior two datasets. Each paragraph was hashed and paragraphs resulting in hash collisions were removed. This was followed by a pass that removed placeholder text, code, and prohibited words. See Dodge et al. (2021a) for a detailed breakdown of the source text in C4. RealNews is a subset of the Common Crawl consisting of articles from news domains (Zellers et al., 2019). It contains 31M documents with average length 793 BPE tokens. RealNews was deduplicated by inserting a hash of the first 100 characters of each document into a bloom filter (Bloom, 1970) and then excluding any document which resulted in a hash collision. Like C4, examples with duplicate URLs were excluded. The simplest technique to find duplicate examples would be to perform exact string matching between all example pairs, but as we will show, this is insufficient. We introduce two complementary methods for performing deduplication. First, using a suffix array (Manber and Myers, 1993), we remove duplicate substrings from the dataset if they occur verbatim in more than one example. Second, we use MinHash (Broder, 1997), an efficient algorithm for estimating the n -gram similarity between all pairs of examples in a corpus, to remove entire examples from the dataset if they have high n -gram overlap with any other example. We consider a dataset D = { x i } Ni =1 as a collection of examples x i . Each of these examples is itself a sequence of tokens : x i = (cid:2) x 1 i , x 2 i ,    , x s i i (cid:3) . Due to the diversity of possibilities in human language, it is rare for the same idea to be expressed identically in multiple documents unless one expression is derived from the other, or both are quoting from a shared source. This observation motivates deduplicating exact substrings. We call our approach EXACTSUBSTR . When two examples x i and x j share a sufficiently long substring (that is, a substring for which x a..a + k i = x b..b + k j ), that substring is removed from one of them. Based on statistical analyses (B), we select k = 50 tokens as the minimum matching substring length. This exact-substring-matching criterion, while conceptually simple, is computationally prohibitive with naive (quadratic) all-pair matching. To improve the efficiency, we concatenate all the examples of the entire dataset D into a giant sequence S , and construct a Suffix Array A of S . A suffix array (Manber and Myers, 1993) is a representation of a suffix tree (Weiner, 1973) that can be constructed in linear time in (cid:107)S(cid:107) (Krkkinen and Sanders, 2003) and enables efficient computation of many substring queries; in particular, they allow us to identify duplicated training examples in linear time. Suffix arrays have the advantage over suffix trees in that they are 10100  more memory efficient (Manber and Myers, 1993), requiring just 8 bytes per input token, though they are asymptotically less efficient for some query types. They have been used widely in NLP, such as for efficient TF-IDF computation (Yamamoto and Church, 2001) and document clustering (Chim and Deng, 2007). The suffix array A for a sequence S is a lexicographically-ordered list of all suffixes contained in the sequence. Formally, A ( S ) = arg sort all_suffixes ( S ) For example, the suffixes of the sequence banana are (banana, anana, nana ana, na, a) and so the suffix array is the sequence (6 4 2 1 5 3). In practice, we construct S from the bytes of the BPE tokenization of the text (6). After constructing A , it is straightforward to identify duplicated training examples. Suppose that the sequence s was repeated exactly twice in the training dataset S at positions i and j , that is, S i..i + | s | = S j..j + | s | . Then the indices i, j will occur adjacent to each other in the suffix array A . Finding all repeated sequences is thus a matter of linearly scanning the suffix array from beginning to end and looking for sequences A i , A i +1 that share a common prefix of at least some threshold length. Any satisfying sequences are recorded. This algorithm is embarrassingly parallel, and so we can efficiently process the dataset. Based on experimentation (Appendix B), we choose a threshold length of 50 BPE tokens for all experiments. We also perform approximate deduplication based on matching entire examples. This method, which we call NEARDUP , is a good complement to the exact substring matching, especially for web crawl text, as it handles the very common case of documents being identical except for interspersed templated fields (such as the last row of Table 1). MinHash (Broder, 1997) is an approximate matching algorithm widely used in large-scale deduplication tasks (Versley and Panchenko, 2012; Gabriel et al., 2018; Gyawali et al., 2020), including to deduplicate the training set for a large Chinese-language LM (Zeng et al., 2021). Given two documents x i and x j , the main idea is to represent each document by its respective set of n -grams d i and d j . We can then use hash functions to approximate the Jaccard Index (Jaccard, 1912): Jaccard( d i , d j ) = | d i  d j | / | d i  d j | If the Jaccard Index between d i and d j is sufficiently high, it is likely that documents are approximate matches of each other. To efficiently approximate the Jaccard index, MinHash constructs document signatures by sorting each of the n -grams via a hash function, and then keeping only the k smallest hashed n -grams. There are multiple ways to construct estimators of the Jaccard index from these kinds of signatures (Cohen, 2016). In our implementation, we use 5-grams and a signature of size 9,000. The probability that two documents are considered a potential match is Pr( d i , d j | Jaccard( d i , d j ) = s i,j ) = 1  (1  s bi,j ) r where b = 20 and r = 450 are user-settable parameters to control the strength of the filter. See Appendix A for more details. For each pair of documents identified as a potential match, more computationally expensive similarity metrics can be employed as a subsequent filtering step. In particular, we identify two documents as duplicates if they are matched by the MinHash algorithm and their edit similarity is greater than 0.8. The edit similarity between token sequences x i and x j is defined as: EditSim( x i , x j ) = 1  EditDistance( x i , x j ) max( | x i | , | x j | ) To build clusters of similar documents, we construct a graph that has an edge between two documents if they are considered a match. Then, we 4 8427 Dataset Example Near-Duplicate Example Wiki-40B \n_START_ARTICLE_\nHum Award for Most Impactful Character \n_START_SECTION_\nWinners and nomi-nees\n_START_PARAGRAPH_\nIn the list below, winners are listed first in the colored row, followed by the other nominees. use the method introduced in  acki et al. (2018) to identify connected components. A breakdown of the computation needed is given in Appendix A. We deduplicate each of the four datasets with both of our two techniques. When text was duplicated across multiple data splits, we prioritized keeping a copy in the test or validation set and removing it from the train set. With NEARDUP , we found that the web-scrape datasets contain between 3.04% (on C4) to 13.63% (on RealNews) near duplicates (Table 2). Near-duplicate text is much less common in Wiki-40B, forming only 0.39% of the train set. 2 In C4, the majority (1.8M) of near-duplicate clusters consisted of just a single pair of examples that matched against each other, but there were 280 clusters with over 5,000 examples in them (Figure 1), including one cluster of size 250,933. On average with EXACTSUBSTR , we remove more total content than with NEARDUP (de-spite EXACTSUBSTR not removing any examples outright)for example removing 7 . 18% of the tokens in C4. The exception is LM1B, where EXACTSUBSTR removes 8  less data than NEARDUP . On investigation, we find this is due to the fact that LM1B documents are significantly shorter: 90% of all documents are under 50 tokens, and so are not even candidates for potential matches even if the entire sequence matched verbatim. We find that both NEARDUP and EXACTSUBSTR remove similar content 77% of the training examples that NEARDUP removes from C4 have at least one verbatim length50 match found by EXACTSUBSTR . While the authors of both RealNews and C4 explicitly attempted deduplication during dataset construction, the methods were insufficient to capture the more subtle types of duplicate text commonly found on the internet. In C4 and Wiki-40B, we qualitatively observe that much of the text identified as near-duplicated is computer-generated. The text is identical except for the names of places, businesses, products, dates, and so on. Because these examples frequently differ by just a few words at a time, deduplication strategies relying on exact string matching would fail to identify a match. Example duplicate pairs from each dataset can be found in Table 1 (more examples in the Appendix). For RealNews and LM1B, derived from news sites, we observe that many near-duplicates occur because the same news article appears on multiple news sites with slightly different formatting. For example, in LM1B, there is one example that starts  MINEOLA , N.Y. New York officials say [...] and another that starts  ( AP ) New York officials say [...]. The two examples are otherwise identical. Both deduplication methods identify overlap between the train set and the validation set (Table 2). For example, 4.6% of the C4 validation set and 14.4% of the RealNews validation set examples had an approximate duplicate in their respective training sets. Such duplication is problematic since it could cause evaluation metrics to be unfairly in-flated for models that are better at memorizing their train sets. We evaluate the effect of this leakage on publicly released models in Section 6.3. . We trained 1.5B parameter XL", decoder-only, Transformer-based language models similar to GPT-2, on C4-O RIGINAL , C4-N EARDUP , and C4-E XACTSUBSTR , respectively. We use the T5 codebase and model architecture from Raffel et al. (2020), and each model was trained for about two epochs on its respective dataset. To better understand the amount of variance in the perplexities of trained models, we also trained three different random seeds of the 110M parameter base" model for each of the above three datasetsfor a total of nine base-sized models. For all experiments, we used a Byte Pair Encoding (BPE) vocabulary trained on C4-N EARDUP 0 5 10 15 20 25 30 35 Perplexity C4 Original C4 Duplicates C4 Unique LM1B Wiki40B E v a l u a t i o n d a t a s e t Training data Original NearDup ExactSubstr Figure 2: Impact of deduplicating the training set on validation perplexity. We plot the results from T5 XL (see Appendix for base-sized model). For C4, we evaluate on C4 Original , the original validation set; C4 Unique , a subset of the validation set identified by NEARDUP as having zero matches across C4; and C4 Duplicates , a subset of the validation set identified by NEARDUP as having a match in the C4 train set. with a budget of 50K tokens, which resulted in a vocabulary the same size as GPT-2's. We trained with a maximum sequence length of 512 tokens (for longer documents, we randomly extracted subsequences of this length.) Further training details can be found in Appendix C. 6.1 Model Perplexity We computed the perplexity of our trained models on the validation sets of LM1B and Wiki-40B, and on subsets of the C4 validation set (Figure 2). For the base size, we observe that all models have similar perplexity on the original C4 validation set and on validation set examples that were identified as unique (no near-duplicate in either train or validation). However, both models trained on deduplicated data have significantly higher perplexity on validation set examples that have duplicates in the training set than the model trained on the original C4. EXACTSUBSTR -deduplicated results in higher perplexity than NEARDUP -deduplicated. These trends holds true for the XL sized model as well. While this may suggest EXACTSUBSTR duplication results in models least overfit on the train set, note that both of these techniques have used separate duplicate thresholds and a different choice of thresholds could change the results. When evaluating on the validation sets of LM1B and Wiki-40B, we found that models trained on NEARDUP -deduplicated C4 consistently achieved lowest perplexity (for LM1B eval with base models, see Appendix Figure 7). EXACTSUBSTR deduplication decreases perplexity of the XL model by almost 3 points perplexity on Wiki-40B which is 6 8429 Model 1 Epoch 2 Epochs XL-ORIGINAL 1.926% 1.571% XL-NEARDUP 0.189% 0.264% XL-EXACTSUBSTR 0.138% 0.168% Table 4: When generating 100k sequences with no prompting, over 1% of the tokens emitted from a model trained on the original dataset are part of a 50-token long sequence copied directly from the training dataset. This drops to 0 . 1% for the deduplicated datasets. much larger than the variation of about 1 point perplexity we observed in the base models. This is despite seeing fewer tokens of training data overall. Lastly, we note all our XL models achieved <35 perplexity on LM1B, which is less than the 42.16 perplexity reported for the 1.5B GPT-2 using a vocabulary the same size as ours. 6.2 Generated Text Data duplication has the effect of biasing the trained LM towards particular types of examples. This can contribute to a lower diversity of generations, and increased likelihood that the generated content is copied from the training data (Carlini et al., 2020). For our generation experiments, we use topk random sampling with k = 50 and experiment with prompted and unprompted generation. No prompt. We first evaluate memorization tendencies in the case where the model is asked to generate text without any prompt sequence. We generate 100,000 samples, each up to 512 tokens in length (examples provided in the Ap-pendix). For each generated token, we say the token is memorized if it is part of a 50-token substring that is exactly contained in the training data. On XL-ORIGINAL , over 1% of the generated tokens belong to memorized sub-sequences (see Table 4). This is  10  more memorization than XL-EXACTSUBSTR or XL-NEARDUP . Some example subsequences that were copied verbatim from the train set can be found in Table 9 in the Appendix. With prompting. In most real use cases, language model generation is controlled by providing a prompt for the model to continue. We experiment with four possible prompt sources: training examples identified by EXACTSUBSTR as having near-duplicates in the train set (train dup), training examples identified as unique (train unique), validation set examples with a near-duplicate in the train set (valid in train), and validation set ex-0.0 0.1 0.2 0.3 0.4 Fraction of LM continuations matching true continuation train dup train unique valid in train valid unique P r o m p t s o u r c e Training data Original NearDup ExactSubstr Figure 3: The proportion of generations which have edit similarity above 0.8 with the groundtruth continuation when using the LM to generate continuations for 32-token prompts identified by NEARDUP as either duplicated or unique. Model Dataset Orig Dups Unique Transformer-XL LM1B 21.77 10.11 23.58 GROVER-Base RealNews 15.44 13.77 15.73 GROVER-XL RealNews 9.15 7.68 9.45 Table 5: For each model, the perplexity of the official validation set ( Orig ), valid set examples which were identified by NEARDUP as matches of train set examples ( Dups ), and valid set examples identified by NEARDUP as unique ( Unique ). Due to the size of the RealNews validation set, we evaluated on only the first 25k examples meeting each condition. amples identified as unique across all splits (valid unique). We select the first 32 tokens of each example as the prompt, which means we can evaluate the fraction of generations which are near-duplicates with the ground-truth continuation for the prompt (Figure 3). When the prompt comes from duplicate examples in the train set, XL-ORIGINAL reproduces the groundtruth continuation over 40% of the time. XL-EXACTSUBSTR and XL-NEARDUP still copy the groundtruth more often when the prompt comes from a duplicate example than when the prompt comes from a unique example, suggesting that more stringent deduplication may be necessary to remove memorization tendencies entirely. 6.3 Impact on Existing Models Train-test leakage does not just impact models trained on C4. Table 5 shows that the presence of near-duplicates of the evaluation set in the train set has a significant impact on model perplexity for two standard models: Transformer-XL (Dai et al., 2019), which was trained on LM1B, and GROVER (Zellers et al., 2019), which was trained on RealNews. For Transformer XL, the perplexity 7 8430 halves on examples identified as near-duplicates. For GROVER, the difference, though not quite as stark, is present in both model sizes considered. Existing models also suffer from the problem of generating text from their train sets. We find that 1 . 38% of the tokens in the official release of 25k GROVER-Mega outputs 3 are part of verbatim matches in RealNews of at least length 50 . Likewise, more than 5% of the tokens in ~200k sequences outputted by GPT-Neo 1.3B (Black et al., 2021) are part of a 50 token matches of its training data, the Pile (Gao et al., 2020). 7 Discussion The focus of this paper is on the datasets used to train language models. While recent work focused on documenting the potential harms that could arise from problematic datasets (Bender and Friedman, 2018; Gebru et al., 2020), less work has been done to quantitatively analyze properties of real language modelling datasets, like Dodge et al. (2021a) has done for C4. Our paper provides analysis on one particular axis, that of data duplication. Our experiments measured what could be quan-tified: the amount of duplicate content in common datasets, the effect of deduplication on trained model perplexity, and the reduction of memorized content in trained models through deduplication. We do not focus on the nature of the data being removed by deduplication or memorized by LMs. Privacy is an important subject for future work, as memorized training data has significant privacy consequences. By this, we mean the standard privacy definition that a model should not reveal anything particular to the specific dataset it was trained on, as opposed to another training dataset from a similar distribution (Shokri et al., 2017). 4 Training on standard datasets that have not yet been deduplicated results in models that are particularly sensitive to examples that happened to be repeated multiple times, and this has negative privacy implications. For instance, it could violate a person's expectations of privacy if their publicly available personal data appeared in a different, surprising context. Downstream applications of LMs, such 3 gs://grover-models/generation_examples/ generator=mega~dataset=p0.90.jsonl 4 Another interpretation of privacy focuses on the sensitivity of the data involved, when a model is trained on and able to reproduce personal identifiers or other forms of private data. Our definition is more expansive. memorized content like adverts for real products. We stress that in our experiments, we do not distinguish between undesired memorized text (such as phone numbers), innocuous memorized text (common phrases), and text we may want to be memorized (such as a quote by a public figure), and instead treat all instances of the LM generating text that closely matches the training set as problematic. While we qualitatively observed that much of the identified memorized content was relatively innocuous, a more systematic study of the risks associated with the detected memorization was beyond the scope of this work. We also do not investigate the negative consequences of deduplication. Some language tasks explicitly require memorization, like document retrieval or closed-book question answering. Also, text that gives attribution is often duplicated across documents, so removing duplicate substrings could correspond to removing just the attribution, which could result in models that learn the content without its attached attribution. Deduplication is also not sufficient to remove privacy-sensitive data like bank passwords and medical records which should never be used in training data (Brown et al., 2022). Ultimately, whether memorization is a desired property of a language model, or else risky and unwanted, depends both on the nature of the text that has been memorized and on the downstream applications of the trained model. However, since the trend has been towards creating datasets and models that are application-agnostic, we encourage researchers to think carefully about the limitations of the data they have collected and the how the model's intended usage constrains what should be part of the training set. Developing techniques to memorize or forget specific sequences depending on the end application is a promising research direction. We encourage future language model research to perform dataset deduplication, either by training on the deduplicated datasets we release, using the deduplication tools we release, or following our approach to deduplicate datasets with new tools. The exact technique used to perform deduplication is less important than performing stringent deduplication in the first place. On the whole, dedu-5 https://play.aidungeon.io/ 8 8431 plication does not harm, and sometimes improves, model perplexity, despite the fact that the deduplicated datasets are smaller and faster to train on. It is especially important that there are no duplicates between the training and testing sets, because overlap here explicitly encourages selecting models that memorize the training data. Lastly, deduplication helps to reduce some of the privacy concerns around LMs memorizing their training data. The developers of large language models typically attempt to create training data that reflects natural human communication, but current methods to collect and curate such datasets are fallible. There are multiple reasons some text ends up over-represented. For example, bot replies, auto-generated templates, and licenses are repeated for structural (e.g., legal, economical) reasons (as was also observed by Dodge et al. (2021a)). Additionally, common techniques for acquiring and cleaning data can result in an over-representation of particular subsets of world users, often those who are English-speaking and publishing in established forums. This effectively under-represents non-English speakers as well as groups whose communication mostly occurs outside of the public web. In this paper, we focus on the problem of over-representation of some types of text (struc-tural duplicates) but do not address the problem of under-representation of others. Additionally, while we discuss when memorized content might be desired and when it might not be desired, our analysis does not disambiguate these two cases. Work to disambiguate helpful from harmful memorization is tremendously complex and would require a different set of research methodologies than are presented in this work. We are grateful to the many researchers whose technical help, feedback, and discussions shaped this project: Jacob Austin, Samy Bengio, Olivier Bousquet, James Bradbury, Fernando Diaz, Mark Diaz, Noah Fiedel, Jonathan Frankle, David Grangier, Stefanie Karp, David Mimno, Gaurav Mishra, Michael Mozer, Sharan Narang, Alex Pas-sos, Adam Roberts, Hanie Sedghi, Jascha Sohl-dickstein, David So, Florian Tramer, and Yun William Yu. We are also grateful to the Google Brain women who have given us continuous support. Chris Callison-Burch and Daphne Ippolito's research is supported in part by the DARPA KAIROS Program (contract FA8750-19-2-1004), the DARPA LwLL Program (contract FA8750-19-2-0201), and the IARPA BETTER Program (con-tract 2019-19051600004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, IARPA, or the U.S. Government. Each of the authors on this paper significantly contributed to the final results. Katherine trained the models used in the paper, built and ran the eval and text generation pipelines, contributed significantly to writing, analysis, and project organization and management. Daphne ran the approximate matching data deduplication pipelines, extracted prompts and evaluation datasets, ran eval pipelines, and contributed significantly to planning, writing, and analysis. Andrew wrote the code to perform deduplication with approximate matching, helped evaluate energy expenditure, and helped with analysis. Chiyuan helped generate plots and contributed to project scoping, writing, and data analysis. Chris offered mentorship and guidance throughout the project and contributed to writing. Doug offered mentorship and guidance throughout the project and contributed to writing. Nicholas wrote the suffix array implementation, ran all EXACTSUBSTR deduplication experiments, contributed significantly to planning, writing, and analysis, as well as scoping the project.
result abstain objective result method other abstain abstain abstain abstain abstain result abstain method objective abstain method abstain objective method objective result method objective abstain objective objective objective objective abstain method abstain result other abstain other other other other other other method other other other other other other other method other other abstain other other other other abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method abstain abstain abstain method abstain abstain abstain method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain method abstain abstain method abstain abstain abstain method method method abstain abstain result result other result abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain objective abstain other abstain abstain abstain abstain abstain abstain abstain method abstain method other other other other abstain abstain abstain abstain abstain abstain abstain abstain