Answers.Body: <p>Wikipedia is a vanishingly small % of the internet. According to the information in the screenshot you posted, Wikipedia comprises less than 0.2% of the Common Crawl. A random sample of Common Crawl is extremely unlikely to have a meaningful amount of Wikipedia text.</p>
  <p>Also, even if a Common Crawl segment does contain Wikipedia, it's still good to include it as a separate component. High quality factual information should be upsampled to achieve the best performance, as the repeated exposure to facts increases a LLM's ability to answer those factual questions correctly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-07T02:28:02.343Z
Answers.LastActivityDate: 2023-11-07T02:28:02.343Z
Answers.UserDisplayName: Stella Biderman
Body: <p>Some LLMs are trained on both CommonCrawl and Wikipedia/StackExchange. Why? Does CommonCrawl already contain Wikipedia/StackExchange?</p>
  <p>E.g., from the <a href="https://arxiv.org/pdf/2302.13971.pdf" rel="nofollow noreferrer">LLaMa 1 paper</a>:</p>
  <p><a href="https://i.sstatic.net/pPFVa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pPFVa.png" alt="enter image description here" /></a></p>
  <p>and from <a href="https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T" rel="nofollow noreferrer">https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T</a>:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Dataset</th>
  <th>Token Count</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Commoncrawl</td>
  <td>878 Billion</td>
  </tr>
  <tr>
  <td>C4</td>
  <td>175 Billion</td>
  </tr>
  <tr>
  <td>GitHub</td>
  <td>59 Billion</td>
  </tr>
  <tr>
  <td>Books</td>
  <td>26 Billion</td>
  </tr>
  <tr>
  <td>ArXiv</td>
  <td>28 Billion</td>
  </tr>
  <tr>
  <td>Wikipedia</td>
  <td>24 Billion</td>
  </tr>
  <tr>
  <td>StackExchange</td>
  <td>20 Billion</td>
  </tr>
  <tr>
  <td>Total</td>
  <td>1.2 Trillion</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Looking at <a href="https://commoncrawl.github.io/cc-crawl-statistics/plots/domains" rel="nofollow noreferrer">https://commoncrawl.github.io/cc-crawl-statistics/plots/domains</a>, it seems that CommonCrawl includes the Wikipedia and StackExchange domains. But maybe it's incomplete?</p>
  <p><a href="https://i.sstatic.net/O9A8t.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/O9A8t.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/qDE2G.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qDE2G.png" alt="enter image description here" /></a></p>
  
Tags: <large-language-models><training-datasets>
Title: Why are some LLMs trained on both CommonCrawl and Wikipedia/StackExchange?
UserDisplayName: Franck Dernoncourt
ThreadText: # Why are some LLMs trained on both CommonCrawl and Wikipedia/StackExchange?
  
  (Asked by: username_0 on 2023-11-02)
  
  Some LLMs are trained on both CommonCrawl and Wikipedia/StackExchange. Why? Does CommonCrawl already contain Wikipedia/StackExchange?
  
  E.g., from the [LLaMa 1 paper](https://arxiv.org/pdf/2302.13971.pdf):
  
  [![enter image description here](https://i.sstatic.net/pPFVa.png)](https://i.sstatic.net/pPFVa.png)
  
  and from <https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T>:
  
  | Dataset | Token Count |
  | --- | --- |
  | Commoncrawl | 878 Billion |
  | C4 | 175 Billion |
  | GitHub | 59 Billion |
  | Books | 26 Billion |
  | ArXiv | 28 Billion |
  | Wikipedia | 24 Billion |
  | StackExchange | 20 Billion |
  | Total | 1.2 Trillion |
  
  Looking at <https://commoncrawl.github.io/cc-crawl-statistics/plots/domains>, it seems that CommonCrawl includes the Wikipedia and StackExchange domains. But maybe it's incomplete?
  
  [![enter image description here](https://i.sstatic.net/O9A8t.png)](https://i.sstatic.net/O9A8t.png)
  
  [![enter image description here](https://i.sstatic.net/qDE2G.png)](https://i.sstatic.net/qDE2G.png)
  
  **Tags:** large-language-models, training-datasets
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-07. Score: 3
  
  Wikipedia is a vanishingly small % of the internet. According to the information in the screenshot you posted, Wikipedia comprises less than 0.2% of the Common Crawl. A random sample of Common Crawl is extremely unlikely to have a meaningful amount of Wikipedia text.
  
  Also, even if a Common Crawl segment does contain Wikipedia, it's still good to include it as a separate component. High quality factual information should be upsampled to achieve the best performance, as the repeated exposure to facts increases a LLM's ability to answer those factual questions correctly.
  
  
  ---
  
  