<p>Wikipedia is a vanishingly small % of the internet. According to the information in the screenshot you posted, Wikipedia comprises less than 0.2% of the Common Crawl. A random sample of Common Crawl is extremely unlikely to have a meaningful amount of Wikipedia text.</p>
<p>Also, even if a Common Crawl segment does contain Wikipedia, it's still good to include it as a separate component. High quality factual information should be upsampled to achieve the best performance, as the repeated exposure to facts increases a LLM's ability to answer those factual questions correctly.</p>

CC BY-SA 4.0
2023-11-07T02:28:02.343Z
2023-11-07T02:28:02.343Z
Stella Biderman
<p>Some LLMs are trained on both CommonCrawl and Wikipedia/StackExchange. Why? Does CommonCrawl already contain Wikipedia/StackExchange?</p>
<p>E.g., from the <a href="https://arxiv.org/pdf/2302.13971.pdf" rel="nofollow noreferrer">LLaMa 1 paper</a>:</p>
<p><a href="https://i.sstatic.net/pPFVa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pPFVa.png" alt="enter image description here" /></a></p>
<p>and from <a href="https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T" rel="nofollow noreferrer">https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T</a>:</p>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>Dataset</th>
<th>Token Count</th>
</tr>
</thead>
<tbody>
<tr>
<td>Commoncrawl</td>
<td>878 Billion</td>
</tr>
<tr>
<td>C4</td>
<td>175 Billion</td>
</tr>
<tr>
<td>GitHub</td>
<td>59 Billion</td>
</tr>
<tr>
<td>Books</td>
<td>26 Billion</td>
</tr>
<tr>
<td>ArXiv</td>
<td>28 Billion</td>
</tr>
<tr>
<td>Wikipedia</td>
<td>24 Billion</td>
</tr>
<tr>
<td>StackExchange</td>
<td>20 Billion</td>
</tr>
<tr>
<td>Total</td>
<td>1.2 Trillion</td>
</tr>
</tbody>
</table>
</div>
<p>Looking at <a href="https://commoncrawl.github.io/cc-crawl-statistics/plots/domains" rel="nofollow noreferrer">https://commoncrawl.github.io/cc-crawl-statistics/plots/domains</a>, it seems that CommonCrawl includes the Wikipedia and StackExchange domains. But maybe it's incomplete?</p>
<p><a href="https://i.sstatic.net/O9A8t.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/O9A8t.png" alt="enter image description here" /></a></p>
<p><a href="https://i.sstatic.net/qDE2G.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qDE2G.png" alt="enter image description here" /></a></p>

<large-language-models><training-datasets>
Why are some LLMs trained on both CommonCrawl and Wikipedia/StackExchange?
Franck Dernoncourt
# Why are some LLMs trained on both CommonCrawl and Wikipedia/StackExchange?

(Asked by: username_0 on 2023-11-02)

Some LLMs are trained on both CommonCrawl and Wikipedia/StackExchange. Why? Does CommonCrawl already contain Wikipedia/StackExchange?

E.g., from the [LLaMa 1 paper](https://arxiv.org/pdf/2302.13971.pdf):

[![enter image description here](https://i.sstatic.net/pPFVa.png)](https://i.sstatic.net/pPFVa.png)

and from <https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T>:

| Dataset | Token Count |
| --- | --- |
| Commoncrawl | 878 Billion |
| C4 | 175 Billion |
| GitHub | 59 Billion |
| Books | 26 Billion |
| ArXiv | 28 Billion |
| Wikipedia | 24 Billion |
| StackExchange | 20 Billion |
| Total | 1.2 Trillion |

Looking at <https://commoncrawl.github.io/cc-crawl-statistics/plots/domains>, it seems that CommonCrawl includes the Wikipedia and StackExchange domains. But maybe it's incomplete?

[![enter image description here](https://i.sstatic.net/O9A8t.png)](https://i.sstatic.net/O9A8t.png)

[![enter image description here](https://i.sstatic.net/qDE2G.png)](https://i.sstatic.net/qDE2G.png)

**Tags:** large-language-models, training-datasets


---

## 1 answers

### Answer by username_1 on 2023-11-07. Score: 3

Wikipedia is a vanishingly small % of the internet. According to the information in the screenshot you posted, Wikipedia comprises less than 0.2% of the Common Crawl. A random sample of Common Crawl is extremely unlikely to have a meaningful amount of Wikipedia text.

Also, even if a Common Crawl segment does contain Wikipedia, it's still good to include it as a separate component. High quality factual information should be upsampled to achieve the best performance, as the repeated exposure to facts increases a LLM's ability to answer those factual questions correctly.


---

