It's been a bit since I took a step back and looked at 
text
It's been a bit since I took a step back and looked at 
https://cdn-avatars.huggingface.co/v1/production/uploads/66b05ca6e7c57eac7cafbbc4/f-BRRaSr0QLq3nHlLqD3o.png
https://huggingface.co/xet-team
xet-team
org
resource
https://huggingface.co/xet-team
 progress to migrate Hugging Face from Git LFS to Xet, but every time I do it boggles the mind.
text
 progress to migrate Hugging Face from Git LFS to Xet, but every time I do it boggles the mind.
new_line
new_line
A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
text
A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
new_line
🤗 700,000 users/orgs
text
🤗 700,000 users/orgs
new_line
📈 350,000 repos
text
📈 350,000 repos
new_line
🚀 15PB
text
🚀 15PB
new_line
new_line
Meanwhile, our migrations have pushed throughput to numbers that are bonkers. In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time). 
text
Meanwhile, our migrations have pushed throughput to numbers that are bonkers. In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time). 
new_line
new_line
These are hard numbers to put into context, but let's try: 
text
These are hard numbers to put into context, but let's try: 
new_line
new_line
The latest run of the Common Crawl from 
text
The latest run of the Common Crawl from 
https://cdn-avatars.huggingface.co/v1/production/uploads/656fbeae7734a829bbd16252/RANGDo_aKsWYL2nKwkISh.png
https://huggingface.co/commoncrawl
commoncrawl
org
resource
https://huggingface.co/commoncrawl
 was 471 TB. 
text
 was 471 TB. 
new_line
new_line
We now have ~32 crawls stored in Xet. At peak upload speed we could move the latest crawl into Xet in about two hours. 
text
We now have ~32 crawls stored in Xet. At peak upload speed we could move the latest crawl into Xet in about two hours. 
new_line
new_line
We're moving to a new phase in the process, so stay tuned. 
text
We're moving to a new phase in the process, so stay tuned. 
new_line
new_line
This shift in gears means it's also time to roll up our sleeves and look at all the bytes we have and the value we're adding to the community. 
text
This shift in gears means it's also time to roll up our sleeves and look at all the bytes we have and the value we're adding to the community. 
new_line
new_line
I already have some homework from 
text
I already have some homework from 
@RichardErkhov
mention
RichardErkhov
  to look at the dedupe across their uploads, and I'll be doing the same for other early adopters, big 
text
  to look at the dedupe across their uploads, and I'll be doing the same for other early adopters, big 
models/datasets
text
models/datasets
, and frequent uploaders (looking at you 
text
, and frequent uploaders (looking at you 
@bartowski
mention
bartowski
 👀)
text
 👀)
new_line
new_line
Let me know if there's anything you're interested in; happy to dig in!
text
Let me know if there's anything you're interested in; happy to dig in!
It's been a bit since I took a step back and looked at https://huggingface.co/xet-team progress to migrate Hugging Face from Git LFS to Xet, but every time I do it boggles the mind.

A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
🤗 700,000 users/orgs
📈 350,000 repos
🚀 15PB

Meanwhile, our migrations have pushed throughput to numbers that are bonkers. In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time). 

These are hard numbers to put into context, but let's try: 

The latest run of the Common Crawl from https://huggingface.co/commoncrawl was 471 TB. 

We now have ~32 crawls stored in Xet. At peak upload speed we could move the latest crawl into Xet in about two hours. 

We're moving to a new phase in the process, so stay tuned. 

This shift in gears means it's also time to roll up our sleeves and look at all the bytes we have and the value we're adding to the community. 

I already have some homework from @RichardErkhov  to look at the dedupe across their uploads, and I'll be doing the same for other early adopters, big models/datasets, and frequent uploaders (looking at you @bartowski 👀)

Let me know if there's anything you're interested in; happy to dig in!
🚀
BrigitteTousi nicoboss assafvayner RichardErkhov rajatarya nyuuzyou bartowski ThijsL202 julien-c BuiDoan AtAndDev pcuenq John6666 victor Chroma111 michellehbn
🔥
rajatarya bartowski ljupco michellehbn
643ac5d2e2b979ae6144d68c
https://cdn-avatars.huggingface.co/v1/production/uploads/643ac5d2e2b979ae6144d68c/Z7PCNopn4cQeAYnVJDoqG.png
nyuuzyou
nyuuzyou
user
65d50e9ef9cbfa798c590004
https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg
Jared Sulzdorf
jsulz
user
630f3e4002ce39336c411048
https://cdn-avatars.huggingface.co/v1/production/uploads/630f3e4002ce39336c411048/H281TzWT-Uxo-Owk_HlXC.png
alkinun
AtAndDev
user
5dd96eb166059660ed1ee413
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg
Julien Chaumond
julien-c
user