As 
text
As 
https://cdn-avatars.huggingface.co/v1/production/uploads/66b05ca6e7c57eac7cafbbc4/f-BRRaSr0QLq3nHlLqD3o.png
https://huggingface.co/xet-team
xet-team
org
resource
https://huggingface.co/xet-team
 infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓
text
 infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓
new_line
new_line
IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.
text
IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.
new_line
new_line
When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?
text
When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?
new_line
new_line
If we can detect and reuse them, we skip them as well saving time and bandwidth for AI builders. More on how that works here:  
text
If we can detect and reuse them, we skip them as well saving time and bandwidth for AI builders. More on how that works here:  
new_line
🔗 
text
🔗 
https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation
https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation
link
new_line
new_line
Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:
text
Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:
new_line
new_line
- Nodes = repositories
text
- Nodes = repositories
new_line
- Edges = shared chunks
text
- Edges = shared chunks
new_line
- Edge thickness = how much they overlap
text
- Edge thickness = how much they overlap
new_line
new_line
https://huggingface.co/spaces/xet-team/repo-graph
xet-team/repo-graph
space
resource
https://huggingface.co/spaces/xet-team/repo-graph
text
new_line
new_line
Come find the many BERT islands. Or see how datasets relate in practice, not just in theory. See how libraries or tasks can tie repositories together. You can play around with node size using storage/likes/downloads too. 
text
Come find the many BERT islands. Or see how datasets relate in practice, not just in theory. See how libraries or tasks can tie repositories together. You can play around with node size using storage/likes/downloads too. 
new_line
new_line
The result is a super fun visualization from 
text
The result is a super fun visualization from 
@saba9
mention
saba9
  and 
text
  and 
@znation
mention
znation
 that I’ve already lost way too much time to. I'm excited to see how the networks grow as we add more repositories!
text
 that I’ve already lost way too much time to. I'm excited to see how the networks grow as we add more repositories!
As https://huggingface.co/xet-team infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓

IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.

When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?

If we can detect and reuse them, we skip them as well saving time and bandwidth for AI builders. More on how that works here:  
🔗 https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation

Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:

- Nodes = repositories
- Edges = shared chunks
- Edge thickness = how much they overlap

https://huggingface.co/spaces/xet-team/repo-graph 

Come find the many BERT islands. Or see how datasets relate in practice, not just in theory. See how libraries or tasks can tie repositories together. You can play around with node size using storage/likes/downloads too. 

The result is a super fun visualization from @saba9  and @znation that I’ve already lost way too much time to. I'm excited to see how the networks grow as we add more repositories!
👀
John6666 lunarflu nicoboss
🚀
lunarflu severo nicoboss
🔥
lunarflu nicoboss
❤️
nicoboss