content.raw: As 
content.type: text
content.value: As 
content.image: https://cdn-avatars.huggingface.co/v1/production/uploads/66b05ca6e7c57eac7cafbbc4/f-BRRaSr0QLq3nHlLqD3o.png
content.raw: https://huggingface.co/xet-team
content.id: xet-team
content.type: org
content.type: resource
content.url: https://huggingface.co/xet-team
content.raw:  infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓
content.type: text
content.value:  infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓
content.type: new_line
content.type: new_line
content.raw: IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.
content.type: text
content.value: IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.
content.type: new_line
content.type: new_line
content.raw: When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?
content.type: text
content.value: When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?
content.type: new_line
content.type: new_line
content.raw: If we can detect and reuse them, we skip them as well saving time and bandwidth for AI builders. More on how that works here:  
content.type: text
content.value: If we can detect and reuse them, we skip them as well saving time and bandwidth for AI builders. More on how that works here:  
content.type: new_line
content.raw: 🔗 
content.type: text
content.value: 🔗 
content.href: https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation
content.raw: https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation
content.type: link
content.type: new_line
content.type: new_line
content.raw: Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:
content.type: text
content.value: Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:
content.type: new_line
content.type: new_line
content.raw: - Nodes = repositories
content.type: text
content.value: - Nodes = repositories
content.type: new_line
content.raw: - Edges = shared chunks
content.type: text
content.value: - Edges = shared chunks
content.type: new_line
content.raw: - Edge thickness = how much they overlap
content.type: text
content.value: - Edge thickness = how much they overlap
content.type: new_line
content.type: new_line
content.raw: https://huggingface.co/spaces/xet-team/repo-graph
content.id: xet-team/repo-graph
content.type: space
content.type: resource
content.url: https://huggingface.co/spaces/xet-team/repo-graph
content.type: text
content.type: new_line
content.type: new_line
content.raw: Come find the many BERT islands. Or see how datasets relate in practice, not just in theory. See how libraries or tasks can tie repositories together. You can play around with node size using storage/likes/downloads too. 
content.type: text
content.value: Come find the many BERT islands. Or see how datasets relate in practice, not just in theory. See how libraries or tasks can tie repositories together. You can play around with node size using storage/likes/downloads too. 
content.type: new_line
content.type: new_line
content.raw: The result is a super fun visualization from 
content.type: text
content.value: The result is a super fun visualization from 
content.raw: @saba9
content.type: mention
content.user: saba9
content.raw:   and 
content.type: text
content.value:   and 
content.raw: @znation
content.type: mention
content.user: znation
content.raw:  that I’ve already lost way too much time to. I'm excited to see how the networks grow as we add more repositories!
content.type: text
content.value:  that I’ve already lost way too much time to. I'm excited to see how the networks grow as we add more repositories!
rawContent: As https://huggingface.co/xet-team infrastructure begins backing hundreds of repositories on the Hugging Face Hub, we’re getting to put on our researcher hats and peer into the bytes. 👀 🤓
  
  IMO, one of the most interesting ideas Xet storage introduces is a globally shared store of data.
  
  When you upload a file through Xet, the contents are split into ~64KB chunks and deduplicated, but what if those same chunks already exist in another repo on the Hub?
  
  If we can detect and reuse them, we skip them as well saving time and bandwidth for AI builders. More on how that works here:  
  🔗 https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation
  
  Because of this, different repositories can share bytes we store. That opens up something cool - we can draw a graph of which repos actually share data at the chunk level, where:
  
  - Nodes = repositories
  - Edges = shared chunks
  - Edge thickness = how much they overlap
  
  https://huggingface.co/spaces/xet-team/repo-graph 
  
  Come find the many BERT islands. Or see how datasets relate in practice, not just in theory. See how libraries or tasks can tie repositories together. You can play around with node size using storage/likes/downloads too. 
  
  The result is a super fun visualization from @saba9  and @znation that I’ve already lost way too much time to. I'm excited to see how the networks grow as we add more repositories!
reactions.reaction: 👀
reactions.users: John6666 lunarflu nicoboss
reactions.reaction: 🚀
reactions.users: lunarflu severo nicoboss
reactions.reaction: 🔥
reactions.users: lunarflu nicoboss
reactions.reaction: ❤️
reactions.users: nicoboss