Toward the end of last year, the Xet team provided an inside look into the foundations of how we plan to enable rapid experimentation and iteration for the AI builders on the Hub: 
text
Toward the end of last year, the Xet team provided an inside look into the foundations of how we plan to enable rapid experimentation and iteration for the AI builders on the Hub: 
https://huggingface.co/blog/from-files-to-chunks
https://huggingface.co/blog/from-files-to-chunks
link
new_line
new_line
But it turns out chunks aren't all you need! 
text
But it turns out chunks aren't all you need! 
new_line
new_line
Our goal is to bring:
text
Our goal is to bring:
new_line
🚀 Faster uploads
text
🚀 Faster uploads
new_line
⏬ Speedy downloads
text
⏬ Speedy downloads
new_line
💪 All without sacrificing your workflow 
text
💪 All without sacrificing your workflow 
new_line
new_line
To do that, we need the infrastructure and system and design to back it up. As we prepare to roll out the first Xet-backed repositories on the Hub, we wrote up a post explaining the nitty gritty details of the decisions that bring this to life 
text
To do that, we need the infrastructure and system and design to back it up. As we prepare to roll out the first Xet-backed repositories on the Hub, we wrote up a post explaining the nitty gritty details of the decisions that bring this to life 
https://huggingface.co/blog/from-chunks-to-blocks
https://huggingface.co/blog/from-chunks-to-blocks
link
new_line
new_line
Complete with an interactive visualization that shows the power of deduplication in action - taking a 191GB repo to ~97GB and shaving a few hours off upload speeds.  
text
Complete with an interactive visualization that shows the power of deduplication in action - taking a 191GB repo to ~97GB and shaving a few hours off upload speeds.  
new_line
new_line
The darker each block in the heatmap, the more we dedupe, the less we have to transfer. Clicking on a file's blocks shows all other files that share blocks. 
text
The darker each block in the heatmap, the more we dedupe, the less we have to transfer. Clicking on a file's blocks shows all other files that share blocks. 
new_line
new_line
Check it out and explore for yourself! 
text
Check it out and explore for yourself! 
https://huggingface.co/spaces/xet-team/quantization-dedup
xet-team/quantization-dedup
space
resource
https://huggingface.co/spaces/xet-team/quantization-dedup
Toward the end of last year, the Xet team provided an inside look into the foundations of how we plan to enable rapid experimentation and iteration for the AI builders on the Hub: https://huggingface.co/blog/from-files-to-chunks

But it turns out chunks aren't all you need! 

Our goal is to bring:
🚀 Faster uploads
⏬ Speedy downloads
💪 All without sacrificing your workflow 

To do that, we need the infrastructure and system and design to back it up. As we prepare to roll out the first Xet-backed repositories on the Hub, we wrote up a post explaining the nitty gritty details of the decisions that bring this to life https://huggingface.co/blog/from-chunks-to-blocks

Complete with an interactive visualization that shows the power of deduplication in action - taking a 191GB repo to ~97GB and shaving a few hours off upload speeds.  

The darker each block in the heatmap, the more we dedupe, the less we have to transfer. Clicking on a file's blocks shows all other files that share blocks. 

Check it out and explore for yourself! https://huggingface.co/spaces/xet-team/quantization-dedup
🚀
Taylor658 Jinkxop davidberenstein1957 John6666 victor oza75 raka-pratama dantezxcd
🤗
nyuuzyou adithyanhere John6666 dantezxcd
➕
davidberenstein1957
❤️
davidberenstein1957