content.raw: Toward the end of last year, the Xet team provided an inside look into the foundations of how we plan to enable rapid experimentation and iteration for the AI builders on the Hub: 
content.type: text
content.value: Toward the end of last year, the Xet team provided an inside look into the foundations of how we plan to enable rapid experimentation and iteration for the AI builders on the Hub: 
content.href: https://huggingface.co/blog/from-files-to-chunks
content.raw: https://huggingface.co/blog/from-files-to-chunks
content.type: link
content.type: new_line
content.type: new_line
content.raw: But it turns out chunks aren't all you need! 
content.type: text
content.value: But it turns out chunks aren't all you need! 
content.type: new_line
content.type: new_line
content.raw: Our goal is to bring:
content.type: text
content.value: Our goal is to bring:
content.type: new_line
content.raw: 🚀 Faster uploads
content.type: text
content.value: 🚀 Faster uploads
content.type: new_line
content.raw: ⏬ Speedy downloads
content.type: text
content.value: ⏬ Speedy downloads
content.type: new_line
content.raw: 💪 All without sacrificing your workflow 
content.type: text
content.value: 💪 All without sacrificing your workflow 
content.type: new_line
content.type: new_line
content.raw: To do that, we need the infrastructure and system and design to back it up. As we prepare to roll out the first Xet-backed repositories on the Hub, we wrote up a post explaining the nitty gritty details of the decisions that bring this to life 
content.type: text
content.value: To do that, we need the infrastructure and system and design to back it up. As we prepare to roll out the first Xet-backed repositories on the Hub, we wrote up a post explaining the nitty gritty details of the decisions that bring this to life 
content.href: https://huggingface.co/blog/from-chunks-to-blocks
content.raw: https://huggingface.co/blog/from-chunks-to-blocks
content.type: link
content.type: new_line
content.type: new_line
content.raw: Complete with an interactive visualization that shows the power of deduplication in action - taking a 191GB repo to ~97GB and shaving a few hours off upload speeds.  
content.type: text
content.value: Complete with an interactive visualization that shows the power of deduplication in action - taking a 191GB repo to ~97GB and shaving a few hours off upload speeds.  
content.type: new_line
content.type: new_line
content.raw: The darker each block in the heatmap, the more we dedupe, the less we have to transfer. Clicking on a file's blocks shows all other files that share blocks. 
content.type: text
content.value: The darker each block in the heatmap, the more we dedupe, the less we have to transfer. Clicking on a file's blocks shows all other files that share blocks. 
content.type: new_line
content.type: new_line
content.raw: Check it out and explore for yourself! 
content.type: text
content.value: Check it out and explore for yourself! 
content.raw: https://huggingface.co/spaces/xet-team/quantization-dedup
content.id: xet-team/quantization-dedup
content.type: space
content.type: resource
content.url: https://huggingface.co/spaces/xet-team/quantization-dedup
rawContent: Toward the end of last year, the Xet team provided an inside look into the foundations of how we plan to enable rapid experimentation and iteration for the AI builders on the Hub: https://huggingface.co/blog/from-files-to-chunks
  
  But it turns out chunks aren't all you need! 
  
  Our goal is to bring:
  🚀 Faster uploads
  ⏬ Speedy downloads
  💪 All without sacrificing your workflow 
  
  To do that, we need the infrastructure and system and design to back it up. As we prepare to roll out the first Xet-backed repositories on the Hub, we wrote up a post explaining the nitty gritty details of the decisions that bring this to life https://huggingface.co/blog/from-chunks-to-blocks
  
  Complete with an interactive visualization that shows the power of deduplication in action - taking a 191GB repo to ~97GB and shaving a few hours off upload speeds.  
  
  The darker each block in the heatmap, the more we dedupe, the less we have to transfer. Clicking on a file's blocks shows all other files that share blocks. 
  
  Check it out and explore for yourself! https://huggingface.co/spaces/xet-team/quantization-dedup
reactions.reaction: 🚀
reactions.users: Taylor658 Jinkxop davidberenstein1957 John6666 victor oza75 raka-pratama dantezxcd
reactions.reaction: 🤗
reactions.users: nyuuzyou adithyanhere John6666 dantezxcd
reactions.reaction: ➕
reactions.users: davidberenstein1957
reactions.reaction: ❤️
reactions.users: davidberenstein1957