content.raw: What does it mean when models share the same bytes? 
content.type: text
content.value: What does it mean when models share the same bytes? 
content.type: new_line
content.type: new_line
content.raw: We've investigated some quants and have seen that a considerable portion of quantizations of the same model share the same bytes and can be deduplicated to save considerable upload time for quantizers on the Hub.
content.type: text
content.value: We've investigated some quants and have seen that a considerable portion of quantizations of the same model share the same bytes and can be deduplicated to save considerable upload time for quantizers on the Hub.
content.type: new_line
content.type: new_line
content.raw: This space where we crack open a repo from 
content.type: text
content.value: This space where we crack open a repo from 
content.raw: @bartowski
content.type: mention
content.user: bartowski
content.raw:  shows we can get significant dedupe 
content.type: text
content.value:  shows we can get significant dedupe 
content.raw: https://huggingface.co/spaces/xet-team/quantization-dedup
content.id: xet-team/quantization-dedup
content.type: space
content.type: resource
content.url: https://huggingface.co/spaces/xet-team/quantization-dedup
content.type: new_line
content.type: new_line
content.raw: You can get a sense of why by reading this write-up: 
content.type: text
content.value: You can get a sense of why by reading this write-up: 
content.href: https://github.com/bartowski1182/llm-knowledge/blob/main/quantization/quantization.md
content.raw: https://github.com/bartowski1182/llm-knowledge/blob/main/quantization/quantization.md
content.type: link
content.type: new_line
content.type: new_line
content.raw: But what about finetuned models?
content.type: text
content.value: But what about finetuned models?
content.type: new_line
content.type: new_line
content.raw: Since going into production the 
content.type: text
content.value: Since going into production the 
content.image: https://cdn-avatars.huggingface.co/v1/production/uploads/66b05ca6e7c57eac7cafbbc4/f-BRRaSr0QLq3nHlLqD3o.png
content.raw: https://huggingface.co/xet-team
content.id: xet-team
content.type: org
content.type: resource
content.url: https://huggingface.co/xet-team
content.raw:  has migrated hundreds of repositories on the Hub to our storage layer, including classic "pre-Hub" open-source models like 
content.type: text
content.value:  has migrated hundreds of repositories on the Hub to our storage layer, including classic "pre-Hub" open-source models like 
content.raw: https://huggingface.co/FacebookAI/xlm-roberta-large
content.id: FacebookAI/xlm-roberta-large
content.type: model
content.type: resource
content.url: https://huggingface.co/FacebookAI/xlm-roberta-large
content.raw:  (XLM-R) from 
content.type: text
content.value:  (XLM-R) from 
content.image: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Kdjz9yPqcoWdSO-GygEfC.png
content.raw: https://huggingface.co/FacebookAI
content.id: FacebookAI
content.type: org
content.type: resource
content.url: https://huggingface.co/FacebookAI
content.type: new_line
content.type: new_line
content.raw: XLM-R, introduced in 2019, set new benchmarks for multilingual NLP by learning shared representations across 100 languages. It was then fine-tuned on English, Spanish, Dutch, and German, generating language-specific derivations for each - check out the paper here 
content.type: text
content.value: XLM-R, introduced in 2019, set new benchmarks for multilingual NLP by learning shared representations across 100 languages. It was then fine-tuned on English, Spanish, Dutch, and German, generating language-specific derivations for each - check out the paper here 
content.label: Unsupervised Cross-lingual Representation Learning at Scale (1911.02116)
content.raw: https://huggingface.co/papers/1911.02116
content.id: 1911.02116
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/1911.02116
content.type: text
content.type: new_line
content.type: new_line
content.raw: These finetunes share much of the same architecture and layout as XLM-R with similar training methods and goals. It makes sense that they would share bytes, but it's still fascinating to see.
content.type: text
content.value: These finetunes share much of the same architecture and layout as XLM-R with similar training methods and goals. It makes sense that they would share bytes, but it's still fascinating to see.
content.type: new_line
content.type: new_line
content.raw: We put together a similar space to explore these models to see where they overlap - check it out for yourself 
content.type: text
content.value: We put together a similar space to explore these models to see where they overlap - check it out for yourself 
content.raw: https://huggingface.co/spaces/xet-team/finetune-dedupe
content.id: xet-team/finetune-dedupe
content.type: space
content.type: resource
content.url: https://huggingface.co/spaces/xet-team/finetune-dedupe
content.type: text
content.type: new_line
content.type: new_line
content.raw: The darker each block in the heatmap, the more the bytes are shared. Clicking on a repos blocks shows all other repos that share blocks.
content.type: text
content.value: The darker each block in the heatmap, the more the bytes are shared. Clicking on a repos blocks shows all other repos that share blocks.
rawContent: What does it mean when models share the same bytes? 
  
  We've investigated some quants and have seen that a considerable portion of quantizations of the same model share the same bytes and can be deduplicated to save considerable upload time for quantizers on the Hub.
  
  This space where we crack open a repo from @bartowski shows we can get significant dedupe https://huggingface.co/spaces/xet-team/quantization-dedup
  
  You can get a sense of why by reading this write-up: https://github.com/bartowski1182/llm-knowledge/blob/main/quantization/quantization.md
  
  But what about finetuned models?
  
  Since going into production the https://huggingface.co/xet-team has migrated hundreds of repositories on the Hub to our storage layer, including classic "pre-Hub" open-source models like https://huggingface.co/FacebookAI/xlm-roberta-large (XLM-R) from https://huggingface.co/FacebookAI
  
  XLM-R, introduced in 2019, set new benchmarks for multilingual NLP by learning shared representations across 100 languages. It was then fine-tuned on English, Spanish, Dutch, and German, generating language-specific derivations for each - check out the paper here https://huggingface.co/papers/1911.02116 
  
  These finetunes share much of the same architecture and layout as XLM-R with similar training methods and goals. It makes sense that they would share bytes, but it's still fascinating to see.
  
  We put together a similar space to explore these models to see where they overlap - check it out for yourself https://huggingface.co/spaces/xet-team/finetune-dedupe 
  
  The darker each block in the heatmap, the more the bytes are shared. Clicking on a repos blocks shows all other repos that share blocks.
reactions.reaction: 🔥
reactions.users: bartowski daniel-ltw dantezxcd John6666 victor
reactions.reaction: 🧠
reactions.users: Aurelien-Morgan dantezxcd hunoutl
commentators._id: 65d50e9ef9cbfa798c590004
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg
commentators.fullname: Jared Sulzdorf
commentators.name: jsulz
commentators.type: user