Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
reach-vbΒ 
posted an update 5 days ago
Post
2963
hey hey @mradermacher - VB from Hugging Face here, we'd love to onboard you over to our optimised xet backend! πŸ’₯

as you know we're in the process of upgrading our storage backend to xet (which helps us scale and offer blazingly fast upload/ download speeds too): https://huggingface.co/blog/xet-on-the-hub and now that we are certain that the backend can scale with even big models like Llama 4/ Qwen 3 - we;re moving to the next phase of inviting impactful orgs and users on the hub over as you are a big part of the open source ML community - we would love to onboard you next and create some excitement about it in the community too!

in terms of actual steps - it should be as simple as one of the org admins to join hf.co/join/xet - we'll take care of the rest.

p.s. you'd need to have a the latest hf_xet version of huggingface_hub lib but everything else should be the same: https://huggingface.co/docs/hub/storage-backends#using-xet-storage

p.p.s. this is fully backwards compatible so everything will work as it should! πŸ€—

Can't wait to get on XET, but of course, I am a bit wary because of the amount of repos, and the fact that we have continuous activity :)

I've installed hf_xet on all the backends, so if it's transparent, we should in theory be able to switch anytime.

We don't have an org, so I assume the mradermacher account needs to join, which we will do soon.

Β·

perfect! can you try and join the waitlist via hf.co/join/xet please!

While we are at it - we have lots of split files due to the 50GB limit. I think that limit is gone with xet. Do you think there is a way to somehow combine these files? Obviously, downloading all repos and re-uploading them would be rather horrible for everybody involved (so we won't do that). Maybe there is some better way to do that?

Β·

we're still optimising the > 50GB path, so at least right now, I'd recommend keeping <50 GB shards but this might change soon and then we can work out a plan

good news for the community!