Title: I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git.<p>Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: <a href="https:&#x2F;&#x2F;xethub.com&#x2F;assets&#x2F;docs&#x2F;how-xet-deduplication-works" rel="nofollow">https:&#x2F;&#x2F;xethub.com&#x2F;assets&#x2F;docs&#x2F;how-xet-deduplication-works</a><p>Today, XetHub works for 1 TB repositories, and we plan to scale to 100 TB in the next year. Our implementation is in Rust (client &amp; cache + storage) and our web application is written in Go. XetHub includes a GitHub-like web interface that provides automatic CSV summaries and allows custom visualizations using Vega. Even at 1 TB, we know downloading an entire repository is painful, so we built git-xet mount - which, in seconds, provides a user-mode filesystem view over the repo.<p>XetHub is available today (Linux &amp; Mac today, Windows coming soon) and we would love your feedback!<p>Read more here:<p>- <a href="https:&#x2F;&#x2F;xetdata.com&#x2F;blog&#x2F;2022&#x2F;10&#x2F;15&#x2F;why-xetdata" rel="nofollow">https:&#x2F;&#x2F;xetdata.com&#x2F;blog&#x2F;2022&#x2F;10&#x2F;15&#x2F;why-xetdata</a><p>- <a href="https:&#x2F;&#x2F;xetdata.com&#x2F;blog&#x2F;2022&#x2F;12&#x2F;13&#x2F;introducing-xethub" rel="nofollow">https:&#x2F;&#x2F;xetdata.com&#x2F;blog&#x2F;2022&#x2F;12&#x2F;13&#x2F;introducing-xethub</a>
Upvote: