Launch HN: Replicate (YC W20) – Version control for machine learning
Hello HN!<p>We're Ben & Andreas, and we made Replicate. It's a lightweight open-source tool for tracking and analyzing your machine learning experiments: <a href="https://replicate.ai/" rel="nofollow">https://replicate.ai/</a><p>Andreas used to do machine learning at Spotify. He built a lot of ML infrastructure there (versioning, training, deployment, etc). I used to be product manager for Docker's open source projects, and created Docker Compose.<p>We built <a href="https://www.arxiv-vanity.com/" rel="nofollow">https://www.arxiv-vanity.com/</a> together for fun, which led to us teaming up to build more tools for ML.<p>We spent a year talking to lots of people in the ML community and building all sorts of prototypes, but we kept on coming back to a foundational problem: not many people in machine learning use version control.<p>This causes all sorts of problems: people are manually keeping track of things in spreadsheets, model weights are scattered on S3, and results can’t be reproduced.<p>So why isn’t everyone using Git? Git doesn’t work well with machine learning. It can’t store trained machine learning models, it can’t handle key/value metadata, and it’s not designed to record information automatically from a training script. There are some solutions for these things, but they feel like band-aids.<p>We came to the conclusion that we need a native version control system for ML. It’s sufficiently different to normal software that we can’t just put band-aids on Git.<p>We believe the tool should be small, easy to use, and extensible. We found people struggling to migrate to “AI Platforms”. A tool should do one thing well and combine with other tools to produce the system you need.<p>Finally, we also believe it should be open source. There are a number of proprietary solutions, but something so foundational needs to be built by and for the ML community.<p>Replicate is a first cut at something we think is useful: It is a Python library that uploads your files and metadata (like hyperparameters) to Amazon S3 or Google Cloud Storage. You can get back to any point in time using the command-line interface, analyze your results inside a notebook using the Python API, and load your models in production systems.<p>We’d love to hear your feedback, and hear your stories about how you’ve done this before.<p>Also – building a version control system is rather complex, and to make this a reality we need your help. Join us in Discord if you want to be involved in the early design and help build it: <a href="https://discord.gg/QmzJApGjyE" rel="nofollow">https://discord.gg/QmzJApGjyE</a>