Ask HN: How do you deploy and scale ML (±DL) models?
We&#x27;re building an internal platform for that. Description in my profile. Please get in touch if you&#x27;d like to know more or play with it, we&#x27;d love your feedback. We have given access to about thirty students to prepare their final year projects in vision, NLP, etc.<p>We&#x27;ve been doing consulting for more than six years and we&#x27;re building a platform precisely to solve the problems we have encountered and you are writing about. We have learned some things that we are encoding in the platform, in case you want to build your own. We have started doing this because we hit a ceiling on the projects we could do, and we were under stress. We&#x27;re a tiny, tiny team.<p>The problems are in interfaces between different roles, with each role having a stack with a gazillion tools, and a different &quot;language&quot; they speak and universe they live in. The stitching of people&#x27;s interaction together, the workflow, the business problems, and the fragmented tooling is problematic. The inflexibility of said tooling and frameworks that you addressed also made us not be able to use them, or other platforms. This is why we are working hard to build a coherent, integrated experience, while still trying to bulid abstractions that allow us to substitute tools and view the tools as simple components, not to be tied.<p>For now, it allows you to create a notebook from several images with most libraries pre-installed. The infra it&#x27;s deployed on allows Tesla K80 which you can use. You can of course install additional libraries.<p>This solves the problem of setting the environment, CUDA, docker engine, runtime versions, and the usual yak shaving. We&#x27;re only using JupyterHub and JupyterLab for Python notebooks for now, as it is what our colleagues use, but we plan to support more.<p>It also solves the problem of the &quot;it works on my machine&quot; and running a colleague&#x27;s notebook.<p>You can click on a button and publish an AppBook and share it with a domain expert right away to play with. It is automatically parametrized for you so you don&#x27;t play with widgets, and automatically generates form fields for parameters. The parameters, metrics are tracked behind the scenes without you doing anything, and the models are saved to object storage. Again, one role we target is the ML practitioner who does not necessarily remember to do these things, so we do it for them.<p>Here&#x27;s a video from a very early version: <a href="https:&#x2F;&#x2F;app.box.com&#x2F;s&#x2F;mwsw79g3d5b974o625f1mw979cc4znf0" rel="nofollow">https:&#x2F;&#x2F;app.box.com&#x2F;s&#x2F;mwsw79g3d5b974o625f1mw979cc4znf0</a><p>We&#x27;re using MLFlow for that, but plan to support GuildAI, and Cortex. We think hard to make things loosely coupled and configurable, so you get to pick the stack and easily integrate the platform with existing stack.<p>The AppBook is super useful in that you can publish it and then use it to train the model, or share it with a domain expert so they can play with different parameters. One of the problems we&#x27;ve seen was that some features are considered unimportant for an ML practitioner, but are critical to domain experts.<p>Thightening that feedback loop from notebook to domain expert makes the one click AppBook important because it saves you scheduling meetings and how to &quot;show&quot; the domain expert the work, while allowing them to interact with it.<p>You can also deploy models you choose with one click and it will give you an endpoint and generate a tutorial on how to hit that endpoint to invoke the model with curl or Python requests. You can generate a token and invoke the model in other places or services.<p>This self service feature is important because it allows an ML practitioner to &quot;deploy&quot; their own model, without asking a colleague to do so who might be doing other things. Self service is super important through this.<p>Right now, we&#x27;re focusing on fixing bugs and improving tests and have added monitoring before going back to feature development. Some features we were working on were a more flexible and scalable model deployment strategies, monitoring, collaboration, retraining, and data streams, and building the SDK.