The Myth That Machine Learning Engineers Are Not Good General Software Engineers
What we noticed delivering products in the field for the last seven years is that we had different profiles:<p>- Some of our members could go through the whole thing: meeting client, understanding business problem, write code to connect to data sources, get the data, clean it, set up training environments, train models, set up serving environments, deploy models, write the application (web, mobile).<p>- Some of our members could do a subset of these: feature engineering, training models. <i>XOR</i> build applications and deploy models.<p>This has put us in a situation where those who cannot &quot;do it all&quot; <i>relied</i> on someone who can do the part they needed. For example, asking someone to deploy their model, or setup their compute environment, etc. The person might be working on something else, or deploying that very person&#x27;s earlier model version. Blocking.<p>We wanted to lower the frictions, so we built our &quot;machine learning platform&quot;[0] to allow people to do things that have higher value and impact, and allow people to be independent. For example, our people who are great at training models but cannot deploy them now can with the click of a button. The platform also takes care of setting up environments, scheduling long-running notebooks, automatically tracking experiments (parameters, metrics, models) because we used to rely on people being consistent and <i>remembering</i> to do that, and then we had a divergence of methods to do so (ad-hoc processes).<p>It also allows people who can &quot;do it all&quot; not to waste time with these tasks that are blocking their colleagues, and work on more valuable things. Just because they can do it all does not mean they ought to be setting up training or serving environments.<p>- [0]: <a href="https:&#x2F;&#x2F;iko.ai" rel="nofollow">https:&#x2F;&#x2F;iko.ai</a>