I spent a few months recently on a tough ML project, and pretty much got my ass kicked the entire time. It seemed like every hurdle that I overcome was met with another as I turned up the &quot;voltage&quot;. I came to regard every decision that I made (this includes all kinds of hyperparameters, but also design decisions) with extreme suspicion. I don&#x27;t really think there is a convincing way around this: any kind of optimization has a context outside of which the optimization no longer makes sense. And so one tries to include this context, but this turns into a meta-optimization with a meta-context assumed, and so-on in infinite regress. I guess I am agreeing with the author: &quot;if two algorithms achieve the same performance on a task, the one with less hyperparameter optimization is generally preferable.&quot;<p>It really seems like there should be more of a theory around these issues. Even a dreadfully abstract and&#x2F;or terse VC-dimension-scariness level of a theory.<p>I keenly await the author&#x27;s next post.