Artificial Intelligence Needs a Bullshit Meter
This is a very confused article with many errors. It appears to imply that no one validates the accuracy of machine learning models, or that they lie about the accuracy. Even more strangely, it suggest that the only way to address this is by ensemble methods.<p>Validation of models is one of the most important parts of any machine learning system. Every expert practitioner measures the accuracy of their models with well established methods such as cross validation or hold out tests. So the basic premise of this article seems quite at odds with the reality.<p>Further, the article users the security domain as an example of the lack of validation. Most applications of ML to security use <i>unsupervised</i> algorithms to perform anomaly detection. This is entirely a different thing to a supervised algorithm. Anomaly detection via unsupervised algorithms is well known to have many false positives.<p>But possible the worst error in the article is suggesting that ensemble methods are a way to validate the accuracy of a model. An ensemble technique is <i>not</i> a way to validate accuracy. Rather it&#x27;s a way to try to obtain higher accuracy. You still need to validate your ensemble via something like cross fold validation to understand the expected error.