The problem with metrics is a big problem for AI
Stepping back a bit, I worked on productizing ML and found a basic principle that causes consternation.<p>The metric is not the ROC curve, the metric is the overall  value the algorithm generates in aggregate, based on the unique business situation as a function of risk.<p>When you look at a trading bot, it is not evaluated on its predictions of market moves, but on how much money it makes - a higher order effect of using the method over time vs. others.<p>You have to look at the symmetry (or distribution) of risk in the problem it&#x27;s being applied to. (famously described in a ROC vs. Indifference curves blog post)<p>In the case of cancer detection, ML is immensely valuable in alerting people who might not otherwise have been tested, but terrible as a diagnostic tool when it is substituted for human judgment. Not because it is inaccurate, but because the consequences of it being wrong are fatal to the patient, where the benefits of it being right are life saving. You need to have a problem with  asymmetric upside to benefit from ML. That is, the downside to using it can&#x27;t be serious, because the machine cannot hold risk or accountability, and no sane organization would expose themselves to catastrophic liability for a probabilistic algorithm being wrong.<p>This view pisses off managers, executives and investors because most of the ML products out there hide their downside. Luckily, most customers are smart enough to recognize this - which is why AI companies are having such trouble finding PMF.<p>Self driving cars are another example of a problem with a catastrophic failure mode. You could argue that if we  made self driving cars as safe as flying, that would be acceptable. Except the net benefit&#x2F;upside of a self driving car is not sufficient to justify the downside of its failure mode.<p>Self-flying vehicles will be more acceptable much sooner than cars because most people don&#x27;t fly. The upside benefit anchors people to a higher degree of risk appetite than for something they already do. If I were in the self driving car business, I&#x27;d switch to the self flying car business for that reason.<p>From a management perspective, AI&#x2F;ML does not produce something you can manage. It&#x27;s not like there are rational levers and incentives you can adjust to achieve outcomes. It&#x27;s a curve, you tweak weights and try to keep it producing aggregate net benefit. If you are not directly engaged in that process, it&#x27;s a lottery with outcomes distributed along its a ROC curve. In many ways, it is a substitute for management.<p>AI&#x2F;ML is useful for marginal fraud detection, policing of various kinds, and other relationships where you are in a position of power over a distribution of outcomes. It is not something you can be honestly served by.<p>This is an unpopular view because it bursts the AI&#x2F;ML bubble, where it has been an excellent source of greater fools for investment, but if you have money in the space, the question to ask is, does the problem this company is solving provide a net benefit that compensates the subjects to this system for its failure mode?<p>Not how low can you get the MTBF FP&#x2F;FN rate, that&#x27;s the hustle. The real question is whether a given individual user or subject of this system can afford the cost of it being wrong. If you don&#x27;t have that answer, metrics are useless.