How to recognize AI snake oil [pdf]
This is good. I co-founded Futurescaper[1], a company that does work in strategic foresight systems  -- collective mapping of complex systems, and analytical tools to help people and organisations to understand them. This put us in a prime position to be vendors of AI snake oil. Due to a stubborn overabundance of ethics, we&#x27;ve refused to do so, which has undoubtedly cost of a lot of business. People <i>want</i> to buy snake oil. It goes down much smoother than hard truths.<p>To amplify what this presentation says, here&#x27;s the hard truths about predicting social outcomes: either they<p>1. Are simple and obvious, and can be easily understood and predicted via regressions and trendlines.<p>2. Are complex, non-obvious, and neither can nor should be predicted. In fact, attempting to predict them is often dangerously wrong. But this doesn&#x27;t mean that they can&#x27;t be understood.<p>In the first instance, you don&#x27;t need any fancy software, so there&#x27;s no snake oil to be sold. The second instance, however, gets caught in a cognitive bias: we think that the future can be predicted. This is because, in simple systems, it can: drop a glass above a hard floor, and you can accurately predict that it will fall and shatter. That is a simple system. We think that complex systems must be similar... just more complex.<p>But complex systems are fundamentally different. Consider a double-pendulum. Its movement can&#x27;t be predicted for more than the next few swings. Even if you know the <i>exact</i> starting configuration of the pendulum -- and by exact, I mean not just every single sub-atomic particle in the pendulum arms, but the gravitational influence of literally every object in the universe -- you still wouldn&#x27;t be able to greatly extend your foreknowledge of its movements. This is because the feedback loops are driven by chaos, and chaos is baked directly into the mathematical fabric of the universe itself.<p>For the mathematically inclined, consider the Mandelbrot set: it is, essentially, the equivalent of a double pendulum. It asks a question: &quot;by starting with this number and iteratively exponentiating it, will in trend towards zero or infinity?&quot;. When you ask this question on a simple number line, then the answer is obvious: below 1, it trends towards zero; above 1, it trends towards infinity. However when you ask this question on the complex number plane -- with two numbers feeding back to each other as they exponentiate -- then the only way to answer the question is to keep iterating the numbers to find out. There&#x27;s no shortcut. In some places, the answer is found quite quickly. In others, it takes thousands of iterations. In still others, it takes infinite iterations: you could build a computer the size of the galaxy and you still wouldn&#x27;t be able to answer the question of whether such-and-such coordinates trends towards zero or infinity. That&#x27;s the complexity you get from just two number lines and a very simple feedback loop.<p>The real world of psychological and social cause-and-effect is <i>far</i> more complex than a double pendulum or a Mandelbrot set, and attempting to predict it is even more futile. In fact, it&#x27;s dangerous.<p>What makes this dangerous is that you <i>can</i> throw statistics at complex futures, and make statements like &quot;Future A has a 40% probability; futures B, C, and D all have a 20% probability&quot;. But the human mind is terrible at making good judgements based on this kind of information. Hearing it, people tend to think: &quot;right, that&#x27;s settled then: Future A is twice as likely as any other scenario, so that&#x27;s what we&#x27;ll plan for.&quot; We fixate on what looks like the most likely future, and disregard the rest.<p>The problem is that if the preparations for Future A are contrary to what you&#x27;d want to do in Futures B, C, and D, then betting everything on Future A means that 60 % of the time, you&#x27;ll lose.<p>What&#x27;s interesting is that increasing the accuracy of those percentages doesn&#x27;t necessarily help, and in many cases can hurt, since it only reinforces our tendency towards target fixation. In a 40&#x2F;20&#x2F;20&#x2F;20 scenario, we might still make some concessions towards planning for the &quot;20s&quot;. But in a 70&#x2F;10&#x2F;10&#x2F;10 scenario, those &quot;10s&quot; will simply be discarded. Which means that 30% of the time you won&#x27;t just lose: you&#x27;ll be blindsided. Utterly fucked.<p>Unfortunately, most of the predictive AI that I&#x27;ve seen is focused on either increasing the &quot;certainty&quot; via extremely dubious means, or simply hiding the non-dominant answers altogether. So the AI does the target fixation for you. That&#x27;s not a good thing.<p>The entire discipline of Scenario Planning[2] evolved to help people and companies &quot;un-predict&quot; the future. Rather than putting percentages on <i>probabilities</i> of outcomes, people need to understand the <i>possibilities</i> of outcomes. Even if complex systems can&#x27;t be predicted, they can be better understood, and that can be very valuable for helping to navigate them in real-time. Rather than giving people the easy (and usually wrong) answer of &quot;here&#x27;s what is going to happen&quot;, you can give them a range of futures that <i>could</i> happen, and understanding those possibilities can aid navigation and lead to better outcomes.<p>It may even be that AI has a legitimate role to play in this process -- but it won&#x27;t be in falsely predicting the outcome of complex systems, nor will it be in absolving people of the responsibility of thinking for themselves. This, unfortunately for my company&#x27;s ability to raise capital, is a much less sexy sales pitch than the AI snake oil. (Although it does get us smarter and less annoying clients, so there&#x27;s that!)<p>1: <a href="https:&#x2F;&#x2F;www.futurescaper.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.futurescaper.com&#x2F;</a><p>1: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Scenario_planning" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Scenario_planning</a>