&gt; Your analogy is incorrect because when it comes to social systems, how people bet actually changes the outcome, at least in the long run.<p>That stipulation doesn&#x27;t make the analogy incorrect. It&#x27;s analogously an argument that we should bet on the team likely to lose tomorrow because more people betting on them will make them more likely to win next year. That is not a successful betting strategy even if it&#x27;s true. It&#x27;s just charity. And there is nothing wrong with charity, but then why use such a convoluted mechanism? If charity is what you want then just give your money to the blue team.<p>&gt; In general, the problem is not the math but our interpretation of the result. All the result says is that <i>if we change</i> nothing, this is the likely outcome.<p>Which is what we need to know unless we&#x27;re specifically trying to change something. And if we can find some particular unjustified bias in the algorithm then we are, and then we know what to do because we can adjust the algorithm. But if we can&#x27;t find any such thing to adjust, if the algorithm is unbiased and accurate so far as we can tell, then what is it we&#x27;re supposed to be trying to change?<p>Just looking at e.g. the racial composition of the output doesn&#x27;t actually tell you if anything is wrong. It could be (and often <i>will</i> be) that the algorithm is measuring other things that are correlated with race.