Sigh.  This article describes a mathematical analysis that is more an artifact of the problem setup than a proof that more evidence decreases confidence.  In particular, what&#x27;s going on is that it assumes that while actual witnesses have a probability of e.g. 50% of convicting, biased witnesses have a probability of 100% of convicting(!), which is an incredibly strong assumption (and in particular, if you changed 100% to anything else, one of the headline results they advertise of unanimity being suspicious and near-unanimity not being suspicious disappears).<p>Those values of the parameters don&#x27;t make sense for many real-world scenarios like picking a subject out of a lineup (you&#x27;d have to have a really really bad lineup to have witnesses be guaranteed to pick the same wrong person!) or to the output of a panel of judges or jurors.<p>However, the result is relevant in scenarios where it&#x27;s very likely that you&#x27;d have a 100% likely error, e.g. you&#x27;re doing an experiment and your apparatus isn&#x27;t functioning properly or you have a bug in your computer program doing the data analysis.  So it&#x27;s relevant for physicists and computer programmers, but probably not to the criminal justice system.<p>For those curious, the math is quite simple.  They model a world where one of the following happens:
* probability p: compromised experiment, in which case fraction y (y=1 in the paper) of witnesses report guilty
* probably 1-p: normal experiment, in which case fraction x of witnesses report guilty<p>Given that, the probability that 100% of N witnesses agree the person is guilty in each case is:<p>prob(unanimous and compromised) = p * y^N
prob(unanimous and not compromised) =  (1-p) x^N ~= x^N (since p is assumed small)<p>So the probability that the experiment is compromised, given a unanimous outcome, scales with their ratio, p y^N &#x2F; x^N = p * (y&#x2F;x)^N.  A few things you notice:
* If y &lt;= x (as you&#x27;d hope to be the case in a lineup where the bias is less convincing than having actually seen the criminal!), unanimity is never suspicious.  This is the usual world where more evidence should increase your belief in a hypothesis!
* If y &gt; x, you have exponential growth and so with sufficiently large N you&#x27;ll see unanimous results are almost always the result of a compromised scenario.  However, unless y&#x2F;x is large and y is close to 1, the actual probability of unanimity is basically 0 so this will essentially never happen.
* If y=1 and x &lt;&lt; 1 (the case they analyze), with large N, unanimity is suspicious (but if you change y to 0.95, near-unanimity is, too, so ).  This is a case very relevant in software, but less relevant in the criminal justice system.<p>They picked y=1 and x=0.5, which means roughly you start to see a big effect of this form when N &gt; log_2(1&#x2F;p), which for their ranges of p from 0.01 to 0.0001 means N ~ 15-30 range.<p>Unfortunately this is part of a much broader trend of people making bold claims about society based on the results of either a study with a sample a freshman psychology class that achieved 95% confidence (just like ~5% of the thousands of other studies done on psychology classes!) or a mathematical analysis of a confusingly designed problem that doesn&#x27;t reflect well the broad swath of reality its authors (or the people reporting on it to a wide audience!) are claiming it describes.