I can't let go of “The Dunning-Kruger Effect is Autocorrelation”
Gah, I wish I had time to fully read this and get into it, but I have to spend the next few hours driving.<p>Unfortunately the original article isn&#x27;t very clearly explained, and it&#x27;s only on reading the discussion in the comments under it that it becomes clear what it&#x27;s actually saying.<p>The point is about signal &amp; noise. Say your random variable X contains a signal component and a noise component, the former deterministic and the latter random. Say you correlate Y-X against X, and further say you use the <i>same sample</i> of X when computing Y-X as when measuring X. In this case your correlation will include the correlation of <i>a single sample</i> of the noise part of X with its own negation, yielding a spurious negative component that is unrelated to the signal but arises purely from the noise. The problem can be avoided by using a <i>separate sample</i> of X when computing Y-X.<p>The example in the original &quot;DK is autocorrelation&quot; article is an extreme illustration of this. Here, there is no signal at all and X is pure noise. Since the same sample of X is used a strong negative correlation is observed. The key point though is that if you use a <i>separate sample</i> of X that correlation disappears completely. I don&#x27;t think people are realising that in the example given the random result X will yield another totally random value if sampled again. It&#x27;s not a random result <i>per person</i>, it&#x27;s a random result <i>per testing of a person</i>.<p>This is only one objection to the DK analysis, but it&#x27;s a significant one AFAICS. It can be expected that any measurement of &quot;skill&quot; will involve a noise component. If you want to correlate two signals both mixed with the same noise sources you need to construct the experiment such that the noise is sampled separately in the two cases you&#x27;re correlating.<p>Of course the extent to which this matters depends on the extent to which the measurement is noisy. Less noise should mean less contribution of this spurious autocorrelation to the overall correlation.<p>To give another ridiculous, extreme illustration: you could throw a die a thousand times and take each result and write it down twice. You could observe that (of course) the first copy of the value predicts the second copy perfectly. If instead you throw the die twice at each step of the experiment and write those <i>separately sampled</i> values down you will see no such relationship.