Why would quartile plotted against percentile not be a straight line? In the famous Dunning-Kruger paper (ungated) perceived and actual scores are plotted, with quartile on the x axis and percentile on the y axis.  The message is the perceived scores, and the actual scores are shown as a line for reference.  In figures 1 and 4 the actual score line is straight, as one would expect as we have the rank score for the same value on both axis. However figure 3 has the 2nd quartile point significantly lower than the identity line, and in figure 2 the 2nd and 3rd quartiles are slightly lower.
What is this describing?


I agree this is odd, but I suppose it could be caused by tied scores showing up in particular quartiles. Suppose we have 100 subjects that get scores from 1 to 100 in 1-point intervals, except that everybody who would have had a score between 26 and 50 instead gets a score of 30. In this case, the average score percentile among the second quartile is either 26 or 50, depending on how you handle endpoints in the percentile calculation. For a dataset with only distinct scores, the average percentile of the second quartile must be 37.5, but it would be possible to find a different average percentile within a quartile that contains tied scores. Not sure if this is the case, or if it's just an error in plotting - even Figures 1 and 4 don't look exactly the same in "actual test score" values (the second quantile in Fig 1 is near 35, but it's closer to 40 in Fig 4).
