Why would quartile plotted against percentile not be a straight line? In the famous Dunning-Kruger paper (ungated) perceived and actual scores are plotted, with quartile on the x axis and percentile on the y axis.  The message is the perceived scores, and the actual scores are shown as a line for reference.  In figures 1 and 4 the actual score line is straight, as one would expect as we have the rank score for the same value on both axis. However figure 3 has the 2nd quartile point significantly lower than the identity line, and in figure 2 the 2nd and 3rd quartiles are slightly lower.
What is this describing?


You said: "The message is the perceived scores, and the actual scores are shown as a line for reference" but I disagree.
It doesn't matter that they aren't straight.
Consider this, a variant of the upper left subplot:

The yellow-colored region is where the perceived ability is higher than the actual ability.  The green region is where the perceived ability is lower than the actual.
I think the important point is that when actual skill is low, estimate of performance is falsely very high, but this decreases as skill increases until the skill is very high, and estimated skill is moderately lower than actual skill.
Distribution of the measured variable can vary with population.  It is easy to make a pathological example, where 25% of values get grades of 10% and the remainder get 100% to make a curve that is concave down.  Similarly if 75% of all grades are 10% and the last quarter are 100% then it is strongly concave up.  There are many general forms of curves and only a subset are linear.
Here is some code for exploring this a little bit, numerically.
#library stuff
if(!require(pacman)) {
install.packages("pacman"); 
require(pacman) }

p_load(dplyr,     #munging
       ggplot2)

#reproducibility
set.seed(575743)

#uniformly random sample of grades
y <- runif(100, min=0, max=100)

# y <- rnorm(100, mean=80, sd=10)
# y <- ifelse(y>100,100,y)

#make background points
q_list_all <- numeric(100)
q_store <- data.frame(q=0:100, y=0*(0:100), bin=0)
for(i in 1:101){
  q_val <- (i-1)/100
  q_store$y[i] <- quantile(y,q_val)
}

q_store$bin <- ceiling(q_store$y/round(diff(range(q_store$y))/3.5))

#make the plot
ggplot(data=q_store, aes(x=q, y=y) ) +
  geom_path(color="blue") + geom_point(color="blue") +
  geom_boxplot(aes(group=bin), alpha=0.5) +
  xlab("Quartile") + ylab("grade on test") 

This is the plot that it makes:

If instead of using a uniform distribution, you were to use a normal one centered at 80% with a standard deviation of 10% and a maximum of 100%, then the plot looks more like this:

