model performance. One might hypothesize that some problems are harder than others, and so the
model gains and loses the ability to solve them in R over the 600B, 800B, and 1000B checkpoints,
but we find that this is not the case. Instead, we find significant variance in per-problem success rates
for several problems (Table D.3). For these problems, the pass rate between different checkpoints
varies in what appears to be a completely uncorrelated manner. Moreover, manual inspection shows
that the failures are caused by minor mistakes, e.g., not taking the absolute value when computing
GCD, not converting a string to a character array, or not checking edge cases.
StarCoder_paper (1)