input to generate xi

74
69
60
54
50
39

Accuracy (%)

†58 ± 1

75

5

provides a more unbiased evaluation of pattern reasoning capabilities in LLMs; PCFG completion accuracy
improves with model scale, and correlates with ARC performance. We use PCFG for evaluation only (rather
than for training [77, 78]) so that one can measure how pre-training regimes or modalities may improve
general pattern capabilities across sequence transformations. We have released the PCFG benchmark.

5 Sequence Completion

Fig. 3: LLMs (d3 shown) can extrapolate various func-
tions y = a·sin(bx) (top row), y = ax·sin(bx) (middle
row), and y = a
2x sin(bx) (bottom row) given amounts of
context. Overall, larger models make better predictions
with lower error rates (right column). More context also
helps prediction accuracy (light vs. dark).
LargeLanguageModelsasGeneralPatternMachines