A.4 PCFG Benchmark: Additional Details and Ablations

Our PCFG benchmark is a procedurally generated, adjustable-difficulty benchmark for measuring abstract
sequence transformation capabilities in LLMs, based on the PCFG from [77]. In Table 3, we show
illustrations of the primitive operations in the PCFG that can be applied on one or two sequences of
tokens. In Table 4, we show examples of two transformations (of different complexities) from our
benchmark, which are composed of the primitive operations. In Table 5, we show independent ablations of
sequence length (number of tokens) k and complexity (number of rules) w in the sequence transformations,
illustrating the way in which the solve rate decreases as either factor increases. In Listing 2, we show an
example context for a PCFG problem on integer sequences.

Tab. 3: Illustrations depicting the unary and binary operators from Hupkes et al. 2020, which we use for our PCFG
benchmark.
LargeLanguageModelsasGeneralPatternMachines