at zero performance). We observe this occurs due to the formatting of the benchmark, where grid elements
are represented as neighboring characters i.e., “8686” (instead of “ 8 6 8 6”). While subtle, this difference
is enough for certain Byte-Pair Encoding (or SentencePiece) tokenizers [75, 76] (that do not tokenize per
digit) to group multiple grid elements (“8” and “6”) into a single token (“86”) which maps to a different
token embedding. This causes inconsistencies with how patterns are expressed at the token level. For
example, given a task expressed as “8686, 6868; 7979,” if the tokenizer groups pairs of digits 86, 68,
79, respectively, the sequential inductive patterns of the task (to swap and repeat individual digits) are lost.
A simple work-around is to directly pass token indices or embeddings to the language model, or use token
alphabets unlikely to be grouped together (which involves some knowledge about the tokenizer). Even
LargeLanguageModelsasGeneralPatternMachines