19

+0+1+2+3+4+5+6+7+8Generalization beyond training0.00.20.40.60.81.0AccuracyAddition Accuracy w/o CoT (300M)3925DigitsFigure 8: This figure describes the generalization accuracy of the model’s addition capabilities over
the course of training over the training run of the 300M model. Blue lines indicate the supervised
training phase, while red lines indicate the self-training phase. While initially, models show little
generalization to lengths greater than than what they have seen in training, models quickly learn to
generalize well beyond their training distribution using chain-of-thought, allowing them to continue
teaching themselves addition problems beyond what they have seen before.

20

+0+1+2+3+4+5+6+7+8Generalization beyond training0.00.20.40.60.81.0AccuracyAddition Accuracy w/ CoT (300M)3925DigitsD CONTROL EXPERIMENTS
CHAIN-OF-THOUGHTREASONING IS APOLICY IMPROVEMENTOPERATOR