Arbitrary-Length Generalization for Addition in a Tiny Transformer
<|reference_start|>Arbitrary-Length Generalization for Addition in a Tiny Transformer: This paper introduces a novel training methodology that enables a Transformer model to generalize the addition of two-digit numbers to numbers with unseen lengths of digits. The proposed approach employs an autoregressive generation technique, processing from right to left, which mimics a common manual method for adding large numbers. To the best of my knowledge, this methodology has not been previously explored in the literature. All results are reproducible, and the corresponding R code is available at github.com/AGPatriota/ALGA-R/.<|reference_end|>
@article{patriota2024arbitrary-length,
    title={Arbitrary-Length Generalization for Addition in a Tiny Transformer},
    author={Alexandre Galvao Patriota},
    journal={arXiv preprint arXiv:2406.00075},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.00075},
    primaryClass={cs.LG stat.AP stat.ML}
}