Teaching Transformers Modular Arithmetic at Scale
<|reference_start|>Teaching Transformers Modular Arithmetic at Scale: Modular addition is, on its face, a simple operation: given $N$ elements in $\mathbb{Z}_q$, compute their sum modulo $q$. Yet, scalable machine learning solutions to this problem remain elusive: prior work trains ML models that sum $N \le 6$ elements mod $q \le 1000$. Promising applications of ML models for cryptanalysis-which often involve modular arithmetic with large $N$ and $q$-motivate reconsideration of this problem. This work proposes three changes to the modular addition model training pipeline: more diverse training data, an angular embedding, and a custom loss function. With these changes, we demonstrate success with our approach for $N = 256, q = 3329$, a case which is interesting for cryptographic applications, and a significant increase in $N$ and $q$ over prior work. These techniques also generalize to other modular arithmetic problems, motivating future work.<|reference_end|>
@article{saxena2024teaching,
    title={Teaching Transformers Modular Arithmetic at Scale},
    author={Eshika Saxena, Alberto Alfarano, Emily Wenger, Kristin Lauter},
    journal={arXiv preprint arXiv:2410.03569},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03569},
    primaryClass={cs.LG}
}