Acceleration of Grokking in Learning Arithmetic Operations via Kolmogorov-Arnold Representation
<|reference_start|>Acceleration of Grokking in Learning Arithmetic Operations via Kolmogorov-Arnold Representation: We propose novel methodologies aimed at accelerating the grokking phenomenon, which refers to the rapid increment of test accuracy after a long period of overfitting as reported in~\cite{power2022grokking}. Focusing on the grokking phenomenon that arises in learning arithmetic binary operations via the transformer model, we begin with a discussion on data augmentation in the case of commutative binary operations. To further accelerate, we elucidate arithmetic operations through the lens of the Kolmogorov-Arnold (KA) representation theorem, revealing its correspondence to the transformer architecture: embedding, decoder block, and classifier. Observing the shared structure between KA representations associated with binary operations, we suggest various transfer learning mechanisms that expedite grokking. This interpretation is substantiated through a series of rigorous experiments. In addition, our approach is successful in learning two nonstandard arithmetic tasks: composition of operations and a system of equations. Furthermore, we reveal that the model is capable of learning arithmetic operations using a limited number of tokens under embedding transfer, which is supported by a set of experiments as well.<|reference_end|>
@article{park2024acceleration,
    title={Acceleration of Grokking in Learning Arithmetic Operations via
  Kolmogorov-Arnold Representation},
    author={Yeachan Park, Minseok Kim, Yeoneung Kim},
    journal={arXiv preprint arXiv:2405.16658},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.16658},
    primaryClass={cs.LG cs.AI}
}