Hard ASH: Sparsity and the right optimizer make a continual learner
<|reference_start|>Hard ASH: Sparsity and the right optimizer make a continual learner: In class incremental learning, neural networks typically suffer from catastrophic forgetting. We show that an MLP featuring a sparse activation function and an adaptive learning rate optimizer can compete with established regularization techniques in the Split-MNIST task. We highlight the effectiveness of the Adaptive SwisH (ASH) activation function in this context and introduce a novel variant, Hard Adaptive SwisH (Hard ASH) to further enhance the learning retention.<|reference_end|>
@article{keskinen2024hard,
    title={Hard ASH: Sparsity and the right optimizer make a continual learner},
    author={Santtu Keskinen},
    journal={arXiv preprint arXiv:2404.17651},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.17651},
    primaryClass={cs.LG cs.CV}
}