Sparsity-Accelerated Training for Large Language Models
<|reference_start|>Sparsity-Accelerated Training for Large Language Models: Large language models (LLMs) have demonstrated proficiency across various natural language processing (NLP) tasks but often require additional training, such as continual pre-training and supervised fine-tuning. However, the costs associated with this, primarily due to their large parameter count, remain high. This paper proposes leveraging \emph{sparsity} in pre-trained LLMs to expedite this training process. By observing sparsity in activated neurons during forward iterations, we identify the potential for computational speed-ups by excluding inactive neurons. We address associated challenges by extending existing neuron importance evaluation metrics and introducing a ladder omission rate scheduler. Our experiments on Llama-2 demonstrate that Sparsity-Accelerated Training (SAT) achieves comparable or superior performance to standard training while significantly accelerating the process. Specifically, SAT achieves a $45\%$ throughput improvement in continual pre-training and saves $38\%$ training time in supervised fine-tuning in practice. It offers a simple, hardware-agnostic, and easily deployable framework for additional LLM training. Our code is available at https://github.com/OpenDFM/SAT.<|reference_end|>
@article{ma2024sparsity-accelerated,
    title={Sparsity-Accelerated Training for Large Language Models},
    author={Da Ma and Lu Chen and Pengyu Wang and Hongshen Xu and Hanqi Li and
  Liangtai Sun and Su Zhu and Shuai Fan and Kai Yu},
    journal={arXiv preprint arXiv:2406.01392},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01392},
    primaryClass={cs.CL}
}