LoQT: Low-Rank Adapters for Quantized Pre-Training
<|reference_start|>LoQT: Low-Rank Adapters for Quantized Pre-Training: Training of large neural networks requires significant computational resources. Despite advances using low-rank adapters and quantization, pretraining of models such as LLMs on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose LoQT, a method for efficiently training quantized models. LoQT uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning of models, which we demonstrate experimentally for language modeling and downstream task adaptation. We find that LoQT enables efficient training of models up to 7B parameters on a consumer-grade 24GB GPU. We also demonstrate the feasibility of training a 13B parameter model using per-layer gradient updates on the same hardware.<|reference_end|>
@article{loeschcke2024loqt:,
    title={LoQT: Low-Rank Adapters for Quantized Pretraining},
    author={Sebastian Loeschcke, Mads Toftrup, Michael J. Kastoryano, Serge
  Belongie and V'esteinn Sn{ae}bjarnarson},
    journal={arXiv preprint arXiv:2405.16528},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.16528},
    primaryClass={cs.LG cs.CL}
}