LoRTA: Low Rank Tensor Adaptation of Large Language Models
<|reference_start|>LoRTA: Low Rank Tensor Adaptation of Large Language Models: Low Rank Adaptation (LoRA) is a popular Parameter Efficient Fine Tuning (PEFT) method that effectively adapts large pre-trained models for downstream tasks. LoRA parameterizes model updates using low-rank matrices at each layer, significantly reducing the number of trainable parameters and, consequently, resource requirements during fine-tuning. However, the lower bound on the number of trainable parameters remains high due to the use of the low-rank matrix model. In this paper, we address this limitation by proposing a novel approach that employs a low rank tensor parametrization for model updates. The proposed low rank tensor model can significantly reduce the number of trainable parameters, while also allowing for finer-grained control over adapter size. Our experiments on Natural Language Understanding, Instruction Tuning, Preference Optimization and Protein Folding benchmarks demonstrate that our method is both efficient and effective for fine-tuning large language models, achieving a substantial reduction in the number of parameters while maintaining comparable performance.<|reference_end|>
@article{hounie2024lorta:,
    title={LoRTA: Low Rank Tensor Adaptation of Large Language Models},
    author={Ignacio Hounie, Charilaos Kanatsoulis, Arnuv Tandon, Alejandro Ribeiro},
    journal={arXiv preprint arXiv:2410.04060},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.04060},
    primaryClass={cs.CL cs.AI}
}