LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted
methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models
(LLMs). LoRA reduces the number of trainable parameters and memory usage while
achieving comparable performance to full fine-tuning. We aim to assess the
viability of training and serving LLMs fine-tuned with LoRA in real-world
applications. First, we measure the quality of LLMs fine-tuned with quantized
low rank adapters across 10 base models and 31 tasks for a total of 310 models.
We find that 4-bit LoRA fine-tuned models outperform base models by 34 points
and GPT-4 by 10 points on average. Second, we investigate the most effective
base models for fine-tuning and assess the correlative and predictive
capacities of task complexity heuristics in forecasting the outcomes of
fine-tuning. Finally, we evaluate the latency and concurrency capabilities of
LoRAX, an open-source Multi-LoRA inference server that facilitates the
deployment of multiple LoRA fine-tuned models on a single GPU using shared base
model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web
application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA
A100 GPU with 80GB memory. LoRA Land highlights the quality and
cost-effectiveness of employing multiple specialized LLMs over a single,
general-purpose LLM.
663447e2133f5282718221c7
Justin Zhao
admin_assigned
2024-05-03T07:47:07.981Z
6462ac71514ee1645bd1f7f7
https://cdn-avatars.huggingface.co/v1/production/uploads/6462ac71514ee1645bd1f7f7/kw2gwI8pyp6uyT4neqV_H.png
Justin Zhao
user
justinxzhao
663447e2133f5282718221c8
Timothy Wang
admin_assigned
2024-05-03T07:47:45.120Z
64da4b603764a5137e564790
/avatars/91fa1acad39978237600ffcdb3201440.svg
Timothy Wang
user
Infernaught
663447e2133f5282718221c9
Wael Abid
admin_assigned
2024-05-03T07:48:05.753Z
64f8bb13bcf14b0c918756f3
/avatars/93b99d434b34bc034373add29f63d84c.svg
Wael Abid
user
abidwael
663447e2133f5282718221ca
Geoffrey Angus
admin_assigned
2024-05-03T07:48:14.202Z
649b332ea523f9cfc653d1da
/avatars/678158ed18bf608701674248b95ab8c4.svg
Geoffrey Angus
user
gdlangus
663447e2133f5282718221cb
Arnav Garg
admin_assigned
2024-05-03T07:48:36.062Z
645db37dd90782b1a6aa3883
https://cdn-avatars.huggingface.co/v1/production/uploads/645db37dd90782b1a6aa3883/0fkR5ev0EBNL2jP3IjWeN.png
Arnav Garg
user
arnavgrg
663447e2133f5282718221cc
Jeffery Kinnison
claimed_verified
2024-05-16T20:21:33.911Z
63c974f8b047b9b12a19fceb
/avatars/3a65fe00cf821f1340edf73e4ca960e1.svg
Jeff Kinnison
user
jeffkinnison
663447e2133f5282718221cd
Alex Sherstinsky
admin_assigned
2024-05-03T07:48:46.992Z
64da9e4bdafcf7595956abcf
https://cdn-avatars.huggingface.co/v1/production/uploads/64da9e4bdafcf7595956abcf/Wf8o2zTUEghptxk-ao_43.jpeg
Alex Sherstinsky
user
alexsherstinsky
663447e2133f5282718221ce
Piero Molino
claimed_verified
2024-05-03T17:32:39.452Z
633654c5d184e6b53c4ec45e
https://cdn-avatars.huggingface.co/v1/production/uploads/633654c5d184e6b53c4ec45e/cZtDfCLckofn7qbfcL8HF.png
Piero Molino
user
w4nderlust
663447e2133f5282718221cf
Travis Addair
admin_assigned
2024-05-03T07:48:56.758Z
64b6ef2f00bac1088cdfaf58
https://cdn-avatars.huggingface.co/v1/production/uploads/64b6ef2f00bac1088cdfaf58/_z4DTu2z4_Y4ZlxGIK9BP.png
Travis Addair
user
tgaddair
663447e2133f5282718221d0
Devvret Rishi
LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted
methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models
(LLMs). LoRA reduces the number of trainable parameters and memory usage while
achieving comparable performance to full fine-tuning. We aim to assess the
viability of training and serving LLMs fine-tuned with LoRA in real-world
applications. First, we measure the quality of LLMs fine-tuned with quantized
low rank adapters across 10 base models and 31 tasks for a total of 310 models.
We find that 4-bit LoRA fine-tuned models outperform base models by 34 points
and GPT-4 by 10 points on average. Second, we investigate the most effective
base models for fine-tuning and assess the correlative and predictive
capacities of task complexity heuristics in forecasting the outcomes of
fine-tuning. Finally, we evaluate the latency and concurrency capabilities of
LoRAX, an open-source Multi-LoRA inference server that facilitates the
deployment of multiple LoRA fine-tuned models on a single GPU using shared base
model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web
application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA
A100 GPU with 80GB memory. LoRA Land highlights the quality and
cost-effectiveness of employing multiple specialized LLMs over a single,
general-purpose LLM.
LoRA fine-tuning reduces computational resources while maintaining high performance across various tasks and models, and LoRAX allows for efficient deployment of multiple LLMs on a single GPU.
Low Rank Adaptation Parameter Efficient Fine-Tuning Large Language Models LoRA quantized low rank adapters GPT-4 LoRAX Multi-LoRA inference server task complexity heuristics latency concurrency Mistral-7B