title: LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
summary: Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted
  methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models
  (LLMs). LoRA reduces the number of trainable parameters and memory usage while
  achieving comparable performance to full fine-tuning. We aim to assess the
  viability of training and serving LLMs fine-tuned with LoRA in real-world
  applications. First, we measure the quality of LLMs fine-tuned with quantized
  low rank adapters across 10 base models and 31 tasks for a total of 310 models.
  We find that 4-bit LoRA fine-tuned models outperform base models by 34 points
  and GPT-4 by 10 points on average. Second, we investigate the most effective
  base models for fine-tuning and assess the correlative and predictive
  capacities of task complexity heuristics in forecasting the outcomes of
  fine-tuning. Finally, we evaluate the latency and concurrency capabilities of
  LoRAX, an open-source Multi-LoRA inference server that facilitates the
  deployment of multiple LoRA fine-tuned models on a single GPU using shared base
  model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web
  application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA
  A100 GPU with 80GB memory. LoRA Land highlights the quality and
  cost-effectiveness of employing multiple specialized LLMs over a single,
  general-purpose LLM.
paper_authors._id: 663447e2133f5282718221c7
paper_authors.name: Justin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:47:07.981Z
paper_authors._id: 6462ac71514ee1645bd1f7f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6462ac71514ee1645bd1f7f7/kw2gwI8pyp6uyT4neqV_H.png
paper_authors.fullname: Justin Zhao
paper_authors.type: user
paper_authors.user: justinxzhao
paper_authors._id: 663447e2133f5282718221c8
paper_authors.name: Timothy Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:47:45.120Z
paper_authors._id: 64da4b603764a5137e564790
paper_authors.avatarUrl: /avatars/91fa1acad39978237600ffcdb3201440.svg
paper_authors.fullname: Timothy Wang
paper_authors.type: user
paper_authors.user: Infernaught
paper_authors._id: 663447e2133f5282718221c9
paper_authors.name: Wael Abid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:48:05.753Z
paper_authors._id: 64f8bb13bcf14b0c918756f3
paper_authors.avatarUrl: /avatars/93b99d434b34bc034373add29f63d84c.svg
paper_authors.fullname: Wael Abid
paper_authors.type: user
paper_authors.user: abidwael
paper_authors._id: 663447e2133f5282718221ca
paper_authors.name: Geoffrey Angus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:48:14.202Z
paper_authors._id: 649b332ea523f9cfc653d1da
paper_authors.avatarUrl: /avatars/678158ed18bf608701674248b95ab8c4.svg
paper_authors.fullname: Geoffrey Angus
paper_authors.type: user
paper_authors.user: gdlangus
paper_authors._id: 663447e2133f5282718221cb
paper_authors.name: Arnav Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:48:36.062Z
paper_authors._id: 645db37dd90782b1a6aa3883
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645db37dd90782b1a6aa3883/0fkR5ev0EBNL2jP3IjWeN.png
paper_authors.fullname: Arnav Garg
paper_authors.type: user
paper_authors.user: arnavgrg
paper_authors._id: 663447e2133f5282718221cc
paper_authors.name: Jeffery Kinnison
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-16T20:21:33.911Z
paper_authors._id: 63c974f8b047b9b12a19fceb
paper_authors.avatarUrl: /avatars/3a65fe00cf821f1340edf73e4ca960e1.svg
paper_authors.fullname: Jeff Kinnison
paper_authors.type: user
paper_authors.user: jeffkinnison
paper_authors._id: 663447e2133f5282718221cd
paper_authors.name: Alex Sherstinsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:48:46.992Z
paper_authors._id: 64da9e4bdafcf7595956abcf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64da9e4bdafcf7595956abcf/Wf8o2zTUEghptxk-ao_43.jpeg
paper_authors.fullname: Alex Sherstinsky
paper_authors.type: user
paper_authors.user: alexsherstinsky
paper_authors._id: 663447e2133f5282718221ce
paper_authors.name: Piero Molino
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-03T17:32:39.452Z
paper_authors._id: 633654c5d184e6b53c4ec45e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633654c5d184e6b53c4ec45e/cZtDfCLckofn7qbfcL8HF.png
paper_authors.fullname: Piero Molino
paper_authors.type: user
paper_authors.user: w4nderlust
paper_authors._id: 663447e2133f5282718221cf
paper_authors.name: Travis Addair
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:48:56.758Z
paper_authors._id: 64b6ef2f00bac1088cdfaf58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b6ef2f00bac1088cdfaf58/_z4DTu2z4_Y4ZlxGIK9BP.png
paper_authors.fullname: Travis Addair
paper_authors.type: user
paper_authors.user: tgaddair
paper_authors._id: 663447e2133f5282718221d0
paper_authors.name: Devvret Rishi
paper_title: LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
paper_summary: Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted
  methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models
  (LLMs). LoRA reduces the number of trainable parameters and memory usage while
  achieving comparable performance to full fine-tuning. We aim to assess the
  viability of training and serving LLMs fine-tuned with LoRA in real-world
  applications. First, we measure the quality of LLMs fine-tuned with quantized
  low rank adapters across 10 base models and 31 tasks for a total of 310 models.
  We find that 4-bit LoRA fine-tuned models outperform base models by 34 points
  and GPT-4 by 10 points on average. Second, we investigate the most effective
  base models for fine-tuning and assess the correlative and predictive
  capacities of task complexity heuristics in forecasting the outcomes of
  fine-tuning. Finally, we evaluate the latency and concurrency capabilities of
  LoRAX, an open-source Multi-LoRA inference server that facilitates the
  deployment of multiple LoRA fine-tuned models on a single GPU using shared base
  model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web
  application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA
  A100 GPU with 80GB memory. LoRA Land highlights the quality and
  cost-effectiveness of employing multiple specialized LLMs over a single,
  general-purpose LLM.
paper_ai_summary: LoRA fine-tuning reduces computational resources while maintaining high performance across various tasks and models, and LoRAX allows for efficient deployment of multiple LLMs on a single GPU.
paper_ai_keywords: Low Rank Adaptation Parameter Efficient Fine-Tuning Large Language Models LoRA quantized low rank adapters GPT-4 LoRAX Multi-LoRA inference server task complexity heuristics latency concurrency Mistral-7B