share a constant intrinsic rank for each low-rank sub-matrix, while not accounting for the varying importance across different
modules. AdaLoRA [322] addresses this limitation by dynamically allocating parameter budgets to weight matrices based on
their importance scores. It assigns higher ranks to more critical incremental matrices, capturing more detailed task-specific
information, while reducing the rank of less important matrices to avoid overfitting and save computational resources.
SoRA[69] introduces an optimize-able gate that dynamically adjusts the rank of the incremental matrix using the proximal
gradient method. In QLoRA [66], the pretrained model is initially quantized to 4 bits. Subsequently, a small set of learnable
low-rank adapter weights are augmented and fine-tuned using backpropagated gradients over the quantized weights. QLoRA
can match the performance of 16-bit full-parameter fine-tuning even with 16-bit, 8-bit, or 4-bit adapters.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey