Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
We assume that everything else is frozen and we don’t need the optimizer states for them. We just need those small injections which are those matrix pairs. The paper empirically shows how they went from 1.2TB to 350GB achieving 90% of the accuracy and 25 times speed-up of the training process, with