Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
WA x WB, as shown below. Figure 3. Matrix decomposition (adapted from the works of Chris Alexiuk) ΔW has dimensions A x B; WA has dimensions A x r; WB has dimensions r x B. The rank during adaptation is r, where r << min(A, B), because LoRA hypothesizes that. How does LoRA work? LoRA proposes r to