Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
does it matter? You wonder, we are just representing the same thing in a different way! So, where does the compute savings come from? Lower compute requirements at training time Let’s assume that ΔW has dimensions 100 x 100 (i.e., 10,000 elements). Let’s also assume that r is 5. Therefore, WA has