Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
benefits! The LoRA paper proposes to replace the Attention weights in LLM transformers. Attention weights contribute to the biggest chunk of all the parameters in any transformer. In fact, LoRA proposes to replace only the query (q) and value (v) parameters in LLM transformer’s Attention weights.