Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
understand the basics of matrix decomposition before we delve deeper into LoRA. What is matrix decomposition? Let’s assume that delta weights is represented by a matrix with AxB dimensions. The key claim in the LoRA paper is that pre-trained language models have a low “intrinsic dimension”. That