Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
the help of r = 8 applied to all 96 layers of attentions (so that every layer or block is of similar structure). One more cool thing. Imagine a situation where we have 1 foundation model and 10 different downstream tasks. With LoRA, we will just have to swap the small matrices for different