Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
is, the weight matrix can be represented almost as accurately using way fewer dimensions (i.e., there are a lot of intrinsic redundancies which we can get rid of without losing much). The LoRA paper then hypothesizes that the matrix will also have a low intrinsic rank during adaptation. Now, rank