Lora, Low Rank Adaptation, Generative Ai, Llm, Fine Tuning.
represent it slightly differently. Assume that we have frozen pre-trained weights (W) and frozen inputs (x), and all the trainable parameters are in delta weights (highlighted in green below). We calculate delta weights and the hidden layer activation gets updated like before. But now we track the