4.2 APPLYING LORA TO TRANSFORMER

In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the
number of trainable parameters. In the Transformer architecture, there are four weight matrices in
the self-attention module (Wq, Wk, Wv, Wo) and two in the MLP module. We treat Wq (or Wk, Wv)
as a single matrix of dimension dmodel × dmodel, even though the output dimension is usually sliced
into attention heads. We limit our study to only adapting the attention weights for downstream
tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity
and parameter-efﬁciency.We further study the effect on adapting different types of attention weight
matrices in a Transformer in Section 7.1. We leave the empirical investigation of adapting the MLP
layers, LayerNorm layers, and biases to a future work.
LORA