Subspace similarity between different random seeds. We further conﬁrm this by plotting the
normalized subspace similarity between two randomly seeded runs with r = 64, shown in Figure 4.
∆Wq appears to have a higher “intrinsic rank” than ∆Wv, since more common singular value direc-
tions are learned by both runs for ∆Wq, which is in line with our empirical observation in Table 6.
As a comparison, we also plot two random Gaussian matrices, which do not share any common
singular value directions with each other.

7.3 HOW DOES THE ADAPTATION MATRIX ∆W COMPARE TO W ?

We further investigate the relationship between ∆W and W . In particular, does ∆W highly correlate
with W ? (Or mathematically, is ∆W mostly contained in the top singular directions of W ?) Also,
7Note that a similar analysis can be carried out with B and the left-singular unitary matrices – we stick with

A for our experiments.

11
LORA