i=1 are orthogonal unit vec-
tors (r is the rank of Wdown). Thus, ∆W is a combination of
r orthogonal vectors, limiting the change of weight residing in
a low-rank space. Consequently, there is no need to store full-
rank input activations simultaneously, alleviating the memory
burden associated with activation storage.
Parameter-EfficientFine-TuningMethods