Parameter-Efﬁcient Adaptation. Many have proposed inserting adapter layers between existing
layers in a neural network (Houlsby et al., 2019; Rebufﬁ et al., 2017; Lin et al., 2020). Our method
uses a similar bottleneck structure to impose a low-rank constraint on the weight updates. The
key functional difference is that our learned weights can be merged with the main weights during
inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3).
A comtenporary extension of adapter is COMPACTER (Mahabadi et al., 2021), which essentially
parametrizes the adapter layers using Kronecker products with some predetermined weight sharing
scheme. Similarly, combining LoRA with other tensor product-based methods could potentially
improve its parameter efﬁciency, which we leave to future work. More recently, many proposed
optimizing the input word embeddings in lieu of ﬁne-tuning, akin to a continuous and differentiable
LORA