354.8±0.5 (+5.0%)
366.3±0.5 (+8.4%)

23.9±2.1 (+20.7%)
25.8±2.2 (+30.3%)

Table 1: Infernece latency of a single forward pass in GPT-2 medium measured in milliseconds, av-
eraged over 100 trials. We use an NVIDIA Quadro RTX8000. “|Θ|” denotes the number of trainable
parameters in adapter layers. AdapterL and AdapterH are two variants of adapter tuning, which we
describe in Section 5.1. The inference latency introduced by adapter layers can be signiﬁcant in an
online, short-sequence-length scenario. See the full study in Appendix B.

4 OUR METHOD

We describe the simple design of LoRA and its practical beneﬁts. The principles outlined here apply
to any dense layers in deep learning models, though we only focus on certain weights in Transformer
language models in our experiments as the motivating use case.

4.1 LOW-RANK-PARAMETRIZED UPDATE MATRICES
LORA