2

(1)

(2)

During full ﬁne-tuning, the model is initialized to pre-trained weights Φ0 and updated to Φ0 + ∆Φ
by repeatedly following the gradient to maximize the conditional language modeling objective:

(cid:88)

|y|(cid:88)

(x,y)∈Z

t=1

max

Φ

log (PΦ(yt|x, y<t))

One of the main drawbacks for full ﬁne-tuning is that for each downstream task, we learn a different
set of parameters ∆Φ whose dimension |∆Φ| equals |Φ0|. Thus, if the pre-trained model is large
(such as GPT-3 with |Φ0| ≈ 175 Billion), storing and deploying many independent instances of
ﬁne-tuned models can be challenging, if at all feasible.
In this paper, we adopt a more parameter-efﬁcient approach, where the task-speciﬁc parameter
increment ∆Φ = ∆Φ(Θ) is further encoded by a much smaller-sized set of parameters Θ with
|Θ| (cid:28) |Φ0|. The task of ﬁnding ∆Φ thus becomes optimizing over Θ:

(cid:88)

|y|(cid:88)

log(cid:0)pΦ0+∆Φ(Θ)(yt|x, y<t)(cid:1)

max

Θ

(x,y)∈Z

t=1
LORA