i

consists of a key-value pair, and the di-

mension of each key and value is 1024.

4584Figure 2: An annotated example of preﬁx-tuning using an autoregressive LM (top) and an encoder-decoder model
(bottom). The preﬁx activations ∀i ∈ Pidx, hi are drawn from a trainable matrix Pθ. The remaining activations are
computed by the Transformer.

3.3 Fine-tuning
In the full ﬁne-tuning framework, we initialize with
the pretrained parameters φ. Here pφ is a train-
able language model distribution and we perform
gradient updates on the following log-likelihood
objective:

log pφ(y | x) = max

log pφ(zi | h<i).

(cid:88)

φ

i∈Yidx

max

φ

(2)

Intuition
Prefix-Tuning