hi = LMφ(zi, h<i),

(1)

i

to logits over the vocabulary.

where the last layer of hi is used to compute the
distribution for the next token: pφ(zi+1 | h≤i) =
softmax(Wφ h(n)
) and Wφ is a matrix that maps
h(n)
i
3.2 Encoder-Decoder Architecture
We can also use an encoder-decoder architecture
(e.g., BART; Lewis et al., 2020) to model pφ(y | x),
where x is encoded by the bidirectional encoder,
and the decoder predicts y autoregressively (condi-
tioned on the encoded x and its left context). We
use the same indexing and activation notation, as
shown in Figure 2 (bottom): each hi for i ∈ Xidx
is computed by the a bidirectional encoder; each
hi for i ∈ Yidx is computed by an autoregressive
decoder using the same equation (1).

1In GPT-2, h(n)

i

consists of a key-value pair, and the di-

mension of each key and value is 1024.
Prefix-Tuning