3We ﬁnd in preliminary experiments that directly optimiz-
ing the preﬁx is very sensitive to initialization.
4Pθ has dimensions |Pidx| × dim(hi) while Pθ has
dimensions |Pidx| × k. We choose k = 512 for table-to-text
and 800 for summarization. MLPθ maps from k to dim(hi).

#examples

input length

output length

E2E

WebNLG

DART
XSUM

50K
22K
82K
225K

28.5
49.6
38.8
473.3

27.8
30.7
27.3
28.1

Table 1: Datasets statistics. The input and output
length is the number of BPE tokens per example. For
the three table-to-text datasets, the input length is the
length of linearized tables (details in Appendix A.1).
Prefix-Tuning