task clusters, and T0++ performed much better in the structure-to-text and summarization task clusters.
Overall, ID-PT + J1-Large slightly surpassed the performance of T0++ in the test score average
across datasets.
Figure 3 shows the average development set scores we observed for ID-PT + J1-Large at different
points during training. The prompt length experiment at 10% of the T0++ training data shows that
longer than usual prompt lengths were indeed useful in this multi-tasked setting (see numerical results
for ID-PT and prompt tuning at different prompt lengths in table 9 in the appendix). We continued
training with the 400 prompt tokens variant, which reached the average development set score of
T0++ after training on roughly 50% of its training data. The test set evaluation given in Table 1 was
performed at this checkpoint. Figure 3 also shows results for the prompt tuning baseline, which
underperformed in this multi-tasked setting, demonstrating the need for input dependence in our
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS