There has been growing interest in parameter-efficient methods to apply pre-trained language models to downstream tasks. Building on the PROMPTTUNING approach of Lester et al. (2021), which learns task-specific soft prompts to condition a frozen pre-trained model to perform different tasks, we propose a novel prompt-based transfer learning approach called SPOT : S oft P r o mpt T ransfer. SPOT first learns a prompt on one or more source tasks and then uses it to initialize the prompt for a target task. We show that SPOT significantly boosts the performance of PROMPTTUNING across many tasks. More remarkably, across all model sizes, SPOT matches or outperforms standard MODELTUNING (which fine-tunes all model parameters) on the SUPERGLUE benchmark, while using up to 27,000  fewer task-specific parameters. To understand where SPOT is most effective, we conduct a large-scale study on task transferability with 26 NLP tasks in 160 combinations, and demonstrate that many tasks can benefit each other via prompt transfer. Finally, we propose an efficient retrieval approach that interprets task prompts as task embeddings to identify similar tasks and predict the most transferable source tasks for a novel target task. The past few years have seen the rapid development of ever larger pre-trained language models, where it has repeatedly been shown that scaling up the model size is a key ingredient for achieving the best performance (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020). While this trend has continued to push the boundaries of possibility across various NLP benchmarks, the sheer size of these models presents a challenge for their practical application. For 100B+ parameter models, fine-tuning and deploying a separate instance (cid:70) Work done during an internship at Google Research. of the model for each downstream task would be prohibitively expensive. To get around the infeasibility of fine-tuning, Brown et al. (2020) propose PROMPTDESIGN , where every downstream task is cast as a language modeling task and the frozen pretrained model performs different tasks by conditioning on manual text prompts provided at inference time. They demonstrate impressive few-shot performance with a single frozen GPT-3 model, although its performance depends highly on the choice of the prompt (Zhao et al., 2021) and still lags far behind state-of-the-art fine-tuning results. More recent work explores methods for learning soft prompts (Liu et al., 2021b; Qin and Eisner, 2021; Li and Liang, 2021; Lester et al., 2021), which can be seen as additional learnable parameters injected into the language model. Lester et al. (2021) propose PROMPTTUNING , a simple method 5039 Source Prompt Library Source Task Embeddings Keys Values Source Prompts Target Task Embedding Query Source Prompt Value Target Prompt Task A Task B Task C Task A Task B Task C Unsupervised Task Source Prompt Pre-trained Model   Target Prompt Pre-trained Model   Initialization Target Prompt Tuning Source Prompt Tuning  tuned  frozen Initialization Target Task Target Task Target Task Figure 2: An illustration of our generic (left) and targeted (right) SPOT approaches. that learns a small task-specific prompt (a sequence of tunable tokens prepended to each example) for each downstream task during adaptation to condition the frozen language model to perform the task. Strikingly, as model capacity increases, PROMPTTUNING becomes competitive with MODELTUNING , which fine-tunes the entire model on each downstream task. Nevertheless, at smaller model sizes (below 11B parameters), there are still large gaps between PROMPTTUNING and MODELTUNING . In this paper, we propose SPOT : S oft P r o mpt T ransfer, a novel transfer learning approach in the context of prompt tuning. SPOT first trains a prompt on one or more source tasks, and then uses the resulting prompt to initialize the prompt for a target (downstream) task. Our experiments show that SPOT offers significant improvements over PROMPTTUNING across tasks and model sizes. For instance, on the SUPERGLUE benchmark (Wang et al., 2019b), we obtain +10.1 and +2.4 point average accuracy improvements using the T5 BASE (220M parameter) and T5 XXL (11B parameter) models (Raffel et al., 2020), respectively. More importantly, SPOT is competitive with or outperforms MODELTUNING across all model sizes (see Figure 1). Motivated by these results, we investigate transferability between tasks, through the lens of soft task prompts. Our goal is to answer two questions: (a) For a given target task, when does initializing the prompt from a source task boost performance? (b) Can we use task prompts to efficiently predict which source tasks will transfer well onto a novel target task? To answer (a), we conduct a systematic study of the T5 model using 26 NLP tasks in 160 combinations of source and target tasks. Our results indicate that many tasks can benefit each other via prompt transfer. To address (b), we interpret the learned task prompts as task embeddings to construct a semantic space of tasks and formalize the similarity between tasks. We design an efficient retrieval algorithm that measures task embedding similarity, allowing practitioners to identify source tasks that will likely yield positive transfer. To summarize, our main contributions are: (1) We propose SPOT , a novel prompt-based transfer learning approach, and show that scale is not necessary for PROMPTTUNING to match the performance of MODELTUNING ; on SUPERGLUE , SPOT matches or beats MODELTUNING across all model sizes. (2) We conduct a large-scale and systematic study on task transferability, demonstrating conditions under which tasks can benefit each other via prompt transfer. (3) We propose an efficient retrieval method that interprets task prompts as task embeddings to construct a semantic space of tasks, and measures task embedding similarity to identify which tasks could benefit each other. (4) To facilitate future work on prompt-based learning, we will release our library of task prompts and pretrained models, and provide practical recommendations for adapting our library to NLP practitioners at https://github.com/google-research/ prompt-tuning/tree/main/prompt_tuning/ spot . To improve performance of PROMPTTUNING on a target task, SPOT introduces source prompt tuning , an intermediate training stage between language model pre-training and target prompt tuning (Fig-ure 2, left), to learn a prompt on one or more source tasks (while still keeping the base model frozen), which is then used to initialize the prompt for the target task. 1 Our approach retains all the computational benefits of PROMPTTUNING : for each target task, it only requires storing a small task-specific prompt, enabling the reuse of a single frozen pretrained model across all tasks. In this section, we present a generic SPOT approach where a single transferred prompt is reused for all target tasks. In 3, we explore a targeted approach that retrieves different source prompts for different target tasks. Our frozen models are built on top of the pretrained T5 checkpoints of all sizes: SMALL , BASE , LARGE , XL , XXL with 60M, 220M, 770M, 3B, and 11B parameters, respectively. In our experiments with SPOT , we leverage the LM adapted version of T5 2 , which was found to be easier to optimize for PROMPTTUNING (Lester et al., 2021). PROMPTTUNING : The vanilla prompt tuning approach of Lester et al. (2021), where an indepen-dent prompt is directly trained on each target task. MODELTUNING & MULTI-TASKMODELTUNING : We compare prompt tuning approaches to MODELTUNING , the standard fine-tuning approach (Devlin et al., 2019; Raffel et al., 2020), where all model parameters are fine-tuned on each target task separately. For an apples-to-apples comparison, we include MULTI-TASKMODELTUNING , a more competitive baseline that first fine-tunes the entire model on the same mixture of source tasks used for SPOT before fine-tuning it on individual target tasks. 3 2.1.2 Evaluation datasets We study downstream performance on a diverse set of tasks from the GLUE (Wang et al., 2019c) and 1 The target task can be treated as one of the source tasks being mixed together. 2 T5 1.1 checkpoints trained for an additional 100K steps using the prefix LM objective (Raffel et al., 2020), available at https://github.com/google-research/ text-to-text-transfer-transformer/blob/main/released_checkpoints.md 3 In preliminary experiments, we found that using the original version of T5 1.1 (which was pre-trained exclusively on span corruption) for model tuning approaches results in better performance than using the LM adapted version. We therefore report results corresponding to the original T5 1.1 for MODELTUNING and MULTI-TASKMODELTUNING . SUPERGLUE (Wang et al., 2019b) benchmarks. 4 We train for a fixed number of steps and report results on the validation set associated with each dataset. 5 2.1.3 Data for source prompt tuning As with language model pre-training, the choice of training data is crucial for successful prompt transfer. To investigate the impact of source training data on downstream performance, we compare a diverse set of source tasks. A single unsupervised learning task: We first consider training the prompt on a fraction of the C4 (Colossal Clean Crawled Corpus) dataset (Raf-fel et al., 2020) using the prefix LM objective discussed in Raffel et al. (2020). Although this task was used to pre-train our frozen T5 models already, it could still be helpful for learning a general-purpose prompt. A single supervised learning task: Alternatively, we can train the prompt using a supervised task. We use either MNLI (Williams et al., 2018) or SQUAD (Rajpurkar et al., 2016) as a single source task. MNLI was shown to be helpful for many sentence-level classification tasks (Phang et al., 2019), while SQUAD was found to generalize well to QA tasks (Talmor and Berant, 2019). A multi-task mixture: So far, we have considered using a single source task. An alternative approach is multi-task training. Within T5 's unified text-to-text framework, this simply corresponds to mixing different datasets together. We explore mixing datasets from different NLP benchmarks or families of tasks, including GLUE , SUPERGLUE , natural language inference ( NLI ), paraphrasing/semantic similarity, sentiment analysis, question answering ( QA ) on MRQA (Fisch et al., 2019), commonsense reasoning on RAINBOW (Lourie et al., 2021), machine translation, summarization, and natural lan-4 These datasets include grammatical acceptability judgments ( COLA (Warstadt et al., 2019)), sentiment analysis ( SST-2 (Socher et al., 2013)), paraphrasing/semantic similarity ( MRPC (Dolan and Brockett, 2005), STS-B (Cer et al., 2017), QQP (Iyer et al., 2017)), natural language inference ( MNLI (Williams et al., 2018), QNLI (Wang et al., 2019c), RTE (Dagan et al., 2005, et seq.), CB (De Marneffe et al., 2019)), coreference resolution ( WSC (Levesque et al., 2012)), sentence completion ( COPA (Roemmele et al., 2011)), word sense disambiguation ( WIC (Pilehvar and Camacho-Collados, 2019)), and question answering ( MULTIRC (Khashabi et al., 2018), RECORD (Zhang et al., 2018), BOOLQ (Clark et al., 2019)). We exclude the problematic WNLI (Levesque et al., 2012) dataset from GLUE , following Devlin et al. (2019). guage generation on GEM (Gehrmann et al., 2021). 6 We create a mixture of source tasks from each of the NLP benchmarks/families of tasks above, and a mixture comprising all datasets (C4 + 55 labeled datasets), using the examples-proportional mixing strategy in Raffel et al. (2020) with an artificial dataset size limit K = 2 19 examples. We closely follow the training procedure in Lester et al. (2021). Specifically, the only new parameters introduced during both source and target prompt tuning are a shared prompt   RL  E prepended to each (embedded) input sequence, where L , E are the prompt length and the embedding size, respectively. In all cases, we set L = 100 tokens and tune the prompt for a fixed number of steps S . 7 While S is set to 30K in Lester et al. (2021), we find that additional tuning is helpful on large datasets. As such, we set S to 2 18 = 262 , 144 , following Raffel et al. (2020), with the exception of ablation experiments (rows   longer tuning) in Table 1 which use S = 30 K. For source prompt tuning, the prompt token embeddings are initialized from sampled vocabulary (i.e., the 5,000 most common tokens). During target prompt tuning, we save a checkpoint every 500 steps and report results on the checkpoint with the highest validation performance. Appendix C contains training details for PROMPTTUNING and model tuning approaches. We compare the results of SPOT and other approaches in Table 1 and Figure 1. Below, we summarize and analyze each of our findings in detail. SPOT significantly improves performance and stability of PROMPTTUNING : Our results on the GLUE and SUPERGLUE benchmarks with T5 BASE (Table 1) suggest that prompt transfer provides an effective means of improving performance for PROMPTTUNING . For example, the best-performing variant of SPOT outperforms the vanilla PROMPTTUNING approach on both GLUE and SUPERGLUE by a substantial margin, obtaining +4.4 and +10.1 point average accuracy improvements, respectively. Our 6 See Appendix B for details about datasets. 7 We use the Adafactor optimizer (Shazeer and Stern, 2018) with default parameters except with a constant learning rate of 0.3, weight decay of 1 e  5 , and parameter scaling turned off. We train with a batch size of 32. The dropout probability is always kept at 0 . 1 . All of our models are implemented using JAX (Bradbury et al., 2018) and FLAX (Heek et al., 2020). ablation study indicates that longer tuning is also an important ingredient for achieving the best performance, and is complementary to prompt transfer. Additionally, when longer tuning is omitted, we observe that SPOT improves stability across runs. Within SPOT , we can compare the effectiveness of different source mixtures (see Table 1). Source prompt tuning on GLUE performs best on both GLUE and SUPERGLUE , obtaining average scores of 82.8 and 73.2, respectively. 8 Interestingly, unsupervised source prompt tuning on C4 (the same task used to pre-train our frozen models) still yields considerable improvements, even outperforming using SUPERGLUE for SUPERGLUE tasks. Using MNLI or SQUAD as a single source dataset is also particularly helpful across target tasks. Other source mixtures can lead to significant gains, with some families of tasks (e.g., NLI and paraphrasing/semantic similarity) showing more benefit than others. Mixing all the datasets together does not yield the best results, possibly due to task interference/negative transfer issues, where achieving good performance on one or more source tasks can hurt performance on a target task. SPOT helps close the gap with MODELTUNING across all model sizes: Figure 1 shows our SUPERGLUE results across model sizes (see Appendix A for full results). As shown in Lester et al. (2021), PROMPTTUNING becomes more competitive with scale, and at the XXL size, it nearly matches the performance of MODELTUNING . However, at smaller model sizes, there are still large gaps between the two approaches. We show that SPOT helps close these gaps and even exceeds MODELTUNING 's performance by a large margin at several model sizes, while retaining all the computational benefits conferred by PROMPTTUNING . Finally, at the XXL size, SPOT achieves the best average score of 91.2, +1.1 points better than the strong MULTI-TASKMODELTUNING baseline, despite having 27,000  fewer task-specific parameters. As a final test of SPOT 's effectiveness, we submitted our XXL model's predictions to the SUPERGLUE leaderboard, achieving a score of 89.2. This far exceeds all previous submissions using parameter-efficient adaptation, such as GPT-3 (71.8), and almost matches fully fine-tuned T5 XXL (89.3), 9 despite tuning 27,000  fewer parameters. To the best of our knowledge, SPOT is the first parameter-efficient adaptation approach that is competitive with methods that tune billions of parameters. See Appendix D for details. So far, we have seen that soft prompt transfer can significantly boost the performance of prompt tuning, but it is critical to pick the right source tasks for transfer. For instance, through an extensive search, we found that GLUE and MNLI provide excellent source tasks for transferring to individual GLUE and SUPERGLUE tasks. But what about a resource-constrained scenario where a user is not able to exhaustively search over a set of source tasks? Can we predict which tasks will best transfer onto a novel target task without testing them one by one? To investigate this, we conduct a large-scale empirical study with 26 NLP tasks. We first measure transferability across all task combinations (3.1). Next, we show that by interpreting task prompts as task embeddings, we can construct a semantic space of tasks, wherein similar tasks cluster together (3.2). Based on this observation, we pro-9 Note that the T5 submission uses the original version of T5 (which was pre-trained on a multi-task mixture of unsupervised and supervised tasks) while we use T5 1.1 (which was pre-trained on C4 only without mixing in supervised tasks). pose a retrieval algorithm (3.3) that leverages task embedding similarity to choose which source tasks to use for a given novel target task (Figure 2, right). Our proposed approach can eliminate 69 % of the source task search space while keeping 90 % of the best-case quality gain. We study a diverse set of 16 source datasets and 10 target datasets (see Table 2). 10 We consider all 160 possible source-target pairs, and perform transfer from each source task to each target task. All source tasks are data-rich or have been shown to yield positive transfer in prior work. To simulate a realistic scenario, we use low-resource tasks (less than 10K training examples) as target tasks. 11 10 Beyond the datasets from 2, we use DOCNLI (Yin et al., 2021), YELP -2 (Zhang et al., 2015), CXC (Parekh et al., 2021), DROP (Dua et al., 2019), WINOGRANDE (Sakaguchi et al., 2020), HELLASWAG (Zellers et al., 2019), COSMOSQA (Huang et al., 2019), RACE (Lai et al., 2017), and CR (Hu and Liu, 2004). 11 The source tasks comprise one unsupervised task ( C4 ) and 15 supervised tasks covering natural language inference ( NLI ), paraphrasing/semantic similarity, sentiment analysis, question answering ( QA ), and commonsense reasoning. The target tasks additionally include grammatical acceptability, word sense disambiguation, and coreference resolution. To limit computational costs, we use T5 BASE in all of our task transferability experiments. We perform 262 , 144 prompt tuning steps on each source task. The prompt checkpoint with the highest source task validation performance is selected to initialize prompts for target tasks. Since the target datasets are small, we only perform 100K prompt tuning steps on each target task. We repeat each experiment three times with different random seeds. Other training details match 2.1.4. Tasks benefiting each other via prompt transfer: Figure 3 shows a heatmap of our results (see Appendix E for full results). In many cases, prompt transfer provides a significant gain on the target task. The transfer MNLI  CB yields the largest relative error reduction of 58.9% (from an average score of 92.7 to 97.0), followed by MNLI  COPA (29.1%) and RECORD  WSC (20.0%). Using the best source prompt (out of 48) for each target task dramatically improves the average score across our 10 target tasks from 74.7 to 80.7. Overall, our results show effective transfer from large source tasks that involve high-level reasoning about semantic relationships among sentences (e.g., MNLI ), or when the source and target tasks are similar (e.g., CXC  STS-B ). Interestingly, positive transfer can occur between relatively dissimilar tasks (e.g., RECORD  WSC , SQUAD  MRPC , CXC  WIC ). 12 3.2 Defining task similarity through prompts Since only prompt parameters are updated during prompt tuning on specific tasks, the learned prompts likely encode task-specific knowledge. This suggests that they could be used to reason about the nature of tasks and their relationships. To 12 Table 7 in Appendix E contains more cases. test this idea, we interpret task prompts as task embeddings and construct a semantic space of tasks. More concretely, we define a task's embedding as the prompt checkpoint after training for 10K steps on that task. 13 Note that using early checkpoints allows for quick computation of task embeddings for novel target tasks. We estimate the similarity between two tasks t 1 , t 2 by measuring the similarity between their corresponding task embeddings e 1 , e 2 , using the following metrics: COSINESIMILARITY OFAVERAGETOKENS : We compute the cosine similarity between the average pooled representations of the prompt tokens: sim ( t 1 , t 2 ) = cos ( 1 L (cid:88) i e 1 i , 1 L (cid:88) j e 2 j ) , where e 1 i , e 2 j denote the respective prompt tokens of e 1 , e 2 , and cos denotes the cosine similarity. PER-TOKENAVERAGECOSINESIMILARITY : We compute the average cosine similarity between every prompt token pair ( e 1 i , e 2 j ) : sim ( t 1 , t 2 ) = 1 L 2 (cid:88) i (cid:88) j cos ( e 1 i , e 2 j ) . 13 Our preliminary experiments with other checkpoint alternatives (in the range 1K to 100K) yielded worse performance. We also found that measuring task similarity using task embeddings derived from a fixed prompt checkpoint (10K steps) gave better results than those derived from the best-performing prompt checkpoint per task. This suggests that prompts trained for a differing number of steps may be less directly comparable than those trained for the same duration. Task embeddings capture task relationships: Figure 4 shows a hierarchically-clustered heatmap of cosine similarities between the task embeddings using the COSINESIMILARITY OFAVERAGETOKENS metric. 14 We observe that our learned task embeddings capture many intuitive task relationships. Specifically, similar tasks group together into clusters, including QA ( SQUAD , RECORD , and DROP ; MULTIRC and BOOLQ ), sentiment analysis ( YELP -2 , SST-2 , and CR ), NLI ( MNLI and CB ; DOCNLI and RTE ), semantic similarity ( STS-B and CXC ), paraphrasing ( MRPC and QQP ), and commonsense reasoning ( WINOGRANDE , HELLASWAG , and COSMOSQA ). We note that QNLI , which is an NLI task built from the SQUAD dataset, is not closely linked to SQUAD ; this suggests that our task embeddings are more sensitive to the type of task than domain similarity. Interestingly, they also capture the unintuitive case of RECORD 's high transferability to WSC . Additionally, task embeddings that are derived from different prompts of the same task have high similarity scores (see Appendix F). We leverage our task embeddings to predict and exploit task transferability. Specifically, we explore methods to predict the most beneficial source tasks for a given target task and then make use of the source task prompts to improve performance on the target task. To enlarge our set of source prompts, we use the prompts from each of the three different prompt tuning runs on each source task, resulting in 48 source prompts. Given a target task t with task embedding e t , we rank all the source prompts  s with associated embeddings e s in descending order by similarity, sim ( e s , e t ) . We denote the ranked list of source prompts as  s r , where r denotes the rank ( r = 1 , 2 , . . . , 48) . We experiment with three methods for using the ranked source prompts: BEST OFTOPk : We select the topk source prompts and use each of them individually to initialize the target prompt. This procedure requires prompt tuning k times on the target task t . The best individual result is used for evaluating the effectiveness of this method. 14 To obtain the highest resolution of similarity between two tasks, we use the average of cosine similarities between their task embeddings obtained with all the three different prompt tuning runs (9 combinations). source prompts (cid:80) kr =1  r  s r so that we only perform prompt tuning on the target task t once. The weights  r are computed as:  r = sim ( e s r , e t ) (cid:80) kl =1 sim ( e s l , e t ) , where e s r denotes the corresponding task embedding of  s r . TOPk MULTI-TASKMIXTURE : We first identify the source tasks whose prompts are in the topk prompts and mix their datasets and the target dataset together, using the examples-proportional mixing strategy of Raffel et al. (2020). Then, we perform source prompt tuning on this multi-task mixture and use the final prompt checkpoint to initialize the target prompt. We report the average score across all target tasks achieved by each method. For comparison, we measure the absolute and relative improvements over BASELINE prompt tuning on each target task from scratch (i.e., without any prompt transfer). 15 Additionally, we include ORACLE the oracle results achieved by a brute-force search to identify 15 For each target task t , we report the average and standard deviation of performance across three prompt tuning runs. Correlation between task similarity and task transferability: Figure 5 shows how the relative error reduction on a target task changes as a function of the similarity between the source and target task embeddings. Overall, we observe a significant positive correlation between task embedding similarity and task transferability on four (out of 10) target tasks, including STS-B ( p < 0 . 001 ), CB ( p < 0 . 001 ), WSC ( p < 0 . 01 ), and RTE ( p < 0 . 05 ), while it is less significant on the other tasks. 16 In some cases (e.g., on BOOLQ ), we observe a large relative error reduction (19.0%, achieved by a source prompt of MNLI ) despite a low cosine similarity (0.4). This suggests that factors other than task similarity (data size, task difficulty, domain similarity, etc.) may also play a role in determining transferability. Retrieving targeted source tasks via task embeddings is helpful: Table 3 compares different methods for identifying which source prompts could be beneficial for a given target task. Overall, our results show the effectiveness of BEST OFTOPk . Simply choosing the source prompt with the highest task embedding similarity to the target task using PER-TOKENAVERAGECOSINESIMILARITY improves over the baseline by a large margin (from an average score of 74.7 to 76.7, a 12.1% average relative error reduction). Trying all the top-3 (out of 48) source prompts for each target task yields an average score of 77.5. With larger values of k , we can retain most of the benefits of oracle selection (80% of the gain in terms of average score with k = 9 and 90% with k = 15 ), while still eliminating over 2/3 of the candidate source prompts. TOPk WEIGHTEDAVERAGE has similar average performance to BEST OFTOPk with k = 1 , but achieves lower variance. Thus, this may be an appealing alternative to BEST OFTOPk in scenarios where trying multiple prompt tuning runs on the target task is computationally prohibitive. Finally, TOPk MULTITASKMIXTURE also provides a means of obtaining strong performance with an average score of 77.8, even outperforming BEST OFTOPk with k  3 . Large-scale pre-trained language models have been shown to exhibit remarkable performance on many NLP tasks (Devlin et al., 2019; Liu et al., 2019b; Yang et al., 2019; Lan et al., 2020; Raffel et al., 2020; Brown et al., 2020; He et al., 2021). To improve practical applicability of these models, early work introduces compression techniques (Sanh et al., 2019; Jiao et al., 2020; Fan et al., 2020; Sanh et al., 2020) to obtain lightweight models. Other work explores updating only small parts of the model (Za-ken et al., 2021) or task-specific modules, such as adapters (Houlsby et al., 2019; Karimi Mahabadi et al., 2021) or low-rank structures (Mahabadi et al., 2021; Hu et al., 2021), while keeping the rest of the model fixed. Recently, Brown et al. (2020) demonstrate impressive few-shot performance with PROMPTDESIGN , where their model is conditioned on a manual text prompt at inference time to perform different tasks. Several efforts have since focused on developing prompt-based learning approaches with carefully handcrafted prompts (Schick and Schtze, 2021), prompt mining and paraphrasing (Jiang 5046 et al., 2020b), gradient-based search for improved prompts (Shin et al., 2020), and automatic prompt generation (Gao et al., 2021). The use of hard prompts, however, was found to be sub-optimal and sensitive to the choice of the prompt (Zhao et al., 2021; Liu et al., 2021b). As such, more recent work has shifted toward learning soft prompts (Liu et al., 2021b; Qin and Eisner, 2021; Li and Liang, 2021; Lester et al., 2021), which can be seen as learnable parameters injected into the model. We refer readers to Liu et al. (2021a) for a recent survey on prompt-based learning research. In concurrent work, Gu et al. (2021) also explore the effectiveness of prompt transfer. Their method uses hand-crafted pre-training tasks tailored to specific types of downstream tasks, being less extensible to novel downstream tasks. In contrast, we use existing tasks as source tasks and show that prompt transfer can confer benefits even when there are mismatches (e.g., in task type or input/output format) between the source and target. Task transferability We also build on existing work on task transferability (Wang et al., 2019a; Liu et al., 2019a; Talmor and Berant, 2019; Pruk-sachatkun et al., 2020; Vu et al., 2020, 2021). Prior work shows effective transfer from data-rich source tasks (Phang et al., 2019), those that require complex reasoning and inference (Pruksachatkun et al., 2020), or those that are similar to the target task (Vu et al., 2020). There have also been efforts to predict task transferability (Bingel and Sgaard, 2017; Vu et al., 2020; Poth et al., 2021). Vu et al. (2020) use task embeddings derived from either the input text or the diagonal Fisher information matrix of the model, while Poth et al. (2021) explore adapter-based alternatives. Here, our use of the same model (without task-specific components) with a unifying text-to-text format allows us to more easily model the space of tasks. Additionally, prompt-based task embeddings are comparatively cheaper to obtain. As other parameter-efficient adaptation methods (see 4) may outperform PROMPTTUNING in specific situations, it would be interesting to test whether an approach similar to SPOT could extend successfully to these methods. At the same time, we believe that PROMPTTUNING has its own merits. As pre-trained language models become larger and larger, some advantages of PROMPTTUNING over other methods are: (1) Among current methods with learnable parameters, PROMPTTUNING is the most parameter efficient, requiring less than 0.01% task-specific parameters for most model sizes. (2) PROMPTTUNING is simpler than other methods, as it does not modify the internal model architecture (cf. the PREFIXTUNING method of Li and Liang (2021), which adds a prefix to each layer of both the Transformer encoder and decoder); as such, PROMPTTUNING allows mixed-task inference and facilitates transfer learning between tasks. (3) As model capacity increases, PROMPTTUNING becomes more competitive with MODELTUNING ; to the best of our knowledge, this has not been shown for other methods. (4) Soft prompts could possibly be interpreted as natural language instructions. Additionally, since our prompt-based task embedding approach does not capture all of the factors that influence task transferability, we leave further exploration of other task embedding methods to future work. In this paper, we study transfer learning in the context of prompt tuning. We show that scale is not necessary for PROMPTTUNING to match the performance of MODELTUNING . On SUPERGLUE , our SPOT approach matches or even exceeds the performance of MODELTUNING by a large margin across model sizes while being more parameter-efficient. Our large-scale study on task transferability indicates that tasks can benefit each other via prompt transfer in various scenarios. Finally, we demonstrate that task prompts can be interpreted as task embeddings to formalize the similarity between tasks. We propose a simple yet efficient retrieval approach that measures task similarity to identify which source tasks could confer benefits to a novel target task. Taken as a whole, we hope that our work will spur more research into prompt-based transfer learning. We thank Mohit Iyyer, Sebastian Ruder, Kalpesh Krishna, Thang Luong, Quoc Le, and the members of the Descartes team and the UMass NLP group for helpful discussion and feedback. We would also like to thank Grady Simon, Lucas Dixon, Slav Petrov, Nader Akoury, Haw-Shiuan Chang, Katherine Thai, Marzena Karpinska, and Shufan Wang for their comments on this manuscript. Finally, we are grateful to Vamsi Aribandi for his work on preprocessing several datasets used in our experiments.
abstain abstain abstain result abstain objective objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective abstain result result abstain objective objective abstain objective abstain method result abstain objective method objective objective objective other abstain abstain abstain method abstain method method abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain other method abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method method abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain method abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other abstain other other objective method other other other abstain other abstain method abstain abstain abstain abstain abstain method method result result abstain objective objective method other other other