The goal of meta-learning is to learn to adapt to a new task with only a few labeled examples. Inspired by the recent progress in large language models, we propose in-context tuning (ICT), which recasts task adaptation and prediction as a simple sequence prediction problem: to form the input sequence, we concatenate the task instruction, labeled in-context examples, and the target input to predict; to meta-train the model to learn from in-context examples, we fine-tune a pre-trained language model (LM) to predict the target label given the input sequence on a collection of tasks. We benchmark our method on two collections of text classification tasks: LAMA and BinaryClfs. Compared to MAML which adapts the model through gradient descent, our method leverages the inductive bias of pre-trained LMs to perform pattern matching, and outperforms MAML by an absolute 6% average AUC-ROC score on BinaryClfs, gaining more advantage with increasing model size. Compared to non-fine-tuned in-context learning (i.e. prompting a raw LM), in-context tuning meta-trains the model to learn from in-context examples. On BinaryClfs, ICT improves the average AUC-ROC score by an absolute 10%, and reduces the variance due to example ordering by 6x and example choices by 2x. Few-shot learning (FSL) refers to a system's ability to quickly adapt to new tasks when very few labeled examples are available for training. FSL is a key feature of human learning (Lake et al., 2016), but current machine learning systems often rely on large amounts of labeled training data (Silver et al., 2016; He et al., 2016; Adiwardana et al., 2020). Recently, prompting large pre-trained language models (LMs) for FSL has achieved remarkable progress (Brown et al., 2020; Schick and Schtze,  Work done during summer internship at AWS AI. 2021a). LM prompting with in-context learning reduces the task learning and predict process to a simple sequence prediction problem. To perform a new task, Brown et al. (2020) prompt a raw LM (i.e., a pre-trained LM not fine-tuned on any labeled data) with the concatenation of the task instruction, some input-output examples, and the target input to be predicted on; then they extract the answer from the LM's continuation of the concatenated sequence (Figure 1 left). For example, to coax the model into performing sentiment classification on the target input  This movie is a waste of time , we prompt the LM with the sequence  I like the movie! Positive review? Yes. Horrible Movie! Positive review? No. This movie is a waste of time. Positive review? ___ , and predict positive if the next word is more likely to be  Yes  rather than  No . However, raw LMs are not optimized for in-context FSL during pre-training, and exhibit undesirable behavior when used for FSL. For example, Zhao et al. (2021) observed that LMs suffer from the recency bias, which assigns higher probability to labels that appear closer to the target input. As a result, the accuracy becomes extremely sensitive to the ordering of the in-context examples. Previous work has also shown that prompting raw LMs is often oversensitive to example choices and instruction wording (Schick and Schtze, 2021a; Jiang et al., 2020; Gao et al., 2021; Liu et al., 2021). We address this weakness through a meta-learning lens and directly fine-tune the LM for FSL. Under the meta-learning framework, we meta-train a model to learn to adapt to new tasks from a few examples on a wide range of tasks, so that it learns to leverage the few-shot examples to adapt to new tasks at test time. Since LM prompting already reduces the task learning and predict process to a simple sequence prediction problem, we meta-train a LM by directly fine-tuning it to optimize for this sequence prediction problem on a wide range of tasks (Figure 1 left). Since we fine-719 Instruction x1 y1 x' Y' x2 y2 Meta-Update via Gradient Descent In-Context Tuning  :=   Few-shot Adaptation via In-context Learning MAML y1 x1 Instruction y2 x2 Instruction   :=   y' x' Instruction Calculate loss with   Meta-Update: Optimize to minimize the loss. tune our model to learn in-context learning, we call our approach in-context tuning (ICT). Unlike optimization-based meta learning approaches such as MAML (Finn et al., 2017), in-context tuning adapts to new tasks through in-context learning where model parameters are frozen, thus it avoids the challenging nested optimization problem in MAML (Figure 1). We benchmark our algorithm on LAMA (Petroni et al., 2019), a dataset for testing models' factual knowledge, and BinaryClfs (Zhong et al., 2021), a wide range of binary classification tasks each annotated with a few language descriptions of the task. Compared to prompting raw LMs, in-context tuning improves performance by 7.6 Precision@1 points on LAMA and 10.6% AUC-ROC score on BinaryClfs. In addition, in-context tuning mitigates the over-sensitivity of raw LM prompting, significantly reducing the variance of the performance with respect to example ordering (by 68% on LAMA and 83% on BinaryClfs), example choices (by 56% on LAMA and 40% on BinaryClfs), and instruction wording (by 19% on LAMA). Our approach also out-performs MAML, which adapts the model by gradient descent on a few examples and learns an initialization that can adapt to a new task through a few gradient steps (Finn et al., 2017; Nichol et al., 2018). Since our approach better takes advantage of the inductive bias of LMs to extrapolate from in-context examples, our approach out-performs first-order MAML by 2.8 points on LAMA and 5.1 points on BinaryClfs, with increasing advantage as models become larger. Given the empirical effectiveness of in-context tuning (Section 4.1), we conjecture that the few-shot learning potential of large LMs (e.g., GPT-3) may be broadly underestimated if prompted without any direct optimization for FSL. We also conjecture that in-context tuning can mitigate various undesirable properties of LM prompting, such as over-sensitivity to example ordering, example choices, and instruction wording (Section 4.2). We introduce the problem setup (Section 2.1), describe our in-context tuning algorithm (Section 2.2), compare our algorithm to gradient-based adaptation methods (Section 2.3) and other baselines (Sec-tion 2.4). We focus on the few-shot classification problem, where the model first learns from a set of training tasks T  T train , each associated with its natural language instructions IT and a large amount of task input-output examples DT = { ( x iT , y iT ) } (see Figure 1 left for examples). At test time, we ask the model to learn a new task  T given its instruction and only a few ( K ) labeled examples, i.e. S  T  D  T , | S  T | = K . We denote the task input to be predicted at test time as x target  T . Note that task input is different from model input. For example, on the left panel of Figure 1, the task input is  Good movie!  while the model 720 input can be a concatenation of the instruction, task inputs and task outputs. In-context tuning directly optimizes pre-trained LMs with the few-shot in-context learning objective (Brown et al., 2020): task-agnostic LMs are meta-trained to perform few-shot in-context learning on a wide variety of training tasks. Similar to in-context learning, LMs trained with in-context tuning adapt to a new task by using few-shot training examples as the input prefix. Formally, during meta-training, we build the model input by concatenating the task instruction IT , task input-output pairs ST  DT , and the task input x target T 1 to be classified. We then fine-tune a pre-trained LM to predict y target T and hope that the model learns to use the in-context examples ST . Here is the few-shot in-context tuning objective L : LT (  ) := (cid:88) ( x tgt T ,y tgt T )  DT [  log p  ( y tgt T | x tgt T , ST , IT )] (1) L (  ) := (cid:88) T  T train LT (  ) (2) To adapt to a new task  T at test time, we directly concatenate the few-shot examples S  T with the instruction I  T and the target task input x target  T to be classified to form the model input, and ask the model to predict its corresponding output. No gradient update is performed during adaptation. We compare in-context tuning with two classical few-shot learning methods: multi-task fine-tuning (instruction tuning + fine-tuning) and MAML. Both methods adapt the model parameters to new tasks by gradient descent on few-shot examples. Instruction Tuning + Fine-tuning (InsT + FT) We extend the recent work on zero-shot instruction tuning (Wei et al., 2021) to the FSL setting as a multi-task fine-tuning baseline. During meta-training, the model is optimized to predict the task output given the task instruction and the task input on a wide range of tasks (Zhong et al., 2021). Formally, we train the model parameter  to predict y iT given IT  x iT , where  is shared across all tasks and  represents the concatenation operation. During the few-shot adaptation phase, the model is presented with a new task  T , its natural language instruction I  T and a small set of ( K ) task input-output examples S  T = { ( x i  T , y i  T ) | i  [ K ] } . We then fine-tune the model to predict the task output y i  T from the new task given I  T  x i  T and update  with a few gradient steps to get   T . Finally, we use the updated model   T to predict the output from the task input x target  T and the instruction I  T under the test task  T . MAML The few-shot adaptation stage of MAML is the same as instruction tuning + fine-tuning, where we update the model parameters (ini-tialized with  ) by gradient descent on K examples S  T  D  T . However, during meta-training, MAML aims to learn a task-agnostic model initialization  such that,  T , which is to be found by initializing with  and performing gradient descent on ST , would lead to good performance (Finn et al., 2017). Therefore, MAML involves two levels of optimization, an inner optimization to learn  T given  and ST  DT , and an outer optimization to learn  given  T . Due to the bi-level structure in this optimization problem, MAML has been found to be empirically unstable, sensitive to hyperparameters, and computationally expensive (Finn et al., 2017; Nikolaev et al., 2020). Even worse, few-shot task adaptation is known to be highly sensitive to optimization hyperparameters (Antoniou et al., 2019), while a large labeled validation set for hyperparameter tuning may not be available under a FSL setting (Perez et al., 2021). In comparison, in-context tuning simplifies the two-stage process of (1) few-shot task adaptation and (2) task-specific prediction as one sequence prediction problem, where task-specific examples are concatenated to the model input to provide information about the task. Hence, in-context tuning removes the bi-level optimization during meta-training, which can be empirically unstable and expensive. Additionally, since model weights are frozen during task adaptation, it is not sensitive to adaptation hyperparameters. Raw In-context Learning (Raw IC-L) We directly evaluate a raw LM on a new task using the same evaluation set-up for in-context tuning, without fine-tuning the LM on any labeled data. Instruction Tuning (InsT) The model learns to predict the target output only based on the instruction and the target input. Only the instruction is available during the adaptation phase, and this setup is also known as zero-shot learning. We categorize all approaches in our paper based on their meta-training objective and how they use task-specific examples in Table 1. In-context tuning is the only method that directly optimizes the FSL objective without gradient-based adaptation. We experiment with two meta-datasets that contain a wide range of tasks, LAMA and BinaryClfs. Each task is associated with several different natural language descriptions, and we call them instructions for convenience, even though some of them are realized as questions. LAMA LA nguage M odel A nalysis (Petroni et al., 2019) is a dataset that tests the factual and commonsense knowledge learned by LMs. In our experiments, we use the TREx-UHN portion of LAMA (Poerner et al., 2020), which consists of (subject, relation, object) triples from Wikidata. LAMA is an entity prediction task, where a model is asked to predict the object entity given the subject entity and the relation. In our experiments, we treat one relation as a task as in Perez et al. (2021). Initial experiments on LAMA showed that LMs take significant advantage of majority label bias (Zhao et al., 2021), where they assign higher probability to object entities that have appeared in the in-context examples, thus inflating the accuracy. To reflect the improvement due to few-shot learning rather than this simple heuristic to copy answers, for all tasks we prune the LAMA dataset so that all object entities appear less than 2.5% of times. Our final filtered LAMA dataset consists of 29 relations (tasks) and 12k (subject, relation, object) examples. We use task instructions from two datasets: LAMA and LPAQA (Jiang et al., 2020). LAMA contains one task instruction for each task, and the auxiliary LPAQA dataset contains on average 10 additional instructions for each LAMA task. We use the same evaluation protocol as in Petroni et al. (2019): 1) the object entity is predicted from a pre-defined vocabulary set of 21k words (each LAMA task is 21k-way classifica-tion); 2) we compute mean precision at one (P@1) for each task, and report the average across tasks. Because LAMA does not have an official train-validation-test split, we use 8-fold cross-validation in our experiments. We randomly partition the 29 tasks into 8 groups of similar sizes. For each cross-validation split, we use six groups for training, one group for validation, and one group for testing. The test sets of the eight folds are disjoint and their union is the set of all tasks. BinaryClfs This dataset contains a wide range of binary cl assi f ication task s , and each task can be described by 1-4 yes/no" questions, which we concatenate to the input context as instructions. There are in total 204 different tasks, and 73 of them are used for testing, which include sentiment classification, topic classification, definition detection, stance classification, etc. We use the same evaluation protocol as in Zhong et al. (2021): 1) we group the tasks by similarity and do not allow training tasks to be similar to testing tasks; 2) we treat Yes answer as the positive class and calculate the AUC-ROC score for each instruction of each task. To fit model inputs (concatenation of in-context examples and task input to classify) within the maximum context length (1024) of our LMs, we leave out five evaluation tasks where the maximum task input length exceeds 230 BPE tokens. We also leave out the spam classification task due to its small test set. BinaryClfs does not come with an official validation set. To perform hyperparameter tuning, for each testing group, we randomly sample another testing group as its validation group. Architecture We use BERT models for LAMA (BERT-Base [110M parameters], BERT-Large [340M] and DeBERTa-XLarge-V2 [900M]) and GPT2 models for BinaryClfs (GPT2-Medium [345M] and GPT2-Large [774M]). We use the Hug-722 LAMA BinaryClfs BERT-Base BERT-Large DeBERTa-xlarge GPT2-M GPT2-L 0-S 1-S 2-S 5-S 0-S 1-S 2-S 5-S 0-S 1-S 2-S 5-S 0-S 5-S 0-S 5-S Raw IC-L 10.3 8.5 10.8 14.1 12.7 12.1 15.4 18.6 11.2 12.6 20.6 23.7 50.5 57.8 51.0 58.3 InsT + FT / 17.5 18.6 20.0 / 21.6 22.6 23.9 / 24.7 25.6 27.0 / 67.0 / 69.4 ICT 14.6 16.3 17.6 19.6 18.0 21.6 23.4 24.3 21.9 26.0 27.5 28.8 62.9 67.4 66.3 69.8 Raw IC-L w/o Ins 1.5 4.9 8.7 12.3 1.4 3.5 7.0 12.5 2.7 13.0 19.5 22.6 / / / / ICT w/o Ins 7.1 14.6 17.0 18.2 9.3 19.4 19.9 22.9 10.6 23.5 26.0 27.6 / / / / Table 2: Few-shot learning accuracy of our in-context tuning approach (ICT) compared to in-context learning with raw LMs (Raw IC-L) and instruction tuning + fine-tuning (InsT + FT). Hyperparameters We select hyperparameters based on few-shot classification accuracy on validation tasks. Our validation tasks and testing tasks are disjoint, so hyperparameter tuning on validation tasks does not use extra labeled examples on the testing tasks (Perez et al., 2021). See Appendix A for the hyperparameters we tuned. Sampling Different instructions and few-shot example choices can lead to different predictions (Section 2.2). At training time, we expose the model to diverse task instructions and few-shot choices by randomly sampling task instructions and few-shot examples for each target example. At test time, we report the average accuracy across task instructions and few-shot choices. Since computing the average across all few-shot choices is intractable (there are combinatorically many distinct few-shot choices), we thus calculate the average accuracy of multiple random samplings of few-shot choices as approximation. In-context tuning out-performs MAML and various baselines on the two text classification meta-datasets (Section 4.1). It also significantly reduces model sensitivity to instruction wording, example choices, and example ordering compared to prompting raw LMs (Section 4.2). In-context tuning improves in-context learning accuracy over raw LMs. We compare ICT with Raw IC-L in Table 2. In-context tuning consistently out-performs raw LM prompting by 7.6 points on LAMA and 10.6 points on BinaryClfs (averaged across model size and number of few-shots). As expected, directly optimizing the few-shot in-context learning objective (Section 2.2) improves the few-shot in-context learning accuracy. Few-shot examples lead to more effective task adaptation. We compare few-shot in-context tuning with instruction tuning (equivalent to 0-shot ICT) in Table 2. Few-shot in-context tuning consistently out-performs instruction tuning on both LAMA and BinaryClfs, with increasing performance gains as number of shots increases. Specifically, we observe that 5-shot in-context tuning out-performs instruction tuning by 6.1 points on LAMA and 4.0 points on BinaryClfs. Results show that demonstration examples besides task instructions facilitate more effective task adaptation. tuning (equivalent to 0-shot ICT) of Table 2, we see that MAML out-performs instruction tuning in most evaluation settings, which indicates that MAML is indeed able to take advantage of the few-shot task examples for task adaptation. However, Table 3 shows that our approach of 5-shot in-context tuning out-performs 5-shot MAML consistently on both datasets with an accuracy gain of 2.8 points on LAMA and 5.1 points on BinaryClfs (averaged across model size). We argue that in-context tuning out-performs MAML because in-context tuning better leverages the existing inductive bias of pre-trained LMs to perform pattern matching with in-context examples. We also compare in-context tuning to the pipeline of instruction tuning + task-specific fine-tuning (Table 2). Surprisingly, fine-tuning an instruction-tuned model on as few as one task-specific example significantly improves task accuracy, without over-fitting to the few labeled examples. We observe that instruction tuning + 1-shot fine-tuning out-performs instruction tuning (equiv-alent to 0-shot ICT) by 3.1 points on LAMA (Ta-ble 2). Our in-context tuning approach performs comparable or better than instruction tuning + fine-tuning, with increasing accuracy gains as models get bigger (Table 2). For DeBERTa-XLarge-v2 (the largest models we use in this work), in-context tuning out-performs InsT + FT across all numbers of shots, achieving an accuracy gain of 1.7 points on LAMA (averaged across all numbers of shots). We conjecture that in-context tuning will be increasingly effective for bigger models that have a stronger inductive bias of pattern matching. In-context tuning reduces the need of task instructions. As coming up with good task instructions can be hard (Schick and Schtze, 2021a; Jiang et al., 2020), we further investigate the effectiveness of in-context tuning without task instructions (Table 2). In-context tuning is effective in the no-instruction setting as well, consistently out-performing raw in-context learning with no instructions by an average margin of 9.5 points on LAMA. Comparing raw in-context learning with (Raw IC-L) and without instructions (Raw IC-L w/o Ins) (Table 2), we observe that task instructions yield the most significant performance gains when model size is relatively small (+2.5 points on BERT-Base, +7.7 points on BERT-Large, only +0.6 points on DeBERTa-xlarge). We conjecture that smaller models may be weaker at inferring patterns LAMA BinaryClfs BB BL GPT2-M GPT2-L Raw IC-L 1.82 2.14 9.26 8.84 ICT 0.66 0.61 1.41 1.58 Table 4: In-context tuning is significantly less sensitive to example ordering compared to in-context learning with raw LMs. from in-context examples alone compared to larger models, which is why instructions yield larger performance gains on smaller models. On BERT-Base and BERT-Large models where task instructions are most helpful, in-context tuning reduces the improvement gain from task instructions from 5.1 points (raw in-context learning) to 1.8 points (aver-aged across BERT-Base and BERT-Large), which indicates that in-context tuning reduces the need of task instructions compared to raw in-context learning. However, we note that instructions still yield performance improvement even if in-context tuning is applied. We analyze the sensitivity of in-context tuning accuracy with respect to example ordering, example choices, and instruction wording, and compare it with prompting raw LMs. Let I denote a random selection of task instruction, ST a random unordered set of few-shot training examples with size K ,  a random permutation of K examples. The accuracy  is a function of these three random variables, i.e.  : ( ST , , I ) (cid:55) [0 , 1] . We can decompose the total variance of  into its variance w.r.t. each of the three random variables, since they are independent (order variance is independent to choice variance because ST is unordered ): Var ST ,,I [  ] = Var I [ EST , [  | I ]] (cid:124) (cid:123)(cid:122) (cid:125) instruction wording variance + EI [ Var ST [ E  [  | I, ST ]]] (cid:124) (cid:123)(cid:122) (cid:125) example choice variance + E I,S T [ Var  [  | I, ST ]] (cid:124) (cid:123)(cid:122) (cid:125) example order variance We analyze each type of variance below. tuning and in-context prompting with raw LMs in Table 4. Results show that in-context tuning is significantly less sensitive to ordering of in-context examples compared to in-context prompting with raw LMs, reducing the sensitivity by 68% on LAMA and 83% on BinaryClfs. In-context tuning is significantly less sensitive to example choices. We compare the variance with respect to example choices for in-context tuning and in-context prompting with raw LMs in Table 5. Results show that in-context tuning is significantly less sensitive to selection of in-context examples compared to in-context prompting with raw LMs across both datasets and all model sizes, reducing the sensitivity by 56% on LAMA and 40% on BinaryClfs (averaged across model sizes). We conjecture that in-context tuning is significantly less sensitive to example ordering and selection because the model is exposed to various example orderings and selections during in-context tuning. In-context tuning is less sensitive to instruction wording. We report the variance with respect to instruction wording for in-context tuning and in-context prompting with raw LMs in Table 6. Results show that in-context tuning is less sensitive to instruction wording compared to in-context prompting with raw LMs in five out of six evaluation settings, reducing the variance by 19% on LAMA (averaged across model size and number of shots). We also observe that in-context tuning is especially effective on task instructions with low accuracy under raw in-context learning. For each task, we compute the Pearson correlation between the raw in-context learning accuracy and the accuracy gain from in-context tuning (over raw in-context learning) on all instructions. On the LAMA dataset, we see a strong negative correlation of -0.563 (aver-aged across all tasks), with p-value < 0.05 on 63% of the tasks. We conjecture that in-context tuning is much less sensitive to instruction wording because the model is exposed to a wide variety of different task instructions during in-context tuning. instructions. We observe that in-context tuning is especially effective on task instructions with low accuracy under instruction tuning. For each task, we compute the Pearson correlation between the instruction tuning accuracy and the accuracy gain from in-context tuning (over instruction tuning) on all instructions. On the LAMA dataset, we see a strong negative correlation of -0.910 (averaged across all tasks), with p-value < 0.01 on 91% of the tasks. We conjecture that in-context tuning is much less sensitive to instruction wording because few-shot in-context examples provide additional task information besides the task instructions. LM Prompting for FSL Pre-trained LMs can be used to perform various FSL tasks when prompted with a natural language task instruction and several task examples (Radford et al., 2019; Brown et al., 2020; Schick and Schtze, 2021b; Li and Liang, 2021; Lester et al., 2021; Qin and Eisner, 2021). However, prompting pre-trained LMs directly for FSL is known to be sensitive to various artifacts, such as the wording of the task instruction and the selection and ordering of few-shot training examples (Schick and Schtze, 2021a; Jiang et al., 2020; Zhao et al., 2021; Gao et al., 2021; Liu et al., 2021). Our work is the first to show that meta-learning with an explicit FSL objective significantly reduces the sensitivity of LM prompting with respect to the in-context examples and instruction wording. Meta-learning for FSL Meta-learning is a widely used technique in NLP to improve cross-domain transfer (Yu et al., 2018; Geng et al., 2019; Holla et al., 2020; Deng et al., 2020) and cross-task transfer (Gu et al., 2018; Bansal et al., 2020; Dou et al., 2019). Existing optimization-based meta-learning methods mostly perform task adap-725 tation by fine-tuning a task-agnostic model on task-specific examples using gradient descent (Finn et al., 2017; Jiang et al., 2019; Nichol et al., 2018). However, fine-tuning on few-shot task examples is sensitive to hyperparameters (Antoniou et al., 2019) and nested optimization during meta-training is often unstable (Nichol et al., 2018; Antoniou et al., 2019; Rajeswaran et al., 2019). In contrast, our approach performs few-shot task adaptation by using task-specific examples as part of the model input while keeping the model parameters frozen and task-agnostic during the adaptation stage. Multi-task Learning In multi-task learning, a single model is trained on the union of training sets of multiple tasks to learn a shared representation (Liu et al., 2019). The multi-task model is then fine-tuned on task-specific examples to adapt to new tasks. Multi-task learning is shown to improve performance on various downstream tasks, especially tasks with small training sets (Khashabi et al., 2020; Ye et al., 2021; Aghajanyan et al., 2021). Compared to meta-learning, multi-task learning does not optimize task adaptation directly. Fine-tuned LMs for Instruction Learning Recent work shows that fine-tuning LMs to learn task instructions on a wide variety of tasks can further leverage the inductive bias of LMs to perform instruction learning (Zhong et al., 2021; Mishra et al., 2021; Wei et al., 2021). Our work is partially inspired by this line of work, but we work under the more generic few-shot meta-learning setting, and show that our approach out-performs both instruction tuning and existing few-shot meta-learning methods (e.g., MAML). While previous work focuses on the accuracy improvement gained from instruction fine-tuning, our work also looks into the well-known over-sensitivity issue of FSL and shows that in-context tuning effectively reduces the sensitivity of FSL with respect to various factors. Concurrent to our work, Min et al. (2021) also explores in-context tuning under more general Seq2Seq tasks. In comparison, our work compares in-context tuning to a meta-learning baseline MAML, and shows that in-context tuning mitigates the well-known oversensitivity issue of LM prompting. Contrary to our paper, Min et al. (2021) finds that in-context tuning under-performs InsT + FT. This might be because they use many more shots (16-shot), which could give gradient-based methods more advantage. Scaling Up and Broader Applications Our work only considers simple binary classification and knowledge retrieval tasks, at most 5 in-context examples, and models with fewer than 1 billion parameters. Nevertheless, it is straightforward to scale up our framework to a wider and more diverse range of general sequence-to-sequence tasks (Ye et al., 2021), more few-shot examples (which requires a longer context size (Dai et al., 2019; Wang et al., 2020)), and larger models (Brown et al., 2020; Kaplan et al., 2020). It is also straightforward to apply in-context tuning to a broader range of scenarios that require adapting to a new setup, e.g., adapting to a new label in classification tasks (Xia et al., 2021), an unseen database in semantic parsing tasks (Suhr et al., 2020; Lee et al., 2021), or a new language pair in machine translation (Gu et al., 2018; Aharoni et al., 2019), etc. Meta-learning for Robustness Our work assumed that the few-shot training examples come from the same distribution as the test examples, but this assumption does not necessarily hold in practice. For example, the test distribution might constitute new input compositions (Lake and Baroni, 2018), rare subgroups (Sagawa et al., 2019), other types of distribution shifts (Hendrycks and Diet-terich, 2019), or even adversarial examples (Kang et al., 2019). More effective meta-learning methods might learn a more robust learning mechanism and combat these generalization challenges. Understanding In-context Learning Many properties of in-context learning are still unknown. Is in-context learning more robust to distribution shift (Lester et al., 2021)? Can we combine in-context learning and gradient learning to get the benefit of both worlds (Wortsman et al., 2021)? In this work, we propose meta-learning via in-context tuning, which recasts the few-shot learning process of task adaptation and task-specific prediction as a simple sequence prediction problem, where few-shot labeled examples are concatenated with the target example to form the model input. In-context tuning out-performs a wide variety of baselines in terms of accuracy, including raw LM prompting, MAML and instruction tuning. Meanwhile, sensitivity study shows that our FSL approach of in-context tuning is significantly 726 less sensitive to few-shot examples and instruction wording compared to raw LM prompting. Given the empirical effectiveness of in-context tuning, we conjecture that the few-shot learning potential of large LMs (e.g., GPT-3) might be broadly underestimated, and that in-context tuning can eliminate well-known artifacts of few-shot LM prompting such as over-sensitivity to example ordering, example selection and instruction wording.
abstain other abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain objective abstain method method method method method abstain abstain abstain abstain method method method abstain method abstain method abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method abstain method abstain method abstain method method method abstain method method method method abstain method method method abstain method method method abstain method abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method other other objective other other other objective other other other other other abstain abstain abstain abstain abstain other method other other method abstain abstain abstain abstain method objective abstain result abstain