Abstract First-order meta-learning algorithms have been widely used in practice to learn initial model parameters that can be quickly adapted to new tasks due to their efficiency and effectiveness. However, existing studies find that meta-learner can overfit to some specific adaptation when we have heterogeneous tasks, leading to significantly degraded performance. In Natural Language Processing (NLP) applications, datasets are often diverse and each task has its unique characteristics. Therefore, to address the overfitting issue when applying first-order meta-learning to NLP applications, we propose to reduce the variance of the gradient estimator used in task adaptation. To this end, we develop a variance-reduced first-order meta-learning algorithm. The core of our algorithm is to introduce a novel variance reduction term to the gradient estimation when performing the task adaptation. Experiments on two NLP applications: few-shot text classification and multi-domain dialog state tracking demonstrate the superior performance of our proposed method. Meta-learning has recently emerged as a promising approach in solving many natural language processing tasks, such as few-shot text classification (Obamuyide and Vlachos, 2019; Bao et al., 2019), low resource language understanding (Gu et al., 2018; Dou et al., 2019; Yu et al., 2020a), and multi-domain dialogue systems (Qian and Yu, 2019; Huang et al., 2020). In particular, model-agnostic meta-learning (MAML) (Finn et al., 2017), a widely-used meta-learning approach, trains an initial model that can be adapted to a new task with a small number of optimization steps and training data. However, MAML requires the computation of second-order derivatives, which can be costly for reinforcement learning and NLP applications. Therefore, numerous computationally-efficient MAML variants (Finn et al., 2017; Li et al., 2017; Nichol et al., 2018; Antoniou et al., 2018; Zintgraf et al., 2019; Song et al., 2020) have been proposed in recent years. First-order meta-learning (Finn et al., 2017; Nichol et al., 2018) is a widely-used method in practice because it is easy to implement, eliminates computationally-intensive second-order derivatives in MAML, and achieves state-of-the-art performance. Although meta-learning including first-order meta-learning has shown promising performances in many applications (Triantafillou et al., 2019), it still somewhat struggles to learn on diverse task distributions (Triantafillou et al., 2020; Rebuffi et al., 2017; Yu et al., 2020c). For first-order meta-learning, it consists of task adaptation and meta updates. Task adaptation aims to obtain a task-specific model for each task by performing several optimization steps based on the current meta model. Then, the meta update aggregates the gradient information of task-specific models to obtain a new meta model. It has been observed in many previous works (Zhao et al., 2018; Karimireddy et al., 2019; Charles and Konecn`y, 2020) that local update methods, including first-order meta-learning, performing multiple optimization steps on local data can lead to overfitting to atypical local data. In the context of first-order meta-learning, due to the large variance of the gradient estimator, task adaptation will drive task-specific models to move away from each other, resulting in that the gradients used in meta update have diverse directions. Furthermore, since the difference in gradient magnitudes will also be large, the task with a much larger gradient in magnitude will dominate the task adaptation. As a result, the meta update will overfit to this dominating task. Similar issues have been studied in multi-task learning: Yu et al. (2020b) showed that conflicting gradients, i.e., two gradients that have a negative cosine similarity, can lead to significantly degraded performance when the difference in gradient magnitudes is large. The above gradient variance issue, i.e., the large variance from the gradient estimator, is significant in NLP applications since many NLP datasets have diverse properties, and the tasks for meta-learning in NLP applications also have their unique characteristics. For example, the MultiWOZ dataset (Budzianowski et al., 2018) for dialog systems and the Spider dataset (Yu et al., 2018) for semantic parsing, both consist of complex and cross domain examples. To address the aforementioned gradient variance issue in NLP applications when applying first-order meta-learning approaches, we propose a variance-reduced first-order meta-learning (VFML) algorithm. The key idea of our algorithm is that we leverage a novel variance reduction term in the task adaptation steps to reduce the variance of the gradient estimator. We evaluate our proposed method on two NLP applications: few-shot text classification and domain adaptation in multi-domain dialog state tracking. We experiment on several benchmark datasets, finding that our method produces models that can achieve better performances than the baseline Reptile (Nichol et al., 2018). Let T = {T i } i I be the set of all tasks and I be the task index set. Suppose T i is drawn from T with probability p i , and we use p to denote the probability distribution over T . Our goal is to find an initial model  such that it will have a small loss on a new task T i after a few steps of updates. Therefore, we want to solve the following problem min   R d E i  p [ L i ( f Ki (  ))] , (2.1) where f Ki (  ) is the function that updates the initial model parameter  for K steps on task T i . To solve the problem in equation 2.1, MAML uses task adaptation, i.e., f Ki (  ) , and the following meta update based on sampled tasks where  is the step size, I b is the index set of the sampled tasks, and f Ki (  ) is usually K steps of gradient descent. A more efficient and effective MAML variant is the first-order method (Finn et al., 2017; Nichol et al., 2018). For instance, Finn et al. (2017) proposed to replace the Hessian matrix in meta update with an identity matrix, which leads to First-order MAML (FOMAML). Nichol et al. (2018) proposed Reptile to further simplify FO-MAML by using the the following meta update  =    (cid:80) i I b (  (cid:48) i   ) / |I b | , where  (cid:48) i = f Ki (  ) . In this work, we propose a new method based on Reptile to improve the performance of first-order meta-learning methods. Our proposed algorithm for meta-learning is illustrated in Algorithm 1. In the following discussion, we use  L i, B it to denote the mini-batch stochastic gradient for task i and B it is the sample index set. The main idea of our method is to construct Algorithm 1 Variance-reduced First-order Meta-learning (VFML) Algorithm input initialization  0 , initial variance reduction term v 0 , step size: ,  , iteration numbers: T , K , parameters: ,  1: for t = 0 , 1 , . . . , T  1 do 2: Sample Tasks I t  I with |I t | = m 3: for i  I t do 4: w i = Task Adaptation (  t , v t , , K, , i ) 5: end for 6: Update  t +1 =  t +  1 m (cid:80) i I t (cid:0) w i   t (cid:1) 7: Update v t +1 = 1 m (cid:80) i I t  L i, B it (  t +1 ) + (1   ) (cid:0) v t  1 m (cid:80) i I t  L i, B it (  t ) (cid:1) 8: end for output  T a variance reduction term v , which is motivated by the stochastic recursive momentum technique proposed in (Cutkosky and Orabona, 2019). v will be used in the task adaptation step (line 4 in Algorithm 1) to reduce the variance of the gradient estimator. More specifically, we use the gradient estimator g ik =   L i, B ik ( w ik ) + (1   ) v (line 3 in Algorithm 2) to update the task-specific model for task T i . g ik is a weighted sum of the mini-batch stochastic gradient  L i, B ik ( w ik ) and the variance reduction term v , and (1   ) is the weight for v . When  = 1 , it reduces to Reptile. We initialize the variance reduction term v 0 by averaging the gradients from a set of tasks which are randomly sampled and computed using the initialization  0 . L i (  ) (cid:13)(cid:13) 2 2   21 and E (cid:13)(cid:13)  L i (  )  L (  ) (cid:13)(cid:13) 2 2   22 , where L (  ) = E [ L i (  )] . 21 is the variance of using  L i, B ik to estimate the gradient  L i for task T i . 22 is the variance introduced by the dissimilarity between tasks. Intuitively, the variance of the gradient estimator in Reptile, i.e., E (cid:13)(cid:13)  L i, B ik ( w ik )   L ( w ik ) (cid:13)(cid:13) 2 2 , will be determined by the following quantity O (  21 +  22 ) . In addition, the variance of the gradient estimator in VFML, i.e., E (cid:13)(cid:13) g ik   L ( w ik ) (cid:13)(cid:13) 2 2 , will be determined by O (cid:0)  21 +  2  22 + (1   ) 2 (  2  (cid:48) 2 2 + (1   ) 2  2 t +1 ) (cid:1) , where  2 t +1 = E (cid:107)  t +1   t (cid:107) 22 ,  (cid:48) 22 = E (cid:13)(cid:13) (cid:80) i I t  L i (  t ) /m   L (  t ) (cid:13)(cid:13) 2 2 . If we have a large number of examples for each task, then  21 will be small, and the variance of the gradient estimator in Reptile will be determined by O (  22 ) . When we have very diverse task distributions,  22 will be large, which can lead to a significant degradation in performance. However, for VFML, the variance will be dominated by O (cid:0)  2  22 +(1   ) 2 (  2  (cid:48) 2 2 +(1   ) 2  2 t +1 ) (cid:1) . Since  (cid:48) 22 can be much smaller than  22 and  2 t +1 goes to zero as our algorithm convergences, the variance of g ik can be much smaller than  22 by choosing appropriate parameters ,  . Therefore, the role of the variance reduction term v is to alleviate the variance introduced by the task dissimilarity. We evaluate our proposed method on one simulation experiment and two NLP applications: text classification and dialog state tracking. wave regression (Finn et al., 2017; Nichol et al., 2018). Our goal is to learn a neural network that can quickly adapt to a given sine wave function after a few adaptation steps. We follow the same experimental setup in the previous work (Nichol et al., 2018), and we compare our proposed method with Reptile (Nichol et al., 2018) in terms of the mean square error between the output of the adapted neural network and the sine wave function. Parameters: For both methods, we sample 10 tasks at each outer loop iteration and use 10 examples, i.e., b = 10 , to compute the mini-batch stochastic gradients. We choose K = 3 ,  = 0 . 01 for the task adaptation step, and choose  = 1 for the meta update. For our proposed method, we choose  by searching the grid { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } and  by { 0 . 05 , 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 , 1 . 0 } . Results: Figures 1(a) and 1(b) shows the training and test accuracy versus the number of iterations for our method and Reptile. Figures 1(c) and 1(d) illustrate the adaptation results for both methods. Figures 1(a) and 1(b) show that VFML can reduce the iteration numbers and achieve better performance in terms of training and test accuracy than Reptile. Figures 1(c) and 1(d) illustrates that our proposed method can quickly converge to a given sine wave function. These results validate the superiority of VFML. We consider two text classification datasets: Amazon (He and McAuley, 2016) and FewRel (Han et al., 2018). For Amazon dataset, it consists of customer reviews from 24 product categories, and we follow the previous work (Bao et al., 2019) to sample 1000 reviews for each category. For this dataset, our goal is to classify a given review into its corresponding product category. FewRel is a relation classification dataset, and each example is a sentence annotated with a head entity, a tail entity, and their relation. For FewRel , we aim to predict the relation between the head and tail in a given sentence. We follow the experimental setup in previous work (Bao et al., 2019). We consider the N -way K -shot setting, where N is the number of classes in each task, and K is the number of examples in the class. model proposed in (Bao et al., 2019). More specifically, we use a CNN as the embedding model to generate the input representation and a one-hidden-layer neural network with 300 units and ReLU activation as the classifier. Parameters: For both Reptile and our method, we choose K by searching the grid { 1 , 3 , 5 , 10 } ,  by { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 3 , 0 . 5 } for the task adaptation step, and choose  = 1 for the meta update. For our proposed method, we choose  by searching the grid { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } and  by { 0 . 05 , 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 , 1 . 0 } . Results: Table 1 summarizes the comparisons of different methods on Amazon and FewRel datasets for text classification. The results are averaged over 10 runs. In the 5-way 5-shot setting, our proposed method can achieve 1% and 0 . 8% improvements in terms of classification accuracy on Amazon and FewRel datasets, respectively. 5-way 50-shot settings. These two settings are used to evaluate our proposed method's performance when the variance of the gradient estimator is dominated by the variance introduced by the task dissimilarity. The results show that, when we have 50 shots, our proposed method can achieve 2 . 02% and 2 . 5% gains on classification accuracy on Amazon and FewRel datasets, respectively. The results in 10 -shot and 50 -shot settings validate the effectiveness of the variance reduction term, i.e., it is used to alleviate the variance of the gradient estimator introduced by the task dissimilarity. We also compare our proposed method with the MAML and FO-MAML methods proposed in (Finn et al., 2017) on Amazon and FewRel datasets in 5-way 5-shot settings. Table 2 shows that our method outperforms these two baselines. We also test our VRML method on the task of multi-domain dialog state tracking (DST). We experiment on the MultiWOZ (Budzianowski et al., 2018), a large scale, multi-domain human-human dialog state tracking dataset. It had been introduced to help facilitate research to solve the DST problem. This corpus contains 8438 multi-turn dialogues with on average of 13.7 turns per dialogue. Multi-domain dialog state tracking in MultiWOZ is a challenging task for meta-learning, due to the differences in dialogues between each domain. For example, the dialog states, and user utterances for hotel and train are quite different. We use the most frequent five domains: ( restaurant , hotel , attraction , taxi , train ). We follow the same setup in (Huang et al., 2020) by training on three source domains: hotel , restaurant and train , and testing on 1% of the target domains: ( taxi , attraction ). We compare our method with Reptile and the train-from-scratch, i.e., we train a randomly initialized model using data from the target domain. We use joint and slot accuracy (Wu et al., 2019) to evaluate different methods. Joint accuracy measures the accuracy of dialogue states, where a dialogue state is correctly predicted only if all the values for ( domain , slot ) pairs are correctly predicted. Slot accuracy measures the accuracy of each ( domain , slot , value ) tuples for the dialog state. Baseline models: We quantify the benefits of different meta-learning algorithms by comparing the results on top of the TRADE model architecture (Wu et al., 2019). TRADE is an encoder-decoder model utilizing two BiGRUs to encode sequences of dialogue turns, and then generating corresponding (domain, slot, value) tuples. We set the hidden size of the encoder and decoder to be 400 and use Glove embedding (Pennington et al., 2014). Parameters: For both Reptile and our method, we choose K by searching the grid { 1 , 3 , 5 } ,  by { 0 . 01 , 0 . 05 , 0 . 1 } for the task adaptation step, and choose  = 1 for the meta update. For our proposed method, we choose  by searching the grid { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } and  by { 0 . 05 , 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 , 1 . 0 } . Following the previous work (Wu et al., 2019; Huang et al., 2020), we set batch size to 32, dropout rate to 0.2. For the finetune step, we search the batch size by the grid { 4 , 8 , 16 , 32 } and setp size by { 0 . 01 , 0 . 05 , 0 . 1 } . We early stop the training of both methods when the validation accuracy converges. Results: Table 3 reports the joint and slot accuracy for different methods. The results show that, when we have 1% of the target domain data for fine-tuning, our proposed method can achieve 2.44% and 1.01% improvements in slot and joint accuracy compared with Reptile for Attraction . Compared with train-from-scratch, we can obtain 15.27% and 10.59% gains in slot and joint accuracy. Similar improvements can be obtained for Taxi . Analysis: We also consider the case when we have more target domain data for finetuning. Table 3 shows that the more target domain data we have, the more gains our method can obtain. For example, when we have 10% data for Taxi , our method can achieve 6.64%/13.13% improvements in slot/joint accuracy compared with train-from-scratch. Compared with Reptile, we can obtain 3.32%/1.09% gains in slot/joint accuracy. Note that there is no change of performance for the train-from-scratch method on 1%/5%/10% Taxi data, due to the small size of the Taxi dataset. If we train on the entire Taxi data, the joint/slot accuracy would be 75.61%/89.61%. These results show that meta-learning indeed helps when the target data is small, and VRML is very effective on using the small amount of target data compared to Reptile. We propose a novel first-order meta-learning method to reduce the variance of the gradient estimator used in task adaptation for NLP tasks. We show in both few-shot text classification and DST that our method can achieve better performance than existing methods. It is interesting to further study domain adaptation methods built upon our new algorithm.
abstain result abstain objective objective objective objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective result abstain method method method abstain abstain abstain abstain abstain method method method method abstain method abstain abstain method abstain abstain abstain abstain other method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result result abstain abstain abstain objective result objective