Improving model generalization on held-out data is one of the core objectives in commonsense reasoning. Recent work has shown that models trained on the dataset with superficial cues tend to perform well on the easy test set with superficial cues but perform poorly on the hard test set without superficial cues. Previous approaches have resorted to manual methods of encouraging models not to overfit to superficial cues. While some of the methods have improved performance on hard instances, they also lead to degraded performance on easy instances. Here, we propose to explicitly learn a model that does well on both the easy test set with superficial cues and hard test set without superficial cues. Using a meta-learning objective, we learn such a model that improves performance on both the easy test set and the hard test set. By evaluating our models on Choice of Plausible Alternatives (COPA) and Commonsense Explanation, we show that our proposed method leads to improved performance on both the easy test set and the hard test set upon which we observe up to 16.5 percentage points improvement over the baseline. Pre-trained language models such as BERT (De-vlin et al., 2019) and RoBERTa (Liu et al., 2019) have enabled performance improvements on benchmarks of language understanding (Wang et al., 2019a). However, improved performance is not only the result of increased ability to solve the benchmark tasks as intended, but also due to models' increased ability to cheat by relying on superficial cues (Gururangan et al., 2018; Sugawara et al., 2018; Niven and Kao, 2019). That is, even though models may perform better in terms of benchmark scores, they often are right for the wrong reasons (McCoy et al., 2019) and exhibit worse performance when prevented from exploiting superficial cues (Gururangan et al., 2018; Sugawara et al., 2018; Niven and Kao, 2019). To analyze reliance on superficial cues and to evaluate methods that encourage models to be right for the right reasons, i.e., to solve tasks as intended, training instances can be divided into two categories (Gururangan et al., 2018): easy training instances contain easily identifiable superficial cues, such as a word that strongly correlates with a class label so that the presence or absence of this word alone allows better-than-random prediction (Niven and Kao, 2019). In contrast, hard instances do not contain easily exploitable superficial cues and hence require non-trivial reasoning. Models that exploit superficial cues are characterized by a performance gap: they show high scores on easy instances, but much lower scores on hard ones. Previous work has aimed at countering superficial cues. A direct, if drastic, method is to completely remove easy instances from the training data via adversarial filtering (Zellers et al., 2018), which leads to better performance on hard instances, but, as Gururangan et al. (2018) point out, filtering easy instances may harm performance by reducing the data diversity and size. Instead of completely removing easy instances, Schuster et al. (2019) propose a loss discounting scheme that assigns less weight to instances which likely contain superficial cues, while Belinkov et al. (2019) use adversarial training to penalize models for relying on superficial cues. A different approach, taken by Niven and Kao (2019) and Kavumba et al. (2019), is to augment datasets in a way that balances the distribution of superficial cues so that they become uninformative. Common to all these approaches is that reduced reliance on superficial cues is reflected in degraded performance on easy instances, while maintaining or increasing scores on hard instances. Here we propose meta-learning as an alternative approach to reducing reliance on superficial cues, which, as we will show, improves performance on both easy and hard instances. Intuitively, we see reliance on superficial cues not as a defect of datasets, but as a failure to learn : If a model learns to rely on superficial cues, it will not generalize to instances without such cues, but if the model learns not to rely on such cues, this generalization will be possible. Conversely, a model that only learns how to solve hard instances may perform poorly on easy instances. Therefore, our meta-learned model learns how to generalize to both easy and hard instances. By evaluating our method on two English commonsense benchmarks, namely Choice of Plausible Alternatives (COPA) (Roem-mele et al., 2011) and Commonsense Explanation (Cs-Ex) (Wang et al., 2019b), we show that meta-learning improves performance on both easy and hard instances and outperforms all baselines. In summary, our contributions are: 1. We propose a meta-learning method that learns how to generalize to both easy and hard instances ( 2), 2. We show that Commonsense Explanation (Wang et al., 2019b) contain superficial cues that are easy to exploit by models ( 3), 3. We empirically show that meta-learning a model to generalise to both easy and hard instances leads to better generalization not only on hard instances but also on easy instances ( 4), 2 Learning to Generalize 2.1 Background Meta-learning has been successfully applied to problems such as few-shot learning (Vinyals et al., 2016; Finn et al., 2017) and continual learning (Javed and White, 2019; Beaulieu et al., 2020). A meta-learning or learning to learn procedure consists of two phases. The first phase, also called meta-training, consists of learning in two nested loops. Learning starts in the inner loop where the models' parameters are updated using the meta-training training set. At the end of the inner loop updates, the models' inner loop learning of the task is meta-train tested in outer loop where a separate meta-training testing set is used. This is called meta-training testing. Unlike a non-meta-training process, the meta-training testing error is also used to update the model parameters, i.e., the meta-training testing error is used to improve the inner loop. Thus, learning is performed in both the inner and the outer loop, hence, learning-to-learn. The second phase, also called meta-testing, consists only of a single loop. Model parameters are finetuned on a meta-testing training set and finally evaluated, only once, on the held out meta-testing testing set. Note that the meta-testing testing set is different from the meta-training testing set. One of the most popular meta-learning algorithms is Model-Agnostic Meta-Learning (MAML) algorithm (Finn et al., 2017). MAML is a few-shot optimization-based meta-learning algorithm whose goal is to learn initial model parameters  for multiple related tasks such that a few gradient updates lead to optimal performance on target tasks. We choose MAML for our experiments because it is model agnostic and, hence, widely applicable. Our goal is to learn a model f  , with parameters  , that generalizes well on both easy instances, with superficial cues, and hard instances, without superficial cues. Specifically, given a large single-task training set D tr , we want to be able to train a model that generalizes well to both the easy test set D test _ easy and the hard test set D test _ hard . To learn such a model, we require a meta-training testing set, D tr _ test , which contains both easy and hard instances. Such a meta-training testing set will ensure that we evaluate the model generalization to both easy and hard instances. Optimizing only for better performance on hard instances can lead to poor generalization to easy instances (Gururangan et al., 2018). We cannot naively apply the meta-learning method designed for learning multiple few-shot tasks to a large dataset. A large dataset presents three main challenges. First, a naive meta-learning method would require using the entire training set during each inner loop update. This would make training very slow and computationally expensive. To address this problem, we use small randomly sampled batches in each inner loop. This is similar to treating each mini-batch as a single MAML task. Second, a naive meta-learning method would require using the entire meta-training testing set for each outer loop update. This, too, would make learning very slow when the meta-training testing set is large. We address this challenge by evaluating the inner loop learning using only a small batch that is randomly drawn from the meta-training testing set. Third, a naive meta-learning method would require storing the entire inner loop computation graph to facilitate second-order gradients' computation. However, for large datasets and large models, such as recent pre-trained language models used in this paper, this is computationally too expensive and impractical on current hardware. To address this problem, we use first-order MAML (Finn et al., 2017) that uses only the last inner-update. We call this method of using random meta-training training batches and meta-training testing batches for meta-updates as Stochastic-Update Meta-Learning (SUML, Algorithm 1). The hyperparameter k is the number of inner loop updates performed for each outer loop update (i.e., i in Algorithm 1 ranges from 1 to k). Setting the value of k to 1 would make training unstable, much like using a batch size of 1 in standard (non-meta) training. On the other hand, a large value of k would make training slow. Here, we briefly describe the English commonsense datasets that we use in this paper. anced COPA) counters superficial cues in the answer choices of the Choice of Plausible Alternatives (Roemmele et al., 2011, COPA) by balancing token distribution between correct and wrong answer choices. Balanced COPA creates mirrored instances for each of the original instance in the training set. Concretely, for each original COPA instance shown below: Premise : The stain came out of the shirt. What was the CAUSE of this? a) I bleached the shirt. (Correct) b) I patched the shirt. Balanced COPA creates another instance that shares the same alternatives but a different manually authored premise. The wrong answer choice the original question is made correct by the new premise (refer to App. B for more examples). Premise : The shirt did not have a hole anymore . What was the CAUSE of this? a) I bleached the shirt. b) I patched the shirt. (Correct) Commonsense Explanation : Commonsense Explanation (Cs-Ex) (Wang et al., 2019b) is a multiple-choice benchmark that consists of three subtasks. Here we focus on a commonsense explanation task. Given a false statement such as He drinks apple. , Cs-Ex requires a model to pick the reason why a false statement does not make sense, in this case either: a) Apple juice are very tasty and milk too ; or b) Apple can not be drunk (correct); or c) Apple cannot eat a human . While COPA has already been shown to contain superficial cues by Kavumba et al. (2019), Cs-Ex has not been analyzed yet. Here, we present an analysis of superficial cues in Cs-Ex. We fine-tuned RoBERTa-base and RoBERTa-large with contextless inputs (answers only). This reveals the models' ability to rely on shortcuts such as different token distributions in correct and wrong answers (Gururangan et al., 2018; McCoy et al., 2019). In this setting, we expect the models' accuracy to be nearly random if the answer choices have no superficial cues. But, we find that RoBERTa performs better than random accuracy of 33.3%. The above-random performance of RoBERTa-base (82.1%) and RoBERTa-large (85.4%) indicates that the answers of Cs-Ex contain superficial cues. To identify the actual superficial cues a model can exploit, we collect words/unigrams that are predictive of the correct answer choice using the productivity measure introduced by Niven and Kao (2019, see definition in App. A). Intuitively, the productivity of a token expresses how precise a model would be if it based its prediction only on the presence of this token in a candidate answer. We found that the word not was highly predictive of the correct answer, followed by the word to (See details in App. A). Following previous work (Gururangan et al., 2018; Kavumba et al., 2019), we split the test set of Cs-Ex into an easy and hard subset. The easy subset consists of all instances (1,572) that RoBERTa-base solved correctly across three different runs in the contextless input (answer only) setting. All the remaining instances, 449, are considered hard instances. For COPA, we use the easy and hard subset splits from Kavumba et al. (2019), which consists of 190 easy and 310 hard instances. In our experiments, we used a recent state-of-the-art large pre-trained language model, namely RoBERTa (Liu et al., 2019)an optimized variant of BERT (Devlin et al., 2019). Specifically, we used RoBERTa-base and RoBERTa-large with 110M and 355M parameters, respectively, from the publicly available Huggingface source code (Wolf et al., 2019). 1 We ran all our experiments on a single NVIDIA Tesla V100 GPU with 16GB memory. We used an Adam optimizer (Kingma and Ba, 2015) with a warm-up proportion of 0.06 and a weight decay of 0.01. We randomly split the training data into training data and validation data with a ratio of 9:1. We trained the models for a maximum of 10 epochs with early stopping based on the validation loss (full training details in App. C). To evaluate the effectiveness of meta-learning a model to be robust against superficial cues, we compare our model that is meta-trained on 450 original COPA instances and 100 balanced meta-training testing examples with three different baselines. Specifically, we compare to: 1. A model trained on 500 original COPA instances. 2. An adversarial trained model to avoid the answer only superficial cues on 500 original COPA instances. 3. A model trained on 1000 Balanced COPA instances, manually created to counter superficial cues. In comparison, our meta-trained model uses only a small fraction of balanced instances. Effectively, our method replaces the need to have a large balanced training set with a small, 100 instances, in this case, meta-training test set. The results show that the models trained on the original COPA perform considerably better on the easy subset (90.5%) than on the hard subset (83.9%) (Table 1). The models trained on balanced COPA improves performance on the hard subset (88.1%) but slightly degrades performance on the easy subset (90.0%). This indicates that training on Balanced COPA improves generalization on the hard instances. As expected, the performance of the adversarial trained model is lower than the vanilla baselines. This finding is similar to the result found in natural language inference (Belinkov et al., 2019). Comparing our meta-trained models to the baselines, we see that meta-training improves performance on both the easy subset and hard subset. Our meta-trained models even outperform the models trained on nearly twice the training data and an ensemble of RoBERT-large. It even matches an ensemble of RoBERTa-large and 1 https://github.com/huggingface/trans formers ALBERT-xxlarge (Lan et al., 2019). 2 4.3 Commonsense Explanation This experiment aims to investigate an automatic method of creating a meta-training testing set. Here we assume that there is no budget for manually creating a small meta-training testing set as in Balanced COPA. We created a meta-training testing set by randomly sampling 288 hard instances. Gururangan et al. (2018) pointed out that optimizing only for hard instance might lead to poor performance on easy instance. This observation motivates us to include both easy and hard instances in the meta-training testing set, with the expectation that this will ensure that performance on easy instances does not degrade. We augmented the hard instances with an equal number of randomly sampled easy instances, resulting into the final meta-training testing set with 576 instances. The results show that the meta-trained models perform better than the baselines on both easy and hard instances (Table 1). For RoBERTa-large we see 0.9 percentage point improvement on easy instances and eight percentage points improvement on the hard instances. We see the largest gains on the RoBERTa-base with 2.6 and 16.5 percentage points on easy and hard instances, respectively. The results indicate that in the absence of a manually authored meta-training testing set without superficial cues, we can use a combination of easy and hard instances. We propose to directly learn a model that performs well on both instances with superficial cues and instances without superficial cues via a meta-learning objective. We carefully evaluate our models, which are meta-learned to improve generalization, on two important commonsense benchmarks, finding that our proposed method considerably improves performance across all test sets. 2 The SuperGlue leaderboard, from which the results shown in the first two rows of Tab. 1 were taken, does not publish system outputs, so it's not possible to compute scores on easy and hard subsets. And, the ensemble models reported have not been published yet, and there is no paper or source code which describes the model and training procedure, so it is not possible to reproduce these results.
abstain abstain abstain abstain objective objective objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective result abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain method abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result other abstain abstain abstain abstain abstain abstain abstain result result result objective objective other other other