In this study, we investigate robustness against covariate drift in spoken language understanding (SLU). Covariate drift can occur in SLU when there is a drift between training and testing regarding what users request or how they request it. To study this we propose a method that exploits natural variations in data to create a covariate drift in SLU datasets. Experiments show that a state-of-the-art BERT-based model suffers performance loss under this drift. To mitigate the performance loss, we investigate distributionally robust optimization (DRO) for finetuning BERT-based models. We discuss some recent DRO methods, propose two new variants and empirically show that DRO improves robustness under drift. A common assumption in machine learning is that training data and test data are independent and identically distributed (i.i.d.). Unfortunately, this may not hold in practice and the test distribution might have drifted from the training distribution which can lead to a significant drop of performance in real-world applications (Moreno-Torres et al., 2012). Consider spoken language understanding (SLU), i.e., the task of mapping an utterance to a machine readable semantic interpretation, which is commonly used in voice controlled devices like Alexa, Siri or Google Assistant. Distributional drifts can be caused by seasonal and non-seasonal factors. For example, festive holidays can lead to many requests outside the daily routine. New users might use an uncommon phrasing to express their intent or they might request an uncommon song to be played. Such drifts in the input distribution are referred to as covariate drift. When users' requests fail to be recognized by the device they might rephrase their intent until they succeed, essentially adapting to the SLU model's distribution. This means that, even when new training samples are drawn from new user utterances, the dominance of the old distribution already present in the SLU model is reinforced. Fine-tuned pre-trained language models (PLM), such as BERT (Devlin et al., 2019) yield strong performance on SLU benchmarks (Chen et al., 2019). Yet, it has been observed that also PLMs are vulnerable to drifts, and there is a high interest in understanding the robustness of PLMs (Oren et al., 2020a; McCoy et al., 2019; Tu et al., 2020; Cao et al., 2020). The goals of this study are to investigate the impact of covariate drift on BERT's performance, and to experimentally investigate distributionally robust optimization (DRO) for finetuning BERT. While we focus on sequence classification for SLU, i.e., intent classification (IC) and slot filling (SF), we expect the insights of this study to be applicable also to other sequence classification tasks. To study the impact of covariate drift on model robustness, we require a dataset with known properties of the drift. However, real data for this setting is not publicly available. Therefore, we devised a method to create a train/test split with a controlled drift for sequence labeling data, which we call SEQDRIFT . Roughly speaking, SEQDRIFT creates clusters of examples based on the example's tokens and sequence labels. Then those clusters are used to create a new train/test split while leaving the label distribution intact. Notably, SEQDRIFT does not artificially alter the utterances and only exploits natural lexical variations in the data in a non-adversarial manner. Our experiments on publicly available SLU datasets repartitioned with SEQDRIFT showed that a state-of-the-art BERT-based model for SLU (Chen et al., 2019) trained with standard optimization suffers up to 5% absolute performance loss. Currently, it is an open question which range of measures are helpful to improve the generalization under drift. In this study, we investigated distributionally robust optimization (DRO), which has 1970 recently gained interest in NLP for overparameterized models (Oren et al., 2019; Sagawa et al., 2020; Liu et al., 2021; Michel et al., 2021). It is an optimization concept which assumes that the training data is a mixture of distributions, e.g., different user demographics. The objective is to be optimal under each distribution. For example, the methods proposed by Oren et al. (2019) and Sagawa et al. (2020) assume knowledge about groups of training instances such as topics or ethnic origin that can be used by the optimization. Roughly speaking, they propose to compute the loss across groups instead across individual instances. However, such group knowledge might not be available and there are other methods which do not require such prior knowledge. TOPK (Levy et al., 2020; Kawaguchi and Lu, 2020), for example, simply uses the top-k largest losses in a batch and was shown to obtain robust models. We performed an extensive experimental analysis to investigate the usefulness of several DRO methods across different scenarios. Most studies only evaluate DRO methods in one setting with in-distribution validation data and one drift type per dataset. To achieve a broader insight into the usefulness of the investigated methods we evaluated them in 8 scenarios per dataset, i.e., for different types of drift, or model selection with in-distribution and out-of-distribution validation data. Additionally, we propose an intuitive variant of TOPK , namely TOPK-GROUP or TOPK-AUTOENCODER to investigate if prior group knowledge or latent group knowledge could improve TOPK . We found that TOPK , TOPK-GROUP and TOPK-AUTOENCODER can significantly improve robustness in many scenarios, where TOPK is more reliable in terms of significant improvement, while TOPK-AUTOENCODER can be better in terms of relative improvement. In this section, we provide a brief background to spoken language understanding. Subsequently, we discuss common categorizations of dataset drifts, empirical analyses of drifts and then describe distributionally robust optimization. In this study, we focus on SLU for single-turn utterances and non-nested intents. Parsing utterances into API calls is broadly either done by task oriented semantic parsing (TOP), or as intent classification (IC) and slot filling (SF). IC is the task to classify an utterance into user intents, such as PlayMusic , FindBook or GetWeather . Meanwhile, SF is a sequence tagging task to identify spans of tokens that represent the intent's slot fillers, such as ArtistName , AlbumName or TrackNumber . In state-of-the-art approaches, IC and SF are typically modeled jointly using deep neural networks (Chen et al., 2019). A common assumption in machine learning is that the training and test data are independent and identically distributed (i.i.d.) and that the distributions are the same between training and test, i.e. P train ( x, y ) = P test ( x, y ) . Unfortunately, in practice, the test data is often out of distribution (o.o.d.), i.e. P train ( x, y ) (cid:54) = P test ( x, y ) . This can be caused by sampling bias, such that subpopulations are not equally represented in the samples of the two distributions. As this phenomenon is often caused by time-varying covariates, i.e., seasonal and nonseasonal changes, this phenomenon is referred to as a drift . However, this can be a general mismatch between the sampled subpopulations, which can be of geographic or demographic type, or they can be topics or domains. Drifts can also be caused by noise or automatic training data generation, in which filtering heuristics introduce a systematic issue, or by adversaries that exploit weaknesses of a specific model or model class. Distributional drifts can be categorized into (Moreno-Torres et al., 2012): concept drift, i.e., when the meaning changes, prior probability drift and covariate drift. Covariate drift. When P train ( x ) (cid:54) = P test ( x ) , then there is a drift in the input distribution, and when the concept P ( y | x ) does not change between training and testing, this is referred to as covariate drift. The challenge is that the population and its subpopulations are unknown because only samples are observed and thus P ( x | y ) cannot be perfectly learned. For covariate drift a model has to generalize to samples from subpopulations that are almost unseen, e.g., a spam classifier should generalize to a new spam campaign ( x, y ) test = ( word_in_email=cheap, is_spam=True ) while only observing ( x, y ) train = ( word_in_email=money, is_spam=True ) . One of the conjectures for transfer learning using PLMs is 1971 that task finetuning PLMs can generalize to inputs that are semantically similar to training instances. One reason for a lack of robustness to covariate drift is when models overfit on patterns between the input and the desired labels, e.g., when they would learn that only cheap is a predictor for spam. Another reason can be spurious correlations between patterns in the input and the label. Sagawa et al. (2020); Tu et al. (2020) showed this for entailment prediction. They grouped instances by a certain input attribute (does or does not contain negation) and target labels (is or is not entailment). The attribute did not have any direct relation to the label. Then they partitioned the data in such a way that there was a correlation between the polarity of the attribute and the label in the training data and an inverse correlation in the test data: D train = [( negation = + , entailment =  ) , ( negation =  , entailment = +)] and D test = [( negation = + , entailment = +)] , ( negation =  , entailment =  )] . They showed that models learn this correlation instead of the semantics of the actual task and then fail on test instances. Currently, there is a rising interest to investigate distributional drifts in various domains (Tu et al., 2020; Sagawa et al., 2020; Koh et al., 2021; Dunn et al., 2020; Shankar et al., 2019; Oren et al., 2020b), most prominently in Computer Vision (CV), but also in NLP. To study distributional drifts, researchers need datasets with a controlled drift between training and test data. This can be broadly achieved in two ways: Synthetic methods. A dataset drift is created by corrupting the input features with synthetic noise, for example, adding pixel noise (Goodfellow et al., 2015), perturbing the input with generative models (Dunn et al., 2020) or perturbing characters and words (Cao et al., 2020). It has been observed that robustness against synthetic noise does not imply robustness against semantically meaningful perturbations (Koh et al., 2021; Dunn et al., 2020). Natural variations. Another option is to exploit natural variations, for example, using video frames for which a model's object prediction flips between adjacent frames (Shankar et al., 2019). Koh et al. (2021) collected a large benchmark for naturally occuring drifts in CV and NLP, e.g., user demographics for toxicity detection in online comments. Sgaard et al. (2021) investigated the difference of model performance comparing random splits with heuristics like splitting the data based on sentence length or by maximizing the divergence of the token feature vectors of the train and test split. In this work, we exploit natural variations in the data to create a drift in a non-adversarial manner. Our conjecture is that this setting is a good proxy to a realistic evaluation scenario. Robustness. The robustness of a machine learning model is the property that characterizes how effective the model is while being tested on a new dataset. In this paper, robustness is formally defined as follows. Let D be a dataset split into D train , D valid , D test and let E be a performance measure E :   D  R (w.l.o.g. greater is better). We assume that there is a covariate drift between D train and D test . Given two models A and B with parameters  A ,  B   estimated on D train . We call a model B more robust than model A when E (  B , D test ) > E (  A , D test ) and E (  B , D valid )  E (  B , D test ) < E (  A , D valid )  E (  A , D test ) Empirical Risk Minimization (ERM ). Commonly used optimization algorithms assume that all examples are from the same population. This assumption stems from ERM 's optimization objective which treats each example in D train  P with equal importance, i.e.,   = inf  EP [ l ( x, y ;  )] . This optimization may have a negative impact on model robustness. For example, Tu et al. (2020) found that using ERM for finetuning PLMs learns spurious correlations even in the presence of a few helpful counter examples. Distributionally Robust Optimization (DRO). DRO is based on the assumption that D train consists of samples from many subpopulations, i.e., distributions Q from an uncertainty set U ( x, y ) . The objective in DRO is then to optimize the parameters such that they are optimal under the worst case distribution in U ( x, y ) , i.e.,   = inf  sup QEQ [ l ( x, y ;  )] . DRO is effective when the proportions of the distributions Q are highly skewed in D train . For example, this can help to avoid learning spurious correlations, because even very few counter examples in the data are amplified. The challenge in applying the DRO concept is that the subpopulations are not observable and U ( x, y ) has to be modeled by some prior knowledge about the data. Our goal is to study the impact of covariate drift on model performance. Therefore, we need a benchmark with controlled drift, but currently there are no publicly available SLU benchmarks in which real drifts can be studied. As motivated in Section 2.3, we do not want to employ synthetic noise, i.e., our goal is to design a method that exploits natural variations in the data. Moreover, the method should not be adversarial, i.e., not designed or optimized to target a specific model or model class. Instead, we target two semantic drifts that might occur in real data due to: how users express their intent, and what users request. We conjecture that it is possible to capture how users express themselves by creating clusters of utterances with similar slot contexts. To capture what users request could be achieved by clusters of utterances with similar slot values. A drift can then be created by partitioning the data based on those clusters into training and testing. We avoid creating a mismatch of the label distributions between training and testing. If a mismatch would occur, it would not be possible to derive conclusions about covariate drift from changes in performance because the shift in the label distribution also leads to changes in the measured performance. In the following, we describe our approach in detail. The high-level overview for creating a drift dataset version is as follows: (i) Join all splits from the original data. (ii) Transform examples into feature representations. (iii) Use spectral clustering to obtain K clusters based on the feature representations. (iv) Create the test split based on the clusters by sampling clusters instead of sampling examples. Slot value drift To cluster examples by  what users request" we chose the feature representation of slot value n-grams. Table 1 shows an example in which only the slot values (the non-gray cells) are used to generate n-grams for an utterance, e.g. song or too poetic . The expected effect of splitting the data based on clusters of examples using this representation is that the training split is missing certain slot values, and thus we encounter unseen artists during testing. Slot context drift The feature representation to cluster training examples by  how users express an intent or slot" are n-grams of slot labels and the tokens around them. For example, using only the non-gray cells in Table 2 to generate n-grams would yield add B-SONG by B-ARTIST or to my B-PLAYLIST as features to represent the example. The expected effect of this drift is that the test data contains phrases which are not seen during training. Now, using the feature representation for either the slot value drift or slot context drift , we use spectral clustering to create K clusters and proceed to create the data splits. Test split. First the test split is created by sampling clusters and all the clusters' examples are added to the test split. To avoid a mismatch of the label distribution between training and the new test split, the method uses a projected label count per split to decide whether a cluster can be used. For example, let's assume we defined a 5% test split percentage and there are 1000 examples with the intent-slot label PLAYMUSIC-ARTIST. Then the test split should have  50 examples with the 1973 intent-slot label PLAYMUSIC-ARTIST. Hence, a cluster which contains 70 examples with the intent-slot label PLAYMUSIC-ARTIST cannot be used for the test split because it would exceed the projected label count. Thus, when a cluster is sampled all of its examples are added to the test split if they do not disturb the projected label count. This is repeated until all clusters have been sampled once and have been added to the test split or not. When the test split does not match the projected label count, it is filled using random examples from clusters that have not been used for test so far. These examples do not count into the controlled drift. O.O.D. validation One variation is that the validation data could be o.o.d. instead of distributed like the training data. This is a hypothetical setting in which we have access to o.o.d. data for validation and can observe to what degree hyperparameter tuning and model selection do factor into the drift effect. To achieve this we create the validation data in the same way as the test data, but validation and test do not share drift clusters. Full drift and partial drift In the default behavior all the examples of a cluster are shifted into the test split which we call a full drift . However, a natural question is what happens when a small percentage of a test cluster leaks into training. We call this setting a partial drift . In the experiments in Section 5 we will show that using ERM optimization on the SEQDRIFT partitioned datasets is not robust. There might be many measures to mitigate this effect, and the best solution will most likely consist of a mix of methods. One candidate is DRO that has seen a rising interest to be applied to overparameterized models. In the following, we first briefly discuss the setting of finetuning a pretrained language model (PLM), and subsequently we describe existing and proposed DRO methods. In our setup, a pretrained language model M consists of a pretrained encoder ENC , and one (or more) task classifier head(s) C task . Let X be a batch of inputs of size b , then the hidden representations of M are the output of the encoder X enc = ENC ( X ) . For example, in our study we denote the averaged hidden token representations of size d after the last layer of BERT as X enc  R b  d . To finetune M for a new task, the parameters of the encoder and the task classifier heads are optimized with a loss function L task to obtain the task batch loss l task = L task ( C task , X enc )  R b . The following methods differ mainly in the way they manipulate the task batch loss l task . The following DRO methods are by no means exhaustive. They represent either methods proposed so far in NLP or have a desirable property, e.g., being simple or conceptually interesting. The main differences between the methods is that they either use or do not use group knowledge in their objective. Those models that do require knowledge about groups in the data will use the clusters created by the SEQDRIFT algorithm. However, using the SEQDRIFT clusters is somewhat artificial because this is perfect information. Therefore, we are especially interested in methods that do not require group knowledge. TOPIC-CVAR This method was proposed by Oren et al. (2019) for language modeling. They use a topic model to obtain a distribution over topics for each sentence to model the uncertainty set. The core idea is to accumulate the losses for each topic over the course of training. In each update a subset of losses in l task is selected, i.e., the losses of those batch items that are assigned to the topic that currently lies in the upper  percentile of accumulated losses. GROUP-DRO This method was proposed by Sagawa et al. (2020) for data where groups are known such that each example is assigned to one group. Similar to TOPIC-CVAR their method keeps statistics of the accumulated losses, but for groups rather than for topics. In GROUP-DRO the batch losses in l task are first averaged per group and the final loss is a weighted average over group losses. For batch construction their method upsamples groups reciprocally to their frequency. TOPK This method does not require group knowledge and is simple to implement: it simply computes the loss as an average over the topk largest losses in l task (Levy et al., 2020; Kawaguchi and Lu, 2020). We found TOPK to be very effective in initial experiments. By contrast, GROUP-DRO and TOPICCVAR did not perform well in our setting, even though both have been shown to work well. Thus, we propose the following TOPK variants: TOPK-GROUP . If group information is available, can TOPK be improved by it? Here the idea is similar to TOPIC-CVAR and GROUP-DRO  to use the precomputed SEQDRIFT clusters as groups and compute the TOPK loss per group. Then only the largest TOPK group loss is picked, which has the effect of upsampling difficult groups and downsampling easy groups over the course of training. However, when the precomputed SEQDRIFT clusters are used, this is more an oracle, i.e., an upper bound of how much can be inferred from the training data using perfect information. TOPK-AUTOENCODER (TOPK-AE). What if we do not have access to the precomputed clusters? Could we approximate them using the PLM's hidden representations X enc ? Our idea is to use the X enc representations to cluster the b batch items into c latent groups. The latent groups are then used in the loss computation like in TOPK-GROUP . The clustering is obtained from an autoencoder which is trained on X enc and is continuously updated during training. Thus, the group assignment of a training example can change over the course of training according to the model's changing hidden representations. We investigated hard cluster assignment TOPK-AE-BIN and soft cluster assignment TOPK-AE-PROB . See Appendix B for all the details regarding the autoencoder and its training. Discussion Table 3 compares the different methods discussed in this study, and shows if the method relies on precomputed groups or if the groups are implicit or adaptively inferred during training. In this section, we present our experiments to investigate the following questions: (Q1) Does the standard optimization ERM suffer a performance loss under the SEQDRIFT covariate drift? (Q2) How well can ERM and DRO methods exploit a scarce signal about the test distribution, i.e., when is DRO relevant? (Q3) As all optimization methods come with hyperparameters, how much better could each method perform with access to o.o.d. validation data to optimize hyperparameters and perform early stopping? Would DRO still be better than ERM ? (Q4) Are the DRO methods more robust than ERM against the SEQDRIFT covariate drift, and which DRO method is the most effective? SLU Model. We use the JointBERT model for SLU (Chen et al., 2019). Two small changes that we introduce are: (i) an intent loss scaler  for the joint tasks loss L = L slot +   L intent and (2) using softmax layer instead of CRF for the sequence tagging classifier. We established the usefulness of those two changes with a hyperparameter study 1 . The source datasets for SEQDRIFT are four commonly used SLU benchmarks, which are listed in Table 6. All technical details and settings for SEQDRIFT are discussed in Appendix A. Table 5 shows an excerpt from a cluster from the ATIS dataset and demonstrates how the slot context drift cluster contains examples with similar phrases, in this case utterances with the phrase between B-from.city and B-to.city . Use of datasets. To study robustness it is inevitable to look at test performance. Thus, we did not use all datasets for all stages of experimentation: Prototyping of SEQDRIFT was only done on ATIS, and then final experiments with ERM were done on all four datasets. The prototyping and initial experiments for the DRO methods were mostly done on ATIS and a few trials on SNIPS. The final DRO experiments were conducted on SNIPS and TOP. All dataset scenarios In total we can evaluate a method in eight different scenarios per dataset, i.e., the cross product of {slot value drift, slot context drift}  {partial drift, full drift}  {i.i.d. validation, o.o.d. validation}. Table 4 shows the resulting statistics for the datasets SNIPS and TOP-NN. The percentage of examples resulting from a controlled drift in the test set are 66  79% for SNIPS and 44  47% for TOP. For the scenario with partial drift 2 3% of the training data split belong to clusters that have been deliberately shifted into the test split. Metrics. We use the following metrics: F1 the slot F1 metric; ACCURACYthe intent accuracy; COMBINED-IC-SF the average of F1 and Accuracy. Hyperparameters. To ensure a fair comparison of methods in the experiments, we performed a hyperparameter search with the objective to optimize for COMBINED-IC-SF for each optimization method for four scenarios {slot value drift, slot #int. context drift}  {i.i.d. validation, o.o.d. validation} with partial drift (see Section 3.4). For each setting we ran 8 hyper-parameter optimization steps 2 , then picked the two best hyper-parameter settings and retrained them with a different random seed. Then we reused the best hyper-parameters for the full drift. See Appendix C and Table C.9 for the remaining details about the hyperparameters. (a) Drop in absolute ERM performance from validation to test on four datasets with two drift types (slot value, slot context) and drift percentage (full, partial). (b) Amount of times a method was significantly worse or better than ERM on SNIPS for either full drift or partial drift out of 4 scenarios each. (c) Amount of times a method was significantly worse or better than ERM on SNIPS when the validation data is i.i.d. or o.o.d., i.e. out of 4 scenarios each. The reported results for each scenario are always averaged from four models, i.e., the models obtained with the two best hyperparameter settings that each have been trained with two random seeds. We computed significance with p < 0 . 05 between models with approximate randomization (Noreen, 1989). Due to the repetition with different random seeds, this effectively results in a family-wise error rate of 0 . 185 . In Figures 1b and 1c we count in how many scenarios a DRO method was significantly better or worse than ERM . The remaining instances performed the same as ERM . (Q1) Does the SEQDRIFT covariate drift lead to a drop in performance for ERM ? In Figure 1a, it can be observed that ERM 's performance does drop up to 5% in slot F1 between validation and test. However, the amount of change varies between datasets. In most scenarios, slot F1 suffers a higher drop in performance than intent accuracy. Slot context drift yields a higher loss than the slot value drift , so it seems that it is easier to generalize to unseen slot values than to unseen phrases. This makes intuitively sense, e.g., Please play New Unknown Artist. can be recognized by just knowing the sequence please play B-ARTIST I-ARTIST ... , but it is more difficult to generalize to a new unseen phrase. See Appendix D Table 10 for the numerical results. which 2  3% of the training data are leaked examples from test clusters. Thus, during training there is some information about test clusters that could be exploited. For ERM , Figure 1a shows indeed that the partial slot context drift leads to a smaller drop in performance than the full slot context drift . Thus, we conclude that ERM can exploit this information. For DRO, Figure 1b shows that there are less scenarios with significant improvement from DRO methods over ERM with a partial drift than with a full drift (see numerical results Appendix D Table 14). It is important to note that all methods E RM and DRO do improve, but ERM improves more than most of the DRO methods. Only TOPKGROUP and TOPK still improve over ERM . Anecdotally, in a SEQDRIFT setting in which only 80% or less of the clusters are drifted into the test split and the percentage of the drift cluster examples make up more than 5% of the training data, the significant improvement of all the DRO methods vanishes. (Q3) Does o.o.d. validation data help hyperparameter optimization and early stopping? In Figure 1c, we observe that the amount of significant improvement over ERM shrinks when the validation data is o.o.d. and thus contains information how to perform well for the test split. This affects hyper-parameter optimization and early stopping which also helps ERM to obtain a model from the training data that performs better on the test distribution. This can serve as an upper bound of possible improvement that can be derived from the 1977 significantly worse significantly better 0 2 4 6 8 10 12 14 16 GROUP-DRO CVaR TOPK-AE-BIN TOPK TOPK-GROUP Figure 2: Amount of times a method was significantly worse or better than ERM on SNIPS and TOP out of 16 scenarios. training data alone. See Appendix D Table 13 for the detailed numerical results. (Q4) Are the DRO methods more robust against the drifts than ERM ? Figure 2 shows in how many scenarios the DRO methods improved significantly over ERM on the SNIPS and TOP dataset. TOPK and TOPK-GROUP only improve significantly over ERM and do not perform worse. TOPK-AE-BIN only performs worse one time but otherwise the same or better. The results indicate that TOPK -based methods do improve robustness. TOPK-GROUP performs best amongst all methods, i.e., group information helps TOPK -based methods. TOPK-AE-BIN performs slightly worse in terms of significant improvement than TOPK , however, in terms of average relative improvement over ERM it is on par or better than TOPK (see Tables 11 and 12 in the Appendix). Yet, the lesser amount of significant improvement of TOPK-AE-BIN in comparison to TOPK-GROUP shows that approximating the group information is difficult. Without perfect group information a simple method like TOPK might be the most reliable method to obtain a robust model. See more detailed results in Appendix D Table 11 and Table 12. Discussion GROUP-DRO , TOPIC-CVAR and TOPK-GROUP use the SEQDRIFT clusters in their optimization. Therefore, these results should be rather seen as an upper bound of how much can be inferred from the training data using perfect information. Still, GROUP-DRO and TOPIC-CVAR both fail to perform well in this experiment. Note that both methods had the same amount of budget for hyper-parameter optimization as other methods. For GROUP-DRO we used the authors' published code 3 and also their implementation of TOPICCVAR . Our conjectures about this finding are: (1) GROUP-DRO and TOPIC-CVAR both have been proposed and studied for groups that have much higher lexical variance than the groups in our data. The groups in our dataset consist by construction of many examples with similar lexical patterns and can be of small size, i.e., as little as 10 examples. This might explain why they seem to overfit heavily. (2) Another difference to our methods is that our proposed methods do not use an exponential average of historical group loss statistics. We studied finetuning BERT for SLU datasets with covariate drift. We presented the SEQDRIFT method to induce a covariate drift for SLU sequence classification tasks. The experimental results showed that this drift in the input distribution leads to a drop in performance on four SLU datasets for a common BERT-based SLU model finetuned with ERM . We investigated DRO methods that either use or do not use knowledge about groups in the data. Our empirical results in an extensive study indicate that TOPK -based DRO methods are successful in improving robustness on the drift datasets. We would like to thank Rainer Gemulla, Patrick Lehnen and ACL reviewers for helpful feedback for revising and improving the paper.
objective abstain objective abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective objective result method abstain method other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other method abstain other other method other method other abstain other other other other other other other other other other objective method objective other abstain method other other method other abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain objective result other