In a corpus of data, outliers are either errors : mistakes in the data that are counterproductive, or are unique : informative samples that improve model robustness. Identifying outliers can lead to better datasets by (1) removing noise in datasets and (2) guiding collection of additional data to fill gaps. However, the problem of detecting both outlier types has received relatively little attention in NLP, particularly for dialog systems. We introduce a simple and effective technique for detecting both erroneous and unique samples in a corpus of short texts using neural sentence embeddings combined with distance-based outlier detection. We also present a novel data collection pipeline built atop our detection technique to automatically and iteratively mine unique data samples while discarding erroneous samples. Experiments show that our outlier detection technique is effective at finding errors while our data collection pipeline yields highly diverse corpora that in turn produce more robust intent classification and slot-filling models. High-quality annotated data is one of the fundamental drivers of progress in Natural Language Processing (e.g. Marcus et al., 1993; Koehn, 2005). In order to be effective at producing an accurate and robust model, a dataset needs to be correct while also diverse enough to cover the full range of ways in which the phenomena it targets occur. Substantial research effort has considered dataset correctness (Eskin, 2000; Dickinson and Meurers, 2003; Rehbein and Ruppenhofer, 2017), particularly for crowdsourcing (Snow et al., 2008; Jiang et al., 2017), but addressing diversity in data has received less attention, with the exception of using data from diverse domains (Hovy et al., 2006). Outlier detection, the task of finding examples in a dataset that are atypical, provides a means of approaching the questions of correctness and diversity, but has mainly been studied at the document level (Guthrie et al., 2008; Zhuang et al., 2017), whereas texts in dialog systems are often no more than a few sentences in length. We propose a novel approach that uses sentence embeddings to detect outliers in a corpus of short texts. We rank samples based on their distance from the mean embedding of the corpus and consider samples farthest from the mean outliers. Outliers come in two varieties: (1) errors , sentences that have been mislabeled whose inclusion in the dataset would be detrimental to model performance, and (2) unique samples, sentences that differ in structure or content from most in the data and whose inclusion would be helpful for model robustness. Building upon this approach, we propose a novel crowdsourcing pipeline that distinguishes errors from unique samples and uses the unique samples to guide workers to give more diverse samples. Experimentally, we find that our outlier detection technique leads to efficient detection of both artificial and real errors in our datasets. We also use the proposed crowdsourcing pipeline to collect new datasets and build models for the dialog system tasks of intent classification and slot-filling. We find that the proposed pipeline produces more diverse data, which in turn results in models that are more robust. Outlier detection (Rousseeuw and Leroy, 1987), also called outlier analysis (Aggarwal, 2015) or anomaly detection (Chandola et al., 2009), is the task of identifying examples in a dataset that differ differ substantially from the rest of the data. For almost two decades, a body of work in NLP has investigated applying these ideas to data in order to identify annotation errors (Abney et al., 1999). Approaches have included the use of scores from trained models for POS tagging (Ab-ney et al., 1999; Eskin, 2000; van Halteren, 2000; Dligach and Palmer, 2011; Fukumoto and Suzuki, 2004), count-based methods that compare examples from across the corpus (Nakagawa and Matsumoto, 2002; Hollenstein et al., 2016), characterizing data based on feature vectors projected down into a low-dimensional space (Guthrie et al., 2008), and tracking the difficulty of learning each example during training (Amiri et al., 2018). One particularly effective approach has been to find n -grams that match but have different labels, as shown for annotations including POS tags (Dick-inson and Meurers, 2003), syntactic parses (Dick-inson and Meurers, 2005; Dickinson, 2010; Dickinson and Smith, 2011), and predicate-argument relations (Dickinson and Lee, 2008). Our work instead uses continuous representations of text derived from neural networks. While finding errors is an extremely useful application of outlier detection, also of interest are examples that are correct even though they are outliers, as these can be the most interesting and informative examples in a dataset. We term these examples unique . The problems of detecting and leveraging the unique examples in a dataset has received less attention, and the work that does exist focuses on identifying complete documents or segments of documents that are outliers out of a broader set of documents: Guthrie et al. (2007) used manually defined feature vectors to identify segments of documents with anomalous style, topic, or tone, and Kumaraswamy et al. (2015) and Zhuang et al. (2017) construct statistical models, identifying complete documents that are outliers within a set based on semantic variation. Finally, a related but distinct topic is novelty detection (Soboroff and Harman, 2005; Lee, 2015; Ghosal et al., 2018), in which two sets of documents are provided, one that is assumed to be known, and one that may contain new content. The task is to identify novel content in the second set. While outlier detection methods are often applied to this problem, the inclusion of the known document set makes the task fundamentally different from the problem we consider in this work. We build on prior work employing online crowd workers to create data by paraphrasing. In particular, we refine the idea of iteratively asking for paraphrases, where each round prompts workers with sentences from the previous round, leading to more diverse data (Negri et al., 2012; Jiang et al., 2017; Kang et al., 2018). We also apply the idea of a multi-stage process, in which a second set of workers check paraphrases to ensure they are correct (Buzek et al., 2010; Burrows et al., 2013; Coucke et al., 2018). Most notably, by incorporating our outlier detection method, we are able to automate detecting detrimental data points while also prompting workers in subsequent rounds to paraphrase more unique examples. We propose a new outlier detection approach using continuous representations of sentences. Using that approach, we explored two applications: (1) identifying errors in crowdsourced data, and (2) guiding data collection in an iterative pipeline. We detect outliers in a dataset as follows: 1. Generate a vector representation of each instance. 2. Average vectors to get a mean representation. 3. Calculate the distance of each instance from the mean. 4. Rank by distance in ascending order. 5. (Cut off the list, keeping only the top k % as outliers.) The final step is parenthesized as in practice we use a dynamic threshold approach, allowing the user to go through as much or as little of the list as they like. The intuition behind this approach is that we expect our representations to capture the semantic structure of the space for each class. An example that is far away from other examples in the set is therefore less semantically similar in some sense, making it an outlier. Importantly, it may be an outlier for two distinct reasons: (1) it is not a valid instance of this class (i.e., an error ), or (2) it is an unusual example of the class (i.e., unique ). two dialog system tasks: intent classification and slot-filling. For classification, data for each possible intent label is considered separately, meaning we find outliers in the data by considering one intent class at a time. For slot-filling, we group the data into classes based on combinations of slots. This outlier detection method is rather simple as it relies only on a sentence embedding method, a distance metric, and a threshold k ; no other hyper-parameters are involved. Moreover, the method requires no training. We shall see in Section 4 that this method performs well compared to baseline methods, no matter what sentence embedding method is used. We use Euclidean distance as our distance metric. 1 3.1.1 Sentence Representations Vector representation of sentences is an active area of research and we leverage the following approaches, each of which has been shown to have state of the art results in different use cases: Universal Sentence Encoder (USE; Cer et al., 2018) A Deep Averaging Network method, which averages word embeddings and passes the result through a feedforward network. The USE is trained using a range of supervised and unsupervised tasks. Smooth Inverse Frequency (SIF; Arora et al., 2017) A weighted average of word embeddings, with weights determined by word frequency within a corpus. We consider word embeddings from GloVe (Pennington et al., 2014) and ELMo (Peters et al., 2018). Average An unweighted average of word embeddings. While simple, this approach has been shown to be effective for classification (Zhang and Wallace, 2017) and other downstream tasks (Zhu et al., 2018). Again, we consider GloVe and ELMo word embeddings as inputs. In addition to ranked lists produced by using these core sentence embeddings, we also investigated aggregating the ranked lists using the Borda count, a rank aggregation technique that has previously been used for combining web search results (Dwork et al., 2001). The Borda count aggregates multiple ranked lists of the same set of items into a single ranked 1 We found similar results in experiments with a density-based metric, Local Outlier Factor (Breunig et al., 2000). list. First, points are assigned to each item in each list, with an item in position i in a ranked list of length N receiving N  i points. Next, points for items are summed across all of the lists. Finally, the items are ranked by their total number of points, producing a final ranking. Our proposed use of outlier detection to identify errors requires no further processing. When used in practice, a user looks through the sorted list of examples, either stopping at a given fraction, or when errors become infrequent enough. One core insight in this work is that outlier detection can be used for more than just finding errors. The outliers that are not errors are likely to be the most interesting and informative examples in our dataset. We propose to use these examples to guide data collection in an iterative process, with the goal of yielding more diverse data. To demonstrate this idea, we developed a novel crowdsourcing pipeline for data collection. Following prior work in crowdsourcing for dialog (Kang et al., 2018; Jiang et al., 2017), we ask crowd workers to write paraphrases of seed sentences with known intents and slot values. This provides linguistic diversity in our data in a way that is easily explained to workers. For instance, given the seed sentence  What is my savings account balance?  a worker might write  How much money do I have in my savings account? . Figure 1a shows a common crowdsourcing pipeline. The task designer writes seed sentences that target an intent (for classification) or a slot (for slot-filling). Crowd workers read the seed and write paraphrases. These paraphrases are then passed to another set of workers who validate if they are in fact accurate paraphrases. There are two major downsides to this standard pipeline. First, the validation step increases the cost-per-example. Second, the diversity of paraphrases depends on the given seed sentences (Jiang et al., 2017), creating a challenge for the task designer to think creatively. We introduce a new pipeline, shown in Figure 1b that uses outlier detection to (1) reduce the number of sentences being checked, and (2) collect more diverse examples. Our new approach initial designer-provided seed sentences paraphrase generation n raw samples curated samples paraphrase validation discard verified samples uses outlier detection to select only a subset of sentences to be checked: namely, the ones ranked as most likely to be outliers. This reduces effort by focusing on the sentences most likely to be incorrect. To try to increase diversity, we also introduce a process with several rounds of data collection. Outlier paraphrases collected in one round are used to seed the next round of data collection. We could directly use the sentences labeled as correct in the validation step, but while these sentences are correct, they may have diverged from the desired semantics (e.g. diverged from the desired intent class). To avoid confusion in the next round, we add a step in which workers are shown the most similar sentence from another intent (based on sentence embedding distance) and asked if the new seed is more similar to its intended intent or the alternative example. Only seeds judged as closer to their intended intent are retained. This iterative process is intended to collect more diverse data by priming workers to think about ways of phrasing the intent that are not well covered in the current data. At the same time, we avoid the correctness issues Jiang et al. (2017) observed by incorporating the validation step. First, we consider error detection, comparing various ranking methods in artificial and real data scenarios. Second, we use our uniqueness-driven pipeline to collect data, measuring the impact on data diversity and model robustness. All experiments were conducted on English language data. We measure error detection effectiveness in two settings, one artificial and the other more realistic. Artificial First, following prior work, we consider an artificial dataset in which we inject noise by mixing data from different intents (Guthrie et al., 2007, 2008; Zhuang et al., 2017; Amiri et al., 2018). This provides an easy way to control the amount and type of anomalous data, but does lead to an easier task as the incorrect examples are generally more different than naturally collected errors would be. The specific data we consider is a set of 20 intents from an in-production dialog system. To generate outliers for a given intent class X i , we randomly sample p  | X i | sentences from other intents (e.g. p = 0 . 04 , or 4%). Real We collected a new set of sentences for ten intents. Workers were given three seed sentences per intent and asked to write five paraphrases per seed. 2 Each seed was given to 15 crowd workers, leading to 2250 samples overall, after which 2 Crowd workers were paid 20 per HIT. duplicates were discarded. To identify errors, the authors independently checked each sentence and discussed any disagreements to determine a consensus label (either inlier or error ). Examples of seed sentences, along with inliers and errors is shown in Table 2. 4.1.1 Evaluation Metrics Since our core outlier detection method produces a ranked list, we are interested in evaluating how effective it is at ranking errors near the top. We use Mean Average Precision (MAP) as an overall measure of list quality. where e is the position of an error in the list. While this gives an overall qualitative measure for comparison, we are also interested in understanding the precisionrecall tradeoff when choosing a threshold k on the ranked lists. We consider defining the cutoff as a percentage k of the list and measure the percentage of errors that are covered for each possible cutoff. This measure is equivalent to Recall@ k , that is, Recall@ k = | errors above k | | errors | . We average these values across intents to get an overall value for each cutoff percentage k . For comparison, we consider four simple baselines: randomly ordering the samples (Random), sorting from shortest to longest (Short), sorting from longest to shortest (Long), and calculating distances in the vector space defined by a bag of words (BoW). Table 1 presents MAP and Recall@ k for error detection in the two settings (Artificial and Real). The neural methods outperform the baselines in both settings, demonstrating the effectiveness of our proposed approach. However, the relative performance of the neural methods differs substantially between the two settings. Specifically, (1) SIF performs better than an unweighted average on artificial data, but on real data we see the opposite trend, (2) combining rankings with Borda appears to help on the artificial data, but not on the real data, (3) ranking by length is surprisingly effective on the real data, and (4) results tend to be lower on the real data than the artificial (even at lower values of p ). This last point suggests that the commonly used artificial setting does not perfectly capture the types of errors that occur in practice. show ........average exchange rate from ten usd to cad (cid:58)(cid:58) last (cid:58)(cid:58)(cid:58)(cid:58) year Figure 3: Example annotated sentence for the slot-filling task. The slot names are (in order of appearance) .......metric, amount, currency, and (cid:58)(cid:58)(cid:58)(cid:58) date. performs particularly well vis-`a-vis other baselines on the real data, but not comparatively well on the artificial data. This can be explained by observing that the length of the average error in the real data is roughly 6 tokens, while the average inlier length is 8 tokens. Lengths of errors and inliers are roughly the same (roughly 8 tokens) in the artificial dataset, due to the outlier selection scheme. While the values in Table 1 allow an overall comparison of the methods, they do not provide a clear qualitative sense of the distribution of errors in the lists. Figure 2 shows the distribution for each method in the two settings. The effective-same 1 unique 2 random 2 same 2 unique 3 random 3 same 3 Round 2 Round 1 Round 3 Figure 4: Data collection rounds. ness of the neural methods, and USE in particular, is again clear. In the real data, when considering just the first 20% of the list, USE covers over 85% of the errors on average. One easy example was  No checkbox, more?  when the intent was to order more checks. This is clearly an error, which would at the very least need to have checkbox replaced by checkbook . In contrast, one hard example for USE was  How much money do my banks  when the intent was to request the user's balance. Until the last word, this example looks like it will be a valid balance request. These examples show that the system is qualitatively fitting our expectations for error detection. The second set of experiments evaluates our proposed uniqueness-driven data collection pipeline. We consider collecting data for two tasks used by dialog systems: intent classification and slot-filling. In each case, we calculate intrinsic measures of data diversity and the robustness of models trained on the data. Tasks We consider intent classification with 10 intents related to banking, and slot-filling for foreign exchange rate requests with four slots: amount , currency , date , and metric . Figure 3 shows an example query with annotated slots. Approaches As well as our proposed data collection pipeline ( unique ), we consider a variant where the next seed is chosen randomly ( random ), and one where the seeds are the same in every round ( same ). The third case is equivalent to the standard pipeline from Figure 1a. All three pipelines start from the same first round and then vary in the subsequent rounds, as depicted in Figure 4. Each pipeline collected data for three rounds. The final dataset for each approach combines data collected from all three rounds. In both tasks, we asked workers to rephrase each seed sentence 5 times and showed each seed sentence to 15 workers. For classification there were 3 seed sentences per intent. For slot-filling D ( a, b ) = 1  1 NN (cid:88) n =1 | n -grams a  n -grams b | | n -grams a  n -grams b | Diversity ( X ) = 1 | I | | I | (cid:88) i =1 1 | X i | 2 (cid:34) (cid:88) a  X i (cid:88) b  X i D ( a, b ) (cid:35) Coverage ( X, Y ) = 1 | I | | I | (cid:88) i =1 1 | Y i | (cid:88) b  Y i max a  X i (1  D ( a, b )) Figure 5: Metrics for diversity and coverage from Kang et al. (2018). we defined 4 example scenarios, each corresponding to a specific combination of slots. We used Borda USE+SG with k set to 10% for the outlier detection model. We consider several different metrics to probe how effectively our proposed pipeline improves data quality. In all cases, higher values are better. Intrinsic We measure the diversity and coverage of each dataset using the metrics introduced in (Kang et al., 2018) and shown in Figure 5. Extrinsic The main reason to increase dataset diversity is to construct more robust models. To directly evaluate that objective, we randomly divided the datasets collected by each pipeline into training and test sets (85-15 split). Our intuition is that a robust model should perform fairly well across all test sets. Training on a dataset that is not diverse will lead to a brittle model that only does well on data collected with the same seed sentences. For intent classification, we measure accuracy of two models: an SVM (Cortes and Vapnik, 1995) using bag of words feature representation, and FastText (Joulin et al., 2017), a neural network that averages across sentence embeddings and passes the result through feedforward layers. For slot-filling, we measure the F 1 score of a bi-directional LSTM with word vectors that are trained, but initialized with GloVe 300-dimensional embeddings. For all models, we average results across 10 runs. Classification Table 3 presents the number of examples and diversity of data collected in each round with each approach. Diversity is consistently higher with seeds chosen using our proposed unique approach. Dataset sizes vary because of the removal of duplicates. The unique approach produces a larger final set as there is less duplication across rounds. expected, the highest scores are on the diagonal training and testing on the same source data. More importantly however, training on the unique data produces a model that is robust, performing well across all three test sets. In contrast, training on the same or random data produces models that perform substantially worse on the unique test set. This trend is also present in the coverage scores in the bottom section of the table. Table 7 shows some of the seed sentences produced by the unique and random approaches. These examples illustrate the trends in our metrics, with the seeds for the random approach often being very similar. Meanwhile, the unique approach produces seeds with grammatical variation and the introduction of quite different expressions, such as ABA instead of routing number. Slot-filling Table 5 shows the number of samples collected per round for each of the data collection pipelines and the diversity of the sets. As in the classifier experiment, we observe that data produced by the unique pipeline is of higher diversity than the other two pipelines. Table 6 displays F 1 -scores and coverage for each traintest combination. Again, we see the same trends, with training on same or random leading to low results on the unique dataset, but not the reverse, and similarly for coverage, though the gaps are smaller than for classification. Outliers are often the most interesting parts of our data, but outlier detection has received relatively little attention in NLP beyond its application to finding annotation errors. This paper introduces the first neural outlier detection method for short text and demonstrates its effectiveness across multiple metrics in multiple experiments. We also propose a way to integrate outlier detection into data collection, developing and evaluating a novel crowdsourcing pipeline. This pipeline supports the creation of higher quality datasets to yield higher quality models by both reducing the number of errors and increasing the diversity of collected data. While the experiments discussed herein are concerned with components of dialog systems, we believe that similar data collection strategies could yield benefits to other areas of NLP as well. The authors thank Yiping Kang, Yunqi Zhang, Joseph Peper, and the anonymous reviewers for their helpful comments and feedback.
abstain abstain abstain method objective result abstain abstain abstain abstain objective method abstain objective result objective objective other other other other other method other abstain other other other method method abstain method abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain method method abstain abstain method method method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result objective objective abstain method other