How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity but not quality, as models that occasionally emit low quality samples would be in-sufficiently penalized. In this paper, we propose a unified framework which evaluates both diversity and quality, based on the optimal error rate of predicting whether a sentence is humanor machine-generated. We demonstrate that this error rate can be efficiently estimated by combining human and statistical evaluation, using an evaluation metric which we call HUSE. On summarization and chitchat dialogue, we show that (i) HUSE detects diversity defects which fool pure human evaluation and that (ii) techniques such as annealing for improving quality actually decrease HUSE due to decreased diversity. Generating text is a core part of many NLP tasks such as image captioning (Lin et al., 2014), open-domain dialogue (Sordoni et al., 2015), story generation (Roemmele, 2016), and summarization (Nallapati et al., 2016). However, proper evaluation of natural language generation has proven difficult (Liu et al., 2016; Novikova et al., 2017; Chaganty et al., 2018). A good evaluation metric should not only capture the quality of generation, but also the diversity of generation, which is especially crucial for creative, open-ended tasks like dialogue or story generation. Human evaluation , which is often viewed as the gold standard evaluation, captures quality but fails to capture diversity. As an example, for language Reference Model Probability (p model ) Agassi bows out of Australian open Agassi withdraws from Australian open Sharon has stroke for stroke Cleared coach facing another grilling from British swim bosses Model Generations Reference H u m a n J u d g m e n t Figure 1: HUSE is twice the classification error of distinguishing reference and generated text based on human judgment scores and model probabilities. modeling, a model that directly plagiarizes sentences from the training set would pass the human quality bar but would have zero generalization ability and thus have inadequate diversity. On the other hand, statistical evaluation i.e., perplexity on a reference test setcaptures diversity, as it ensures a model must assign reasonable probability to novel sentences, but perplexity provides an inadequate measure of quality (Theis et al., 2015). For example, modifying a perfect model by removing its ability to generate even a single test sentence results in infinite perplexity even though the model is still near-perfect. Automatic metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin and Rey, 2004) capture quality better than perplexity but still correlate poorly with human evaluation and fail to capture diversity (Novikova et al., 2017; Chaganty et al., 2018). Existing approaches to combining statistical and human evaluation have been ad-hoc, leading to misleading performance measures. A common approach is to measure diversity through the perplexity of a probabilistic model and quality through human evaluation on beam-searched outputs. This gives the illusion that a single model is high-quality and diverse, while the reality is that it shows we can have either a diverse model (when sampling from the distribution used to compute perplexity) or a high-quality model (when beam-searching). In this paper, we define the idealized evaluation metric as twice the error of the optimal discriminator for classifying sentences as coming from the reference distribution or the model (Section 2). If a model generates gibberish (low quality), the optimal discriminator can classify these accurately as coming from the model. If the reference distribution contains sentences the model cannot generate (low diversity), the optimal discriminator can classify these accurately as coming from the reference. Unfortunately, the optimal discriminator is unavailable. Human discriminators cannot capture diversity effectively, and learned discriminators e.g., from a Generative Adversarial Network (Goodfellow et al., 2014) or one trained on human judgments (Lowe et al., 2017)are too unreliable to use for rigorous evaluation. Our key result (Section 3) is based on the observation that the optimal classifier depends only on two numbers: the probability of a sentence under the model and the probability under the reference distribution. The former can be computed directly from the model, and we show that the latter can be well-approximated by human judgment scores. The resulting two-dimensional space is illustrated in Figure 1. We apply a simple k -nearest neighbor classifier in this space and define Human Unified with Statistical Evaluation (HUSE) as twice the leave-one-out error of this classifier. We apply HUSE to four natural language generation tasks (Section 5): language modeling, chitchat dialogue, story generation, and summarization. First, we show that human evaluation alone is insufficient to discriminate model generations from the references, leading to inflated estimates of model performance. In contrast, HUSE is able to reveal deficiencies of current models. We also show that common techniques for improving sample quality such as annealing actually increase distinguishability between the model and reference due to losses in diversity. Consider a natural language generation task where the model is given a context x (e.g., a dialogue history) history) drawn from some prior p ( x ) and must output a distribution over possible sentences p model ( y | x ) . We define an idealized evaluation metric based on whether p model is close to a reference distribution p ref , which is generally human-generated. 1 Specifically, consider a random variable y drawn from either the reference or the model based on an indicator z  Bernoulli (cid:0) 12 (cid:1) : y | x, z  (cid:40) p ref ( y | x ) if z = 1 p model ( y | x ) if z = 0 . Obstacles. Unfortunately, L  is unattainable because it requires computing the optimal discriminator. In the spirit of the Turing Test, we could consider using the error rate of a human discriminator f hum instead, often considered the gold standard for evaluation. However, while humans might have knowledge of p ref , they do not have full knowledge of p model and thus would have difficul-ties determining which sentences a model cannot generate. As a concrete example, suppose p ref placed a uniform distribution over some set S . Without knowledge of p model the most sensible discriminator is to predict z = 1 (reference) when y  S . This discriminator achieves the same classification error of 0 . 5 for both the perfect model p model = p ref and one which can only return a single y  S . We could try to reveal p model to humans by showing multiple samples simultaneously, but this is expensive and, as we will later see, unnecessary. Another option is to learn f over an expressive class of functions such as neural networks on data 1 While some tasks only care about quality and thus only require p model to place mass on some high quality y , we demand that p model places mass on all high quality y as given by p ref . This diversity is important for open-ended tasks such as dialogue or story generation. Also note that p ref need not be the human distribution, or match the training distribution. It can be defined as the distribution given by experts. 2 Note that L  is a linear function of the total variational divergence: (cid:107) p model  p ref (cid:107) TV def = (cid:80) x,y p ( x ) | p model ( y | x )  p ref ( y | x ) | = 1  L  . See Appendix A.1 for details. sampled from p model and p ref . This is analogous to learning the discriminator in a Generative Adversarial Network (GAN) (Goodfellow et al., 2014) or learning an evaluation metric from human judgments (Lowe et al., 2017). However, as ( x, y ) are high-dimensional objects, training a good classifier is extremely difficult (and perhaps not significantly easier than solving the original generation problem). Indeed, learned evaluation metrics do not generalize very well (Lowe et al., 2017; Chaganty et al., 2018). Unlike these approaches which seek to replace human evaluation, our focus will instead be on combining human and automatic statistical evaluation to estimate the optimal classifier error. Our key result is that the optimal discriminator depends on ( x, y ) only through a two-dimensional sufficient statistic (Section 3.1), motivating an approximation which we call HUSE (Section 3.2). For any feature map  that maps ( x, y ) to  ( x, y )  R d , define the evaluation score L (  ) to be twice the error rate of the optimal discriminator that depends on ( x, y ) only through  : L (  ) def = 2 inf f P [ f (  ( x, y )) (cid:54) = z ] . Note that the evaluation score L (  ) given by a feature map  optimizes over all functions that depend on  (3). Thus, the more information  contains, the lower L (  ) is. This has two implications: First, any feature map  yields an (opti-mistic) upper bound on L  (2), meaning that L (  ) might be able detect when a model is poor but cannot certify that it is good. Second, adding features to  can only improve this bound. Proposition 1. The two-dimensional feature map  opt achieves the optimal discriminator score: L (  opt ) = L  . Proof We compute the true posterior over z given x, y . Since p ( z = 1) = p ( z = 0) = 12 , p ( y | x, z = 1) = p ref ( y | x ) and p ( y | x, z = 0) = p model ( y | x ) , by Bayes' rule: p ( z = 1 | x, y ) = p ref ( y | x ) p ref ( y | x ) + p model ( y | x ) . The optimal discriminator simply predicts z = 1 if p ref ( y | x ) > p model ( y | x ) and z = 0 otherwise. In other words, the decision boundary is given by  opt ( x, y ) 1 >  opt ( x, y ) 2 . More generally, we can obtain this equality with a wider class of  . It will hold exactly for any invertible transformation of  opt (Appendix Corollary 1), and approximately for any  which has high mutual information with  opt (Appendix Theorem 1). This means that we can substitute p ref with noisy, possibly un-normalized estimates and still obtain accurate estimates of L  . While we can directly compute p model ( y | x ) for many probabilistic models, p ref ( y | x ) is unattainable, so L (  opt ) is not computable. However, the wisdom of the crowds (Surowiecki, 2004; Ungar et al., 2012) suggests that pooling together the judgments of many humans can often produce surprisingly reliable estimates of real-world probabilities such as p ref ( y | x ) , even if no individual human is particularly reliable. With this motivation, we ask Amazon Mechanical Turk workers to rate a sentence from 15 based on how typical it is as a way to estimate p ref ( y | x ) . (see Appendix A.3 for more details). We define HJ ( x, y ) to be the average response over 20 crowdworkers. Figure 2 shows that for a language modeling task on the Reddit corpus, 3 HJ ( x, y ) strongly correlates with the actual log-frequency of y in the corpus. The high correlation suggests that human judgments HJ ( x, y ) are a good surrogate for log p ref . In addition, we found that rather than using the model probability p model ( y | x ) directly as a feature, normalizing by sentence length len ( y ) yielded lower (tighter) scores. We therefore define the HUSE features as follows:  huse ( x, y ) def = (cid:20) log p model ( y | x ) len ( y ) , HJ ( x, y ) (cid:21) , (5) 3 We used the Reddit corpus due to crowdworker familiarity, corpus size, and short average sentence length, which results in a wide range of sentence frequencies. We now show that the HUSE score satisfies two nice properties: (i) HUSE does at least as well as human evaluation and (ii) a low HUSE score is sufficient to show that a model is far from the reference distribution. To show (i), consider a feature map that only includes human evaluation:  hj ( x, y ) def = [ HJ ( x, y )] . Because  huse also incorporates human evaluation, L (  huse ) is always tighter (lower) than the human discriminator error L (  hj ) : Proposition 1 (Relationship between HUSE, human evaluation, and optimal scores) . Furthermore, the main difference between L (  huse ) and L  is that the former uses HJ ( x, y ) and the latter uses p ref . But as we argued using Figure 2, HJ ( x, y ) is strongly correlated with p ref , and good approximations to p ref provide approximation guarantees for L (  huse ) (Appendix Theorem 1). In this section, we show how we can estimate the error rate L (  ) from finite data (Section 4.1). We then show how the HUSE estimate ( L (  huse )) can be decomposed into a score that measures quality (HUSE-Q) and a score that measures diversity (HUSE-D), which allows us to study quality-diversity tradeoffs (Section 4.2). For any feature map  , we show how to produce an estimate of L (  ) . Fix n contexts x 1 , . . . , x n . First, we draw n examples y 1 , . . . , y n from the reference distribution p ref ( y | x ) , which are usually human-generated sentences from a test set. We also draw n examples y (cid:48) 1 , . . . , y (cid:48) n from the model p model ( y | x ) we wish to evaluate. Next, for each of the 2 n examples ( x, y ) , we compute the feature map  ( x, y ) , which might involve evaluating the model probability p model ( y | x ) as well as collecting human judgments HJ ( x, y ) from crowdworkers. Finally, we compute the leave-one-out error of a classifier that tries to predict whether a given example ( x, y ) comes from the reference distribution ( z = 1 ) or the model ( z = 0 ). The classification problems for HUSE are two-dimensional, which allows us to accurately estimate error rates using a k -nearest neighbors classifier. We opt to use nearest neighbors classifiers as they are simple, require no training, and can asymptotically capture arbitrary continuous decision boundaries. Specifically, we set k = 16 and define neighbors using L 2 distances over the feature vectors  ( x, y ) scaled componentwise to have unit variance. The overall procedure for computing the estimate  L (  ) is formally defined in Algorithm 1. Algorithm 1 Estimating error rates under  Require: Feature map  , number of neighbors k Contexts x 1 , . . . , x n Reference outputs y 1 , . . . , y n Model outputs y (cid:48) 1 , . . . , y (cid:48) n 1: Construct dataset: D = n (cid:91) i =1 { (  ( x i , y i ) , 1) , (  ( x i , y (cid:48) i ) , 0) } 2:  L (  ) def = leave-one-out error of k -NN on D 4.2 Quality-diversity decomposition We now define the (empirical) HUSE score using the feature map  huse : HUSE def =  L (  huse ) . Since humans can detect quality defects in a model, any increase in error from removing p model must come from a model's lack of diversity. Therefore, we define the diversity component (HUSE-D) as follows: HUSE-D def = 1 + HUSE  HUSE-Q , (9) which implies the decomposition (1  HUSE-D )+ (1  HUSE-Q ) = 1  HUSE. As long as the discriminators are non-degenerate (obtaining better performance than chance and HUSE > HUSE-Q), all scores are contained in [0 , 1] . Here, HUSE-D = 1 implies that the model suffers no diversity defects, while HUSE-D = 0 indicates that the examples could be discriminated perfectly due to a lack of diversity. We use HUSE to evaluate three different types of single-sentence natural language generation tasks: (i) unconditional and high entropy (language mod-eling); (ii) conditional and high entropy (story generation, chit-chat dialogue); and (iii) conditional and low entropy (summarization). We show that HUSE provides a direct and interpretable measure of diversity on high-entropy tasks, while also serving as a useful model diagnostic on low-entropy ones. The four tasks along with the datasets and models are as follows:  Summarization : Giganews story to headline dataset and the pre-trained model from Gehrmann et al. (2018). The dataset consists of 3.8 million news story-headline pairs. Examples from this dataset are shown in Table 2.  Story generation : Last sentence generation for ROC stories (Mostafazadeh et al., 2016) consisting of 96,198 examples of partially written four-sentence stories as input, and a single sentence which completes the story as the target. We use a standard OpenNMT model with global attention (Klein et al., 2017). Language modeling : One billion word benchmark pre-trained language model from Jozefowicz et al. (2016). The task consists of generating a single sentence from the one billion word newswire text distribution. Chit-chat dialogue : Two-turn chit-chat dialogue dataset consisting of 37.3 million comment-response pairs from Reddit (Ap-pendix A.4). Comments are generally short (515 tokens) and cover a single topic (e.g. given wow how did i not notice that, the response is you were focusing on other things its understandable). We train a convolutional model using fairseq (Gehring et al., 2017). For all the tasks, we train neural models and evaluate their diversity-quality tradeoffs as we change the decoding scheme for generation. Our primary evaluation concerns diversity trade-offs involving temperature annealing which is a generation technique applicable to any probabilistic model that generates words sequentially. In temperature annealed models, we sample a word w proportional to p 1 /t ( w ) where p is the model probability of w given previous words and t is the temperature parameter. We excluded beam search since it qualitatively behaves similarly to temperature annealing with low temperatures and HUSE  0 due to beam search being extremely Score Summarization Story generation Chit-chat dialogue LM t = 1 . As a non-neural baseline, we also consider retrieval based models based on Apache solr on a few tasks. For this approach, we retrieve the single most relevant response from the training set using the BM25 similarity metric on inputs. Such models are known to perform well in tasks with complex outputs such as program generation (Hayati et al., 2018; Hashimoto et al., 2018) and style transfer (Li et al., 2018). For cost reasons, we did not measure certain combinations of task and generation mechanisms. We did not measure retrieval for chit-chat dialogue, as we observed its outputs were lower quality than a low-temperature neural model. We also did not anneal language models, as the generation quality from the language model was already high, and our goal was to show that they achieved high HUSE. Our set of measurements, while not comprehensive, generally covers the available quality-diversity tradeoffs for conditional tasks. Finally, we collect human judgments HJ ( x, y ) as per Section 4.1 where we query 20 Amazon Mechanical Turk crowdworkers for typicality ratings on 100 reference and 100 model sentences. Since our models generate UNK (unknown and out-of-vocabulary) tokens, we instructed crowdworkers to treat UNK tokens as rare, but appropriate words for the context. The HUSE scores across the four tasks vary widely. Table 1 shows that single-sentence language models are nearly indistinguishable, with HUSE = 0 . 86 and implied discriminator error of 43% . In contrast, both summarization and dialogue are highly distinguishable (HUSE  0 . 5 ) with relatively low quality when sampled from t = 1 . 0 . Human evaluation alone (HUSE-Q) would suggest that using temperature annealing ( t = 0 . 7) to emphasize high-probability outputs substantially improves the model (HUSE-Q goes from 0 . 58 to 0 . 92 for summarization and 0 . 56 to 0 . 92 for dia-logue). However, we find that this increase in sample quality comes at the cost of diversity (HUSE-D goes from 0 . 95 to 0 . 34 for summarization and 1 . 0 to 0 . 57 for dialogue). Examining the achievable HUSE and diversity tradeoffs in Figure 3 shows that mechanisms such as annealing which improve sample quality actually degrade HUSE due to severe losses in diversity. We find that all generation schemes and models are inadequate for story generation on ROC stories. The original model ( t = 1 . 0 ) is very easily distinguishable by a human (HUSE-Q = 0 . 15 ), corresponding to a discriminator error of 7% . The retrieval models can improve this to HUSE-Q = 0 . 47 , but this comes at the expense of diversity. Finally, we observe that directly sampling from the model ( t = 1 . 0) is always diverse. This suggests that human evaluation is an appropriate evaluation for generation systems that are directly sampled (rather than beam-searched). Since HUSE is estimated from a two-dimensional classification problem, we can directly visualize the classification problem to understand defects in both model quality and diversity. Figure 4 shows both reference points  huse ( x i , y i ) (blue squares) and model points  huse ( x i , y (cid:48) i ) (red circles) for the summarization task. The shaded areas indicate the decision boundary of the 16 -nearest neighbor classifier. At temperature t = 1 . 0 , we find that the classification boundary is mostly horizontal, implying that human judgment alone can distinguish model outputs from references. There is a cluster of sentences with high HJ and high p model which are es-T=1.0 T=0.9 T=0.7 Figure 4: The two-dimensional classification problem in Algorithm 1 on the summarization task with different softmax temperatures (three panels). sentially indistinguishable. Examining the samples in this top-right region reveals that these are news stories with short headlines such as Nadal pulls out of Sydney International which can be reliably generated even at t = 1 . 0 . However, the model frequently generates low quality samples that can easily be distinguished such as two new vaccines in the poor countries were effective against go-it-alone study says (Table 2). At lower temperatures of t = 0 . 9 and t = 0 . 7 , the boundary shifts towards becoming diagonal. Although the distribution is no longer directly separable on human judgment, the two distributions are clearly separable with the inclusion of p model . Using Figure 4, we can identify individual examples which were correctly and incorrectly classified based on p model and HJ. Table 2 shows examples of both quality failures and diversity failures identified by HUSE. For example, the di-versity failure table shows that the summarization model ( t = 0 . 7 ) has an extremely low probability of generating some reference sentences (NFL's bills shake up front office) and is thus under-diverse. Closer examination of the model shows that the probability of generating front office is low, since it is an unusual way to refer to the president and general manager. Improving these models on the diversity failures will require that the model understand more subtle paraphrases. We can also identify model successes, where the model outputs are indistinguishable from the reference in terms of quality (Agassi bows out of Australian Open after injury), and the model assigns high probability to the reference (Agassi withdraws from Australian Open). Since HUSE depends on human crowdworker annotations, one might ask if it is possible to reduce either the number of annotated examples, or number of distinct crowdworkers for each example. We show that for low-quality models, substantially fewer annotations are needed. Figure 5 shows the result of subsampling our original data of 200 sentences and 20 crowdworkers and estimating HUSE. First, we find that using 50 test set examples (Figure 5, left) is often sufficient to give accurate estimates of HUSE. Next, we find that the necessary number of crowdworkers per example depends heavily on the task. Easily distinguishable tasks (story generation), require only 10 crowdworkers, while less distinguishable tasks (summarization) require more than 20 crowdworkers to obtain accurate estimates. The current state of NLG evaluation. Existing approaches to NLG evaluation use a hodgepodge mix of quality and diversity measures. Out of the 26 NLG papers at ACL 2018, six perform only hu-Quality failure log p model HJ Context: Two new vaccines have been shown effective against rotavirus, which is responsible for a half-million infant deaths in poor countries each year, research studies published Wednesday said. man evaluation, fourteen measure human evaluation and a diversity metric such as perplexity or n-gram diversity, and six do not evaluate using human judgments. While perplexity and n -gram counts can in principle evaluate diversity, their practical implementations suffer from serious drawbacks. When human evaluation and perplexity are both evaluated, they are almost always done on separate modelshuman evaluations are done on beam-searched output, while perplexity is computed on the softmax outputs. This makes it appear as if the models can simultaneously generate high quality outputs while also being diverse, when in fact they can only be one at a time based on whether they sample or run beam search. On the other hand, n -gram diversity was proposed by Li et al. (2016) to identify models with the generic utterance problem where models repeat phrases such as I don't know'. Unfortunately, n -gram diversity is computed across contexts by counting the number of unique n -grams generated, and so does not measure a model's ability to generate multiple valid utterances at any single context. In particular, a model which only outputs a single memorized utterance per context (e.g., via memorization or retrieval) can still have high n -gram diversity as long as the memorized sentences differ across contexts. Finally, all existing diversity measures are computed separately from human evaluation. This results in two incomparable evaluation metrics, which prevent us from reasoning about tradeoffs between diversity and quality. In contrast, HUSE allows us to make precise statements about the tradeoffs between model quality and diversity because it is a single metric which decomposes into diversity and quality terms. Related evaluations of diversity. The importance of diverse responses has previously been acknowledged for summarization (Nenkova et al., 2007) and information retrieval (Clarke et al., 2008). Our work differs in considering a single evaluation measure that captures quality and diversity applicable to any generation task. Automated metrics based on n -gram overlap such as BLEU, METEOR, ROUGE (Papineni et al., 2002; Lavie and Denkowski, 2009; Lin and Rey, 2004) work well for machine translation but do not generalize well to domains with a diverse spectrum of correct responses. While variants (Sun and Zhou, 2012; Galley et al., 2015; Shima and Mitamura, 2011) have adapted such metrics to high entropy generative environments, they are still significantly inferior to the human judgments they attempt to mimic. Caccia et al. (2018) recently examined the diversity and quality tradeoffs for different language model architectures on synthetic datasets. However, as their approach relies on measuring log-likelihoods under both the model and reference distributions, it cannot be applied to real data where p ref is unavailable. Our main conceptual contribution overcomes this by showing that HJ is an acceptable proxy for p ref . Sajjadi et al. (2018) also examines diversity and quality (which they call precision and recall) in the context of generative image models. However, they rely on assuming that p ref and p model can be estimated accurately using the Frechet Inception Distance (FID) (Heusel et al., 2017). HUSE avoids such assumptions and instead directly leverages human judgments, resulting in a simple and reliable metric more suitable for use as a gold-standard. Estimating optimal classification error. Evaluating a model by estimating its optimal classification error has been considered by several earlier works (Olsson et al., 2018; Kannan and Vinyals, 2016; Li et al., 2017; Bruni and Fernandez, 2017; Bowman et al., 2016). However, these methods have focused on classifying sentences directly, which is quite challenging to do reliably. Existing adversarial evaluation methods do not yet reliably outperform human classification (Kannan and Vinyals, 2016; Bruni and Fernandez, 2017). We propose the use of both human evaluation and model probabilities as part of the adversarial evaluation framework, and demonstrate that the resulting classifier reliably outperforms humans and captures both the sample quality and diversity of a model. Distributional divergence estimation. Our proposed evaluation metric is closely related to the total variation distance which has been studied extensively in the distribution testing literature. It is known that total variation distance estimates have pessimistic minimax estimation rates in high dimensions (Balakrishnan and Wasserman, 2017). Our work overcomes this by utilizing p model and an estimate of p ref . Other approaches to distributional testing include the maximum mean discrepancy (MMD) and Wasserstein distances, but these approaches require knowledge of a ground truth metric or kernel space (Tolstikhin et al., 2016; Singh et al., 2018). Although such divergences are easier to estimate than the total variation distance from samples, the implied convergence rates are still too slow to be practically useful. In this paper, we demonstrate that the current gold standard of human evaluation does not penalize under-diverse models. To remedy this, we propose HUSE, a general purpose evaluation strategy which can be applied to any model for which we can calculate a model's sampling probabilities. HUSE is an upper bound on the optimal classification error of distinguishing reference and model-generated text, and never does worse than human classification. HUSE leverages both model probabilities and human judgments, ensuring that models which do well on the metric are both high-quality and diverse. Our work can be viewed as a superhuman ver-sion of the classic Turing Test (Turing, 1950). Instead of relying on just a human classifier, we approximate the optimal classifier, which can utilize information about the model in addition to the reference. We also modify the classification problem and seek to identify whether a sample comes from a (potentially superhuman) reference distribution, rather than the human distribution. These two changes lead to tractable, rigorous estimators which can quantify tradeoffs between model quality and diversity on a wide range of generation tasks. Acknowledgements. We would like to thank Arun Chaganty, Robin Jia, and Peng Qi for extensive comments and feedback on the paper. This work was funded by DARPA CwC program under ARO prime contract no. W911NF-15-1-0462. Reproducibility. All code, data, and experiments are available on the CodaLab platform at https://worksheets. codalab.org/worksheets/ 0x88644b5ee189402eb19d39d721d1005c .
method abstain abstain objective objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method abstain abstain abstain abstain result result result method objective abstain result abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method method method method abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method method method method method method method abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain other other other other other other other other other other other abstain objective other other objective other other other other objective other other other other other other other objective other objective other method other other abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain