Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentenceor document-level detection may fail to provide the fine-grained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HADES ( HA llucination DE tection data S et) 1 . To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models. Automatic text generation using neural natural language generation (NLG) systems is increasingly fluent and thus seemingly plausible in many real-world applications. Large-scale pretrained models like GPT-3 (Brown et al., 2020) are proven to be powerful in understanding and performing free form text generation tasks at human-quality level with a few in-context examples, which dramatically reduces the manual labor needed in many text-based applications and services. Despite their  Work was done when Tianyu (intern) and Yizhe was at Microsoft. 1 Code and data are provided in https://github. com/microsoft/HaDes great success, however, neural NLG systems using very large pre-trained models struggle to generate factually accurate and trustworthy text (Devlin et al., 2019; Radford et al., 2019), and exhibit a propensity to hallucinate non-existent or incorrect content that is unacceptable in most user-oriented applications. This poses a major challenge for deploying production NLG systems with realtime generation, where post-examination is impossible. Existing work has sought to detect hallucination and quantitatively measure generation consistency against a provided reference. Such reference-based hallucination detection has been proposed for abstractive summarization (Maynez et al., 2020), machine translation (Wang and Sennrich, 2020), data-to-text generation (Rebuffel et al., 2021), and image caption generation (Rohrbach et al., 2018). For many free-form text generation tasks, however, references are not readily available. For example, in a production NLG system such as a social chatbot using real-time response generation or a document auto-completion system, the generation model often cannot pair its outputs with sufficient reference information, rendering reference-based methods less applicable: i ) It may be difficult to even know where to obtain the reference, as obtaining it may be as hard as generating consistent information in the first place; ii ) Generation may be at a real-time online setting that demands leveraging only existing context to create new content. One common setup for qualitatively measuring the level of hallucination is performed at sentence-or document-level (Dhingra et al., 2019; Scialom et al., 2019). Related tasks such as fake news detection (Zellers et al., 2019) or fact checking (Thorne and Vlachos, 2018) also adopt this strategy. However, sentenceor document-level detection may not always provide high-resolution signals sufficient to pinpoint the hallucinated text, or can only judge whether a generated sentence or a document 6723 Input: . as a whole is a hallucinated artifact. Consequently, these high-level strategies may be insufficient to avoid hallucinations. As an alternative, at decoding time of an NLG system, we suggest that if the locus of hallucination can be identified at the token level, it may be possible to guide beam search or suppress the probability of certain tokens at real-time. To this end, we propose a reference-free, token-level hallucination detection task and introduce an annotated training and benchmark testing dataset that we call HADES ( HA llucination DE tection data S et). The reference-free property of this task yields greater flexibility in a broad range of generation applications. We expect the token-level property of this task to foster the development of models that can detect fine-grained signals of potential hallucination. In conjunction with consulting context to identify self-contradictory statements and access to commonsense and world knowledge, such fine-grained signals, when detected, should further mitigate real-time hallucination. Our contributions include: 1) We propose a reference-free , token-level hallucination detection task for free-form text generation. 2) We support this task with a dataset that we call HADES , with  11k instances extracted from English Wikipedia using an iterative data collection strategy to address data imbalance issues. We also present comprehensive analyses on the statistical features to shed light on what is commonly recognized as hallucination in crowd-sourced judgments and its salient characteristics in free-form text generation. 3) We create multiple baselines, including feature based models and pretrained models as a first step towards addressing the proposed task. tion 2 (abbreviated as  H ) or a not hallucination (abbreviated as  N ) label to the highlighted spans. To simulate real-world NLG applications, we propose two sub-tasks with  offline  and  online  settings. In the offline setting, it is assumed that generation is complete, so the the model is able perceive the bidirectional context. This could be used in the post-generation examination of NLG systems. For online detection, the model can only access the unidirectional preceding context, which simulates on-the-fly generation. Online detection is important in practice as it enables NLG systems to proactively forestall potential hallucinations. To collect the HADES dataset, we first perturb raw text web data into perturbed text (Fig 2A) (Sec 3.2). We then ask human annotators to assess whether the perturbed text spans are hallucinations given the original text (Fig 2B) (Sec 3.3). Our raw data are sampled from English WIKI -40B (Guo et al., 2020) dataset. WIKI -40B-E N is a cleaned collection of English Wikipedia articles. We randomly sample from the first paragraphs of these articles and filter out short text of fewer than 5 sentences. We use Wikipedia as our text source since it is stylistically formal and of high quality, and covers diverse topics and domains. 2 Hallucination in our paper refers to certain types of mistakes (Fig 3) made by the NLG models. The notions of consistency and not hallucination are only for annotation purposes (Sec 3.3). To acquire machine generated text in the free-form, we perturb the raw text 3 using BERT. In applying this contextual perturbation we maintained two principles: i ) the fluency and syntactic correctness of the perturbed text should be preserved; ii ) the perturbed text should be lexically diverse. We leave the first two sentences in the raw text unchanged to serve as the preceding context, so as to avoid the early token curse (Press et al., 2020) where tokens are evaluated at the beginning with limited context. The text perturbation process is split into three pipelined operations, namely MASK , REPLACE and RANK . i) In the MASK operation, we mask the to-kenized words to be replaced with the special token [MASK] in the BERT vocabulary. Starting from the third sentence, we randomly mask word spans by a pre-defined mask ratio  . By default we only mask one word in each perturbation, except for named entities identified by Spacy . We view the entity boundaries as minimal masking units to avoid collocation errors ( e.g. San Diego should be masked as a whole). To reduce trivial instances, we do not mask stop words or punctuation identified by NLTK (Bird, 2006). ii) In the REPLACE operation, we leverage a pretrained BERT-base model to predict the masked span. The mask-then-predict training framework of the BERT model contex-tualizes the replacement with both preceding and subsequent text. For better fluency, we replace the masked tokens from left to right, e.g. a 3-token REPLACE operation will be [MASK] [MASK] [MASK]  [A] [MASK] [MASK]  [A] [B] [MASK]  [A] [B] [C] 4 . When performing the replacement, we remove the original token from the predicted distribution over the vocabulary at 3 In a pilot study, we tried to annotate a token-level dataset based on GPT-3 generated text. However, we found that annotators had trouble achieving consensus if we don't provide the original text. The size of the resulting data would be small. We thus reduce the ambiguity and subjectivity in the annotation process by asking if the pinpointed position in perturbed text is consistent/hallucinated compared with the original reference text. 4 It is possible to substitute the original tokens with more or fewer of tokens. However enumerating all possible token lengths is difficult, and empirically we see marginal gain in diversity in the resulting perturbed text. In our experiments we use same number of tokens for replacement. each position of the text span, to avoid duplicated text after perturbation. We compared several decoding strategies in token substitution, including greedy, top-k (k=5/10/50) and top-p (p=0.95/0.9/0.8) (Holtzman et al., 2020) sampling methods. For comparison we sample 30 perturbed text for each sampling method and count the number of incoherent perturbations. We choose top-k (k=10) sampling as its good trade-off between diversity (via number of distinct tokens) and coherence (via number of incoherent perturbations). iii) For each perturbed text, we substitute multiple word spans. Although being locally coherent, the perturbed text may still exhibit some global incoherence and syntactic issues, especially for longer text. We thus postprocess the perturbed text with a RANK operation as an additional screening step. For each raw text, we generate 20 perturbed candidates and rank them according to language model perplexity using a GPT-2 (117M) model. We only keep the the candidate with lowest perplexity to ensure the fluency and syntactic correctness. We ended up with  1M perturbed text segments in the pool after contextual perturbation, not all of which contain hallucination, as the BERT model can generate factual information given that it is pretrained on a rich open web corpus. Thus, we sought to further annotate the automatically perturbed texts via crowd-sourcing. Human annotation is prohibitively expensive at this scale, so instead of annotating all 1M perturbed texts, we annotated a subset that is less trivial and would lead to a more balanced distribution, using an iterative model-in-the-loop annotation approach that is conceptually related to active learning (Cohn et al., 1996; Jia and Liang, 2017; Zellers et al., 2018; Nie et al., 2020). Human annotation settings To perform the annotations, we hired judges on an internal (the name is redacted for double-blind review) crowd-sourcing platform comparable to AMT. The judges were limited to the North American English speakers with good records (recognized as experts in the platform, rejection rate  1%) and were screened via a simple 10-question qualification test (answer-ing 8 out of 10 questions correctly). They were paid 0.15$ per HIT, which is more than prevailing 6725 local minimum wage. Protocols were implemented to block spammers in real time 5 . For each annotation, both original text and perturbed text were shown to the judges, with perturbed text span highlighted. The annotators were asked to determine whether the perturbed text spans are H (halluci-nation) or N (not hallucination) with the original text in terms of factualness and semantic coherence given the context. Each pair was judged by 4 annotators, and up to 6 if consensus was not reached. We retained only those annotations for which consensus was reached. Out of 12,719 annotated instances, 86.12% instances reach consensus and are included in HADES dataset; 78.47% instances reach  80% agreement among annotators, e.g. 4/5 or 5/6 vote for hallucination label; 71.24% instances reach 100% agreement in the annotation. For inter-annotator agreement (IAA), the Krippendorf's alpha between the annotators is 0.87. Iterative Model-in-the-loop annotation Annotating all perturbed text segments is expensive and time-consuming. Thus, we resort to annotating a subset. We applied two principles for selecting the data to be annotated: i ) the data should be balanced . We found that with randomly sampled instances, the annotated label distribution is heavily skewed toward the hallucination class. Presumably most contextualized perturbations result in factual inconsistency to certain extent. However, we aim to have the number of instances in both classes on par with each other, so that the ROC (receiver operating characteristic) curve of tested models can be better characterized. ii ) the data for annotation should be less trivial 6 . The obvious instances contribute little to model training and method benchmarking, but cost as much annotation effort as other instances. The challenge is that we cannot know a priori the annotation labels and ease of labeling, hence selecting less trivial instances and forming a balanced label distribution for annotation is not straightforward. To address this challenge, we adopt an iterative Model-in-the-loop annotation strategy. Specifically, we split the annotations into several rounds. 5 If a worker keeps choosing the same label for all HITs, or the average time spent per HIT is less than 10 seconds, or more than 30% of their judgments conflict with others', we would manually check their annotations and block the spammers. 6 Many perturbations are trivial to predict, e.g. replacements that change a specific date to a non-date-related phrase must be a hallucination. For each round 7 , we first retrain a hallucination detection model (initiated with BERT) based on the annotated instances in the previous rounds. This model is used for selecting the next batch of data to be annotated from the remaining unlabeled data. To filter out trivial instances and focus on the more useful cases, we use a heuristic rule for the automatic screening by abandoning instances where the detection model assigns low or high probability to hallucination class (the threshold varies in different rounds to yield reasonable number of can-didates). To eliminate cases where the perturbed text paraphrases the original text, we also measured the cosine similarity between the replaced text (through [CLS] representation) and corresponding original content using a RoBERTa model (without fine-tuning), and then filtered out cases with a similarity score greater than 0.9. We also remove a large portion of obvious hallucination instances where the target text span is recognized as a DATE or NAME , and replaced by a different DATE 8 or NAME . In the initial rounds of annotation, we observed extreme label imbalance (around 90% are H class) between H (hallucination) and N (not hallucination) cases. To rebalance the label distribution so that each class received a decent amount of annotation, we performed additional subsamping based on the label predicted by the aforementioned detection model. We assume the human annotation for H and N cases is the oracle, indicating actual H / N . Since the actual hallucinated is dominant, we seek to subsample from instances that are predicted as H by the detection model to make the distribution of actual H / N even. To do this, we estimate the true positive rate (TPR,  ), true negative rate (TNR,  ) and true precision (  ) of the detection model based on the annotation from last round. The hope is that after subsampling, the actual H (TP + FN) is roughly equal to actual N (FP + TN). The estimated subsampling ratio R for the predicted H (TP + FP) is given by 9 : R =  2  +  +  +    (2   1)  (1   ) (1) 7 Except the first round, where we use random sampling. 8 We only remove cases where the replaced date is definitely different (e.g., from Monday to Tuesday). We do not remove ambiguous cases such as from today to Tuesday. 9 Details are provided in the appendix. Data statistics In total, after accumulating annotations for several rounds, we obtain 12,719 instances with 71,226 HITS from judges. We conduct 14 rounds of annotation, increasing the annotation scale with each round (ranging from  200 instances/round to  4000 instances/round). Out of 12,719 annotated instances, 10,954 instances reached consensus among judges and are included in the HADES dataset. We split the dataset into train, validation and test sets with sizes of 8754, 1000, 1200 respectively. In the final dataset, hallucination cases slightly outnumber not hallucination cases, with a ratio of 54.5%/45.5%. We summarize some typical hallucination types seen in the HADES dataset in Fig 3. Parsing features In Fig 4 we show the ratio of hallucination( H )/ not hallucination ( N ) cases for different Part-of-Speech (POS) and Name Entity Recognition (NER) tags, identified by Spacy . From a POS perspective, around two-thirds of verbs and verbal phrases in the dataset are identified as not hallucination, while in other types of words/phrases, hallucination cases are in the majority, e.g., most adverbs (ADV), adjectives (ADJ) and acronyms of proper nouns (PROPN) are labeled as hallucination. Presumably many verbs or verbal phrases are lower in word concreteness (Nelson and Schreiber, 1992) than other word types ( e.g. make and create can be used interchangeably in many circumstances), and thus, as we observe in our dataset, are less prone to be perturbed into hallucinations. For NER tags, about 90% of word spans are not recognized as name entities. However, of the 10% of remaining instances, over 90% are hallucination cases. Statistical and model-based features To analyze the characteristics of hallucinations in HADES , we compute the correlation between a selected group of statistical/model-based features and hallucination labels. As shown in Table 1 10 , we obtain the average word probability and average word entropy of a given text span with a BERT base model (without fine-tuning), as well as term frequencyinverse document frequency (TF-IDF), 10 More statistical feature analysis is in the appendix. positive pointwise mutual information (PPMI) features of the given word span. By comparing the features of the two labels ( H / N ) (Table 1A), we observe that in our dataset, hallucinations typically associate with higher entropy. A counter-intuitive observation is that the hallucinations tend to have higher average probability than factually consistent content. We presume the underlying reason might be that the word distribution generated by machine may diverge from the word distribution of real human-written text (Holtzman et al., 2020; See et al., 2019) owing to self-reinforcing the current generation based on previous generation. Consequently, many overconfident generation outputs are likely to fall into hallucination. We observe no strong correlation between hallucination labels and TF-IDF or PPMI as demonstrated in Table 1B. As an initial step towards tackling the proposed hallucination detection task and benchmarking methods, we create several baseline detection models 11 Feature-based models As elaborated in Sec 3.4, the statistical/model-based features like average word probability, average entropy, TF-IDF, PPMI, as well as parsing features like POS and NER tags can be vague indicators of hallucinations. The former two are context-aware and the latter four are not. We incorporate them as features to build classifiers including logistic regression ( LR ) and support vector machine ( SVM ) using scikit-learn (Pe-dregosa et al., 2011). The maximum number of iteration is set as 100, with an early-stop strategy which stops training if the loss does not drop within 11 The proposed token-level, reference-free hallucination detection hasn't been covered in the existing literature. Thus this thread is first-of-its-kind. We are unable to find a feasible baseline that perfectly fits in our setting, therefore we propose multiple feature-based/pretrained baselines. Transformer-based models We also build baseline detection models based on pretrained transformer models including BERT, GPT-2, XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2020). These transformer-based models represent the state-of-the-art, and can potentially better leverage context or embedded world knowledge to detect self-contradictory or anti-commonsense content. Specifically, for an input text segment, we fine-tune a pretrained model M to predict binary hallucination labels y for each given text span. During inference time, from the last layer hidden states H  R l  h ( h, l are hidden size and sequence length, respectively) of M , suppose the target text span starts at position s and ends at position t , we first obtain the representation w  R h for the target span with max pooling ( i.e. , w = max pool( H s : t ) ). We then map w to a binary hallucination label y  { 0 , 1 } with a MLP network using tanh as activation. During training time, we fine-tune the model using cross entropy objective between the predicted labels and the actual labels. Baseline configurations For the transformer-based baselines, we experiment with a variety of pretrained models via Hugging Face Transformers (Wolf et al., 2020), including BERT large (335M), GPT2 -medium (345M), XLNet large (340M), RoBERTa -large (355M). We use Adam optimizer (Kingma and Ba, 2015) with different learning rates, i.e. 5e-3 for GPT2 and BERT and 1e-3 for other models. We explored multiple model architectures and setups to determine the optimal configuration using BERT-large model. These include i ) span representation with mean/max pooling ; ii ) number of layers of the MLP network; iii ) hidden dimension 6728 Model Acc G-Mean (  ) BSS (  ) AUC Not Hallucination Hallucination P R F1 P R F1 LR 62.25 60.77 -62.35 72.08 66.86 62.10 51.24 60.33 SVM 63.67 61.50 -62.89 76.18 68.90 65.05 49.65 56.31 BERT 71.92 71.95 19.06 78.63 74.46 71.29 72.84 69.31 72.61 70.92 RoBERTa 72.83 70.94 18.78 78.72 74.06 74.76 74.41 71.43 70.67 71.05 XLNet 72.33 71.39 18.79 78.93 71.15 80.13 75.37 74.07 63.60 68.44 Table 2: Benchmark (numbers in percentages (%)) for the offline setting on HADES , where detecting models have access to the bidirectional context. of the MLP ; iv ) whether or not to freeze the parameters of M up to the last layer, and choose the best configuration according to model performance on the validation set. The best configuration uses max-pooling, employs 2 layers of MLP with hidden dimension of h/ 2 , and freezes the model parameters up to the last layer of M and just fine-tunes the binary MLP classifier. We apply the same network configuration to all other pretrained models as empirically we see marginal performance gain after enumerating different configurations for individual pretrained models other than BERT. As discussed in Sec.2, HADES can serve as benchmark for hallucination detection in both offline (model can see bidirectional context) and online (only preceding context can be leveraged) settings. Note that we apply the feature-based baselines only in the offline setting (Table 2), because a good estimation of those features requires bidirectional context. The transformer with causal attention (GPT-2) can only fit in the online setting. Evaluation metrics We evaluate the baselines on HaDes with standard classification metrics including accuracy, precision, recall, F1 and AUC (Area Under Curve) with respect to ROC. We also utilize the G-Mean metric which measures geographic mean of sensitivity and specificity (Espndola and Ebecken, 2005) and they were reported useful especially for the imbalanced label distribution scenarios. We also employ the Brier Skill Score (BSS) metric (Center, 2005), which calculates the mean squared error between the reference distribution and the hypothesis probabilities. Baseline performance Table 3 and Table 2 show the performance of the baseline models 12 in both online and offline settings respectively. In both settings, the predictions for not hallucination cases have higher F1 scores than hallucination cases. All models perform better in the offline setting compared with the online setting, indicating that the succeeding context of the target words helps identify hallucinations. The transformer-based baselines are generally on par with each other. Under the offline setting, the pretrained models outperform feature-based models by a large margin; this indicates that the powerful contextualized feature extractor is important for successfully identifying hallucinations at fine granularity. Under the online setting, we observe that, for most of the metrics, GPT-2 yields the best performance of all baselines. 12 To identify the clear winner among baseline models, we report the significant tests for the baseline models in Table 3 and Table 2 as follows: For the offline setting (Table 2), there is no obvious winner among pretrained models, e.g. RoBERTa wins in ACC; XLNet wins in F1 for not hallucination cases; BERT wins in G-mean. For the online setting (Table 3 ), we ran significant tests for the mean performance (over 5 runs) between GPT-2 and BERT; GPT-2 and XLNet; GPT-2 and RoBERTa, the differences in terms of ACC; G-mean; F1 scores for both hallucination and not hallucination labels are significant (alpha=0.01) after Bonferroni correction. Presumably, the causal language model pretraining method makes GPT-2 perform better in the auto-aggressive (online) detection setting. Context matters in HADES To investigate extent to which contextual information helps the hallucination detection in HADES , we run BERT-large detection model with different context lengths and characterize its performance in both online and offline settings in Fig 6. Starting from the target words, we set a fixed size (5/10/20/40/80/160) context window and truncate all text beyond this window. As we enlarge the context window, model performance grows rapidly when context length is smaller than 80, and then gradually converges. This observation highlights the importance of context in hallucination detection. Interestingly, we observe that the model obtains higher performance in the offline mode than in the online setting. The performance gap between the two settings maximizes when context length is around 75, and vanishes with long ( > 150 ) or short ( < 20 ) context windows. We surmise that for long ( > 150 ) context window, the preceding context information might already be adequate for detection, while for short ( < 20 ) context windows, the context, regardless whether it is unidirectional or bidirectional, might not contain enough information for detection. Model predictions on GPT-3 generated text We visualize the predictions of BERT-large (offline) model on GPT-3 generated text in Fig 5. According to the 2021 census instruments 13 , some identified spans like greenhouse gas emission and com-plete enumeration are indeed not included in the census, we assume they are recognized due to the topic or knowledge irrelevance with the census of agriculture in the pretrained corpus. Interestingly, the detection model predicts the high hallucination risk on structures and buildings, which has subtle differences with total greenhouse area including enclosed structures (included in the instruments). The case study demonstrates the potentials of our model in identifying hallucinated content in the actual outputs of large-scale pretrained models. Reference-based Hallucination Detection Apart from human verification (Chen and Bansal, 2018), researchers have developed effective reference-based methods which automatically detect hallucination in the generated text using statistical n-gram matching (Dhingra et al., 2019; Liu et al., 2019), edit distance heuristics (Zhou et al., 2021), natural language inference (Kryscin-ski et al., 2020; Falke et al., 2019), information extraction (Zhang et al., 2020; Goodrich et al., 2019) or question answering (Scialom et al., 2019; Eyal et al., 2019; Wang et al., 2020a). Our approach differs from them in that we investigate the reference-free hallucination detection scenario. To reduce hallucinations in the reference-based setting, researchers have applied iterative training (Nie et al., 2019), post editing (Dong et al., 2020), 13 https://www.statcan.gc.ca/en/ statistical-programs/instrument/3438_Q1_V6 6730 soft constraints, e.g. attention manipulation (Kid-don et al., 2016; Hua and Wang, 2019; Tian et al., 2019) or optimal transport (Wang et al., 2020b), and template/scaffold guided schema with explicit plans (Ma et al., 2019; Moryossef et al., 2019; Bal-akrishnan et al., 2019; Du et al., 2020; Liu et al., 2021), e.g. text sequences which specify the narrative ordering, and implicit plans (Wiseman et al., 2018; Ye et al., 2020; Shen et al., 2020; Li and Rush, 2020), e.g. (structured) hidden variables that corresponds to certain surface realization. Reference-free Detection Approaches Reference-free hallucination detection is closely related to fake news detection (Zellers et al., 2019; Zhou and Zafarani, 2020; Zhong et al., 2020), which aims to identify deliberate disinformation in a reference-free manner on social media and usually involves common-sense and world knowledge reasoning (Monti et al., 2019), or fact checking (Thorne et al., 2018), where practitioners are asked to verify given claims without references by retrieving related evidence from Wikipedia. Another line of research is to classify sentence-level language specificity (Li and Nenkova, 2015; Gao et al., 2019), which scales from 1 (very general) 5 (very specific) for short text, e.g. tweets, according to human annotation. The proposed hallucination detection aims to examine the text in a finer granularity than fake news detection and fact checking. In the proposed task, most parts of the text remain faithful; our goal is to identify subtle hallucinations at the token-level. Fake news detection or specificity assessment, on the other hand, usually focus on sentence-or document-level detection. We have proposed a token-level reference-free hallucination detection task and introduced a benchmark dataset HADES for identifying fine granularity hallucination in free-form text generation. To create this dataset, we perturbed texts to simulate hallucination in NLG system, and performed an in-terative model-in-the-loop annotation approach to annotate the perturbed text in an imbalanced label scenario. We have further provided comprehensive analyses of HADES and evaluated several baseline models to establish initial benchmarks. We hope that the proposed task and dataset will shed light on high-resolution hallucination detection in freeform text generation and will eventually lead to real-time hallucination prevention. This study aims to facilitate the recognition of potential hallucinated content produced by large-scale pretrained models in the free-form generation. We support this goal with a novel reference-free, token-level hallucination task and the corresponding annotated dataset HADES. The detection model trained with HADES could be potentially useful in both online and offline settings. For online settings it is possible to guide beam search or suppress the probability of hallucinated tokens through the detection models. For offline settings our system may expedite the human-in-the-loop post-examination in product deployment. We design our model to detect hallucination to factual statement. The learned knowledge should be able to be transferred to other domain like social chatbot once the chat is regarding certain facts (e.g. a celebrity, a historical event). Wikipedia dataset covers a lot of facts, domains and topics, making it ideal for our study. We thus collect the HADES dataset from Wikipedia. All text on Wikipedia is licensed under the Creative Commons Attribution/Share-Alike 3.0 Unported License. During the annotation, all involved annotators voluntarily participated with decent payment. The authors would like to thank the anonymous reviewers for their thoughtful and constructive comments. Tianyu and Zhifang gratefully acknowledge the support of the National Key Research and Development Program of China 2020AAA0106701 and National Science Foundation of China project U19A2065.
abstain abstain abstain objective objective method method abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method objective abstain abstain abstain objective method method objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method method abstain abstain abstain method method method abstain method abstain method abstain method method abstain abstain method abstain abstain method abstain method method method abstain abstain abstain abstain method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method abstain abstain abstain method method method abstain method method method abstain abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain method abstain method method method other abstain abstain method abstain method abstain method method method abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other objective other other other other objective other objective method method objective abstain abstain abstain abstain abstain method abstain abstain method abstain abstain other other