The largest store of continually updating knowledge on our planet can be accessed via internet search. In this work we study giving access to this information to conversational agents. Large language models, even though they store an impressive amount of knowledge within their weights, are known to hallucinate facts when generating dialogue (Shuster et al., 2021); moreover, those facts are frozen in time at the point of model training. In contrast, we propose an approach that learns to generate an internet search query based on the context, and then conditions on the search results to finally generate a response, a method that can employ up-to-the-minute relevant information. We train and evaluate such models on a newly collected dataset of human-human conversations whereby one of the speakers is given access to internet search during knowledge-driven discussions in order to ground their responses. We find that search-query based access of the internet in conversation provides superior performance compared to existing approaches that either use no augmentation or FAISS-based retrieval (Lewis et al., 2020b). Open-domain dialogue, which involves chat about any topic, rather than a specific goal-directed topic, is commonly studied by training large language models (Adiwardana et al., 2020; Zhang et al., 2020; Roller et al., 2021). These models are trained either in a encoder-decoder or decoder only setting on large datasets of human-human conversations, and any knowledge obtained during training is stored in the weights of the model. Such static language modeling fails to take into account the dynamic state of the world, where new information is coming in by the day  or even by the minute  as the knowledge in static models is gleaned from the point in time when the dataset was collected, and then frozen into the model that is trained; see Figure 1: Cherry picked example of a model with internet-augmentation trained on our new Wizard of the Internet task. (Lazaridou et al., 2021) for criticisms of this approach. Further, static language models are known to hallucinate , that is they generate plausible looking statements that are factually incorrect, which can be interpreted as a form of lossy compression when employing training to encode that knowledge within the weights of a neural network; see Shuster et al. (2021) for an in-depth study. In this work we study generative models that are instead capable of accessing the vast knowledge of the internet dynamically in order to inform their responses. Utilizing encoder-decoder archi-8460 tectures, we consider models that, given a dialogue context, first generate a search query. The queries are then used to retrieve relevant knowledge that is prepended to the conversational history, which is encoded using the Fusion-in-Decoder method (Izacard and Grave, 2021). Taking into account this encoded knowledge, a response is finally generated using the decoder. This ability to access the internet means the model is always up-to-date, unlike existing models that only know about facts in their fixed training set. Our model, in contrast, can potentially make use of the latest sports scores, movies or TV shows that were just released, the latest reviews, and so forth  amongst the countless other topics available on the internet. In order to train and evaluate such models, we collect a new crowdsourced English dataset involving human-human conversations, where one of the workers plays the role of a wizard who conducts internet searches in order to inform their responses during knowledge-grounded conversations. We show that internet-augmented models trained to replace the human wizard outperform conventional non-augmented generation models on this task as measured by automatic metrics as well as human evaluations, and with our search query generation based approach also outperforms existing retrieval-augmented FAISS-based approaches such as RAG (Lewis et al., 2020b) and FiD-RAG (Shuster et al., 2021). We make our final models and the new task we have collected, publicly available. 1 . The majority of work on dialogue generation has focused on training on natural or crowdsourced data where the task is, given a dialogue context (history), to generate the next response. Datasets such as pushshift.io Reddit (Baumgartner et al., 2020), PersonaChat (Zhang et al., 2018) or Empathetic Dialogues (Rashkin et al., 2019) (see Huang et al. (2020) for a review) are typically employed to train the weights of a Transformer encoder-decoder. This is the standard approach in state-of-the-art chatbots such as Meena (Adiwardana et al., 2020) or BlenderBot (Roller et al., 2021). Such models do not augment their generations with access to external knowledge, instead relying on facts originally provided in the training datasets themselves being stored into the weights of the model. ing generative models with external knowledge. Earlier works such as Memory Networks (Weston et al., 2015) and DrQA (Chen et al., 2017) utilized TFIDF-based retrieval over documents to provide additional input to neural models for the task of question answering, following the well studied area of non-neural methods that use retrieval for QA (Voorhees and Tice, 2000). More recently, the RAG (Retrieval-Augmented Generation) (Lewis et al., 2020b) and FiD (Fusion-in-Decoder) (Izac-ard and Grave, 2021) models developed these ideas further, using a neural retriever as well, with superior results. Retrieval-augmentation is also studied in the area of language modeling, where it is used for pre-training (Guu et al., 2020), and as a memory (Yogatama et al., 2021), especially using k -nearest neighbor-based cache models (Khandelwal et al., 2021, 2020; Grave et al., 2017; Merity et al., 2017). In dialogue, knowledge grounding is becoming more popular an area, with several datasets developed to study it (Zhou et al., 2018; Dinan et al., 2019; Ghazvininejad et al., 2018; Gopalakrishnan et al., 2019; Galetzka et al., 2020). Some of these such as Topical-Chat (Gopalakrishnan et al., 2019) and CMU_Dog (Zhou et al., 2018) are constructed given a gold passage of knowledge, and the task analyzes whether the model can use this knowledge in dialogue. Other works (Zhao et al., 2020; Kim et al., 2020; Bruyn et al., 2020) study whether knowledge selection is possible from a (small) set of knowledge. However, a retrieval step (or search engine) is not used, as we consider here. Perhaps the closest to our work is the Wizard of Wikipedia task (Dinan et al., 2019) which involves conversations grounded in Wikipedia, using a TFIDF retrieval model to find relevant knowledge from that database. Our work can be seen as a much richer task, covering all of the information that is publicly available on the internet and hence a more diverse range of conversational topics rather than just Wikipedia, while allowing human wizards to search for relevant knowledge themselves. Moreover, we consider sophisticated neural-in-the-loop retrieval mechanisms and real search engines. Shuster et al. (2021) studied neural-retriever-in-the-loop methods on this dataset. neighbor database, FAISS (Johnson et al., 2019), or (ii) using an Internet Search Engine directly to retrieve pages. For the FAISS-based methods, there are a number of possible variants that we consider, which we will describe first. In our experiments, the FAISS-based methods share the same core setup. First, we store and utilize the Common Crawl dump of the internet from Wenzek et al. (2020) 2 in a FAISS database, with keys that are dense vectors. The retrieval system uses a DPR (Dense Passage Retrieval) (Karpukhin et al., 2020) Transformer-based model which scores document-context pairs in order to rank them based on their match using a bi-encoder framework, where the base DPR model is pre-trained on QA data pairs. We use the pre-trained DPR model from the KILT Benchmark (Petroni et al., 2021). The documents (webpages) are encoded using DPR into dense vectors and these are stored in the FAISS index. During dialogue-based retrieval, the dialogue context is also encoded by DPR into a dense vector and FAISS approximate nearest-neighbor lookup is performed, where the top N documents are returned. We then consider several recent neural methods for utilizing this retrieval mechanism in various ways. RAG (Retrieval Augmented Generation) RAG (Lewis et al., 2020b) is an approach which consists of two components which are trained end-to-end: (i) the neural-in-the-loop retrieval system; and (ii) an encoder-decoder for generating final responses given the results of the retrieval. Using DPR, the top N documents are returned as described above, and in the RAG-Token model (just called RAG in the rest of the paper) each in turn is encoded along with the context for each token, and the most likely sequence is generated from the set. During backpropagation training steps, the DPR context encoder is also tuned to perform well at FAISS retrieval, but the document encodings are held fixed. This approach has been shown to optimize both retrieval and generation jointly, improving results. 2 We use the November 2020 dump, head only, consisting of  109M English webpages. Each document is split into 100-word chunks, giving 250M passages to index in FAISS. We also consider the dump of Wikipedia from (Karpukhin et al., 2020) in this work. 2021). In this case, the pre-trained retriever is used, i.e. DPR with FAISS, and then each of the top N documents returned is prepended to the context and encoded separately by the encoder, and finally all the results are concatenated. The decoder then attends to these encodings to produce a final response, so all fusion happens in the decoding stage. This relatively simple method was shown to outperform RAG in some cases. FiD-RAG The FiD approach works well, but there is no end-to-end training of the retriever in that case, and so it relies completely on being pre-trained well, as opposed to RAG which tunes the retrieval for generation. FiD-RAG, proposed in (Shuster et al., 2021) combines the two methods. First the retriever is trained in a RAG setup, and then FiD is used with that retriever. This was shown to give superior results to both RAG and FiD on dialogue tasks. FAISS + Search Query-based Retrieval Instead of just encoding the context into a dense vector, in this approach an encoder-decoder is employed to generate a search query given the context. The search query is input into a DPR model to produce a dense vector, and is matched to documents in the FAISS index. Returned documents can then be used in the final response generation encoder-decoder as before. Any of the existing approaches (RAG, FiD or FiD-RAG) could potentially be used to fuse the DPR and generator models. We used the standard DPR FiD setup. The previously described FAISS-based approaches can take advantage of many existing methods developed for QA and dialogue tasks, as we saw, but have several disadvantages. First, they may be difficult to update to real-time web documents; second, there may be a limit to the number of documents storable in local FAISS deployments; and third, such methods will not take advantage of the high quality ranking that has been finely tuned in Internet Search engines over decades of use. We thus consider using Internet search engines directly. Method Our proposed method consists of two components: 1) A search query generator: an encoder-decoder Transformer that takes in the dialogue context as input, and generates a search query. This is given to the black-box search engine API, and N documents are returned; 2) A FiD-style 8462 Train Valid Test Total Dialogues 8,614 516 503 9,633 Utterances 82,952 5,781 4,932 93,665 Avg. encoder-decoder model that encodes each document individually, concatenates them to the dialogue context encoding, and then finally generates the next response. We can train each of these modules separately if we have supervised data available for both tasks, the first module requiring (context, search query) pairs, and the second module requiring (context, response) pairs. As we will see, the data we collect in this work (detailed in section 4) fulfills both of these requirements. For FiD, we try two methods: (i) Conventional FiD whereby we use the returned search results from using our trained search query generator in order to build the relevant document contexts for the FiD training set; (ii) FiD-Gold: as we will have available human-written search queries for the training set, and their corresponding search results, we can use these gold results to build training document contexts instead. Although these might not look like the queries and hence results predicted at test time, they are more likely to contain the knowledge used in generating the training set responses, thus a clearer grounding may be apparent for the model to learn correspondences. Search Engine The search engine is a black box in this system, and could potentially be swapped out for any method. In our numerical experiments we use the Bing Search API to generate a list of URLs for each query; then, we use these URLs as keys to find their page content from a lookup table we built for our Common Crawl snapshot, in order to populate a set of pages for that query. This makes our comparison more direct with our FAISS-based methods. In addition, we can also consider if the URL is from English Wikipedia, in that case we can extract the page title from the URL and look up its corresponding page inside the dump of Wikipedia. In order to both train and evaluate generative models that can use search engines in-the-loop, we design, collect and release a dataset for this purpose. The overall setup involves pairing crowdworkers that are instructed to have a conversation together. One plays the role of the wizard , who has access to a search engine during conversation, while the other, the apprentice , does not. The apprentice however has an assigned persona that describes their interests. The purpose of the exchange is to have an in-depth conversation about [those] assigned interests. This mirrors conversations we expect to be more prevalent between a human and a bot: the conversations are more likely to be centered around the human's interests than the bot's, and the bot is the one that is going to be using the search engine to ground their knowledge. Hence, when we train or evaluate on this task, a given model will replace the role of the wizard. Apprentice Persona We show the apprentice several possible persona choices for the character that they are going to play, and let them choose one, e.g. I love tennis, Rafael Nadal is my favorite player.. The intent here is that they can choose a topic they are both more interested in themselves to talk about and also have enough knowledge of so that they can conduct a reasonable conversation. The choices we show are themselves mined from the interests provided in the existing Persona-Chat dataset (Zhang et al., 2018) and the topics given in the existing Topical-Chat dataset (Gopalakrishnan et al., 2019). More details of the choices we give 8463 are provided in Appendix A. Wizard Active and Passive Openings We randomize which speaker takes their turn first. If the wizard speaks first, we encourage them to start with an opening that addresses the apprentice's interests. For example, if they know their partner is interested in tennis, they could search for the latest tennis news, and open with an interesting point based on that knowledge. If the apprentice goes first, their goal is to converse with the wizard more based on their own interests, e.g. in this same case they could talk about tennis in detail. Wizard Search At each turn, the wizard can enter free text search terms in a left-hand panel (with the main conversation panel on the right) much like in a conventional search engine. The top few results are shown in the left panel, below the search query 3 . For each document the titles are shown for space reasons, and each document is expandable. If the wizard finds one or more search results useful for their response, they can click on the sentences they find relevant, and then enter their conversational response in the right-hand panel. They are also free to try another search query if they did not find their first results appropriate, or else can enter a conversational response and choose to ignore the search results entirely. Full System Each crowdworker has to pass an on-boarding task to be able to be part of the main data collection task, and pass some automatic checks (average response length, use of search). They are asked to play a particular role ("Create an interesting character that you want to play"), and are given instructions to avoid toxic or biased language. We randomly assign for any given crowdworker a fixed choice of either wizard or apprentice for all of their data collection, otherwise we found that switching role introduced lower quality conversations, probably due to confusion between the different goals and instructions per role. After pairing, we collect between 5-6 turns (10-12 utterances) for each conversation. We ask workers to skip initial greeting messages, as these bring little extra value to the task. Screenshots of the crowdworker task can be seen in Figure 4 in the appendix. Example collected dialogues are shown in Figure 5 and Figure 6 3 We run two searches, one with the given query, and one with the query terms plus the word news (with the news results shown as the top two knowledge candidates), in order to encourage topical discussion. The overall collected data consists of 9633 dialogues in total, with 82952 utterances in the training set, and validation and test sets of 5781 and 4932 utterances, respectively. Overall statistics can be found in Table 1. We find that 84.81% of all turns by the wizard involve search, so a large amount of knowledge grounding based on internet results is taking place. Of those, the wizard is allowed to repeat the search with different search terms if they did not find what they were looking for. When the wizard searches, we find 1.19 search queries are performed on average, so while mostly a single search is employed, a number of further knowledge searches are attempted. Wizards use the search results (indicated by selecting relevant sentences) 80.3% of the time. We show in Figure 2 a breakdown of the most common domains used during search on the validation set. We see that the domains are rather diverse, coming from all kinds of topics, and in particular that the Wikipedia domain is actually fairly small (8.56% of queries), which is interesting because most other studies have used Wikipedia only as their knowledge resource (Chen et al., 2017; Lewis et al., 2020b; Dinan et al., 2019; Shuster et al., 2021). Our training set spans 26192 unique selected URLS for grounding knowledge from 10895 domains, indicating a wide variety of topics and knowledge is used across all conversations. We evaluate models on our new Wizard of the Internet (WizInt) task, using its dedicated training set. We also consider the existing Wizard of Wikipedia (WoW) training resource as well, either for building baselines or for multi-tasking. We consider fine-tuning various existing pre-trained models: T5 (Raffel et al., 2020), BART-Large (Lewis et al., 2020a) and BlenderBot variants (Roller et al., 2021). For all retrieval-augmented methods we use N = 5 returned documents. For all models, when generating responses we fix the decoding parameters to beam search (beam size 3) with a minimum sequence length of 20 and beam blocking of 3-grams within the response (but not the context), similar to choices in (Roller et al., 2021). Following Shuster et al. (2021) we report perplexity, F1 and Knowledge F1 (KF1) metrics. F1 measures the overlap between the model's response and the human response from the dataset. KF1 instead measures the overlap between the model's response and the knowledge on which the human grounded during dataset collection (i.e., the sentences they clicked as relevant from the web search documents retrieved, see section 4). We note that KF1 and F1 can be traded off, for example a model that could copy the knowledge directly would have a high KF1 but a low F1  it would be knowledgeable, but not conversational. Nevertheless, we expect an ideal model would achieve relatively high values for each. Finally, we also perform a human evaluation, the details of which will be discussed further in subsection 5.3. Pre-training models We evaluate the performance of using different standard pre-training models when training on our new task. Results are given in Table 3. Comparing BlenderBot (BB) 400M and 2.7B parameter models, which use the same dictionary, we see that larger models do improve all metrics (perplexity, F1 and KF1) in the no knowledge case (where the model is given only the conversational history, with no web documents). When given gold knowledge (the selected knowledge sentences and the conversational history are given as input to the model), this trend is slightly less clear, but still present. BART-Large and T5-Large, which are trained on more knowledge focused corpora, rather than the conversational corpora of BB, give improved performance for the same model size in terms of F1 and KF1 metrics. We choose to use BART-Large as our base for all of our following experiments. No knowledge vs. gold knowledge baselines We compare Transformers that are given only the dialogue context (no knowledge) to Transformers that are given both the dialogue context and the gold knowledge from the task which human annotators (wizards) labeled as being used to craft responses. They can be compared in Table 3 across different models. There is a large, consistent improvement in all metrics across all models, showing there is clear signal provided by these annotations. While in practice gold annotations will not be available, this can be seen as both an upper bound on possible performance, as well as confirmation that knowledge retrieval has the potential to bring sig-8465 nificant gains over non-retrieval augmented (no knowledge) models. Wizard of Wikipedia baselines We train models on the Wizard of Wikipedia (WoW) dataset as baselines, to compare the difference between coverage of the WoW task and our new WizInt task, in both the no knowledge and gold knowledge settings. Results are given in Table 4, evaluating on both the WoW and WizInt validation sets. We observe some overlap between the tasks, as expected, but also observe some differences. Perplexity improves from 20.4 to 17.4 and a corresponding boost in F1 of 15.8 to 17.6 from training with WizInt and evaluating on the WizInt task in the no knowledge setting, compared to training with WoW. Similarly, the WoW task provides better training data for its own task. We draw similar conclusions in the gold knowledge case as well. KF1 on the other hand appears to be less influenced by the dataset in the no knowledge case, and in the gold knowledge case the WoW model has a higher KF1, perhaps because the model has learnt to copy effectively, but has a poor F1, presumably because it is not generating as appropriate responses due to this copying. Multi-tasking with Wizard of Wikipedia We can also multi-task the WoW and WizInt tasks together, perhaps bringing improvements as we have shown they have some similarity in their tasks. Results are also given in Table 4. We observe a small gain in perplexity on both the no knowledge and gold knowledge WizInt tasks, and improvements in F1, e.g. from 17.6 to 18.0 on the no knowledge task, and from 25.4 to 26.3 on the gold knowledge task. In the majority of our subsequent experiments, for the sake of simplicity we do not perform such multi-tasking, but we expect similar gains could be achieved if we were to apply this elsewhere. DPR+FAISS-based models We trained DPR+FAISS-based models using either the WoW or WizInt training datasets, and either Wikipedia or Common Crawl (CC) as the database. Results of the most salient methods on the test set are given in Table 2, with full results on the validation set in Table 9. Comparing to WoW-trained Transformers with no augmentation (no knowledge), we find the WoW-trained DPR+FAISS-augmented methods using FiD give unclear improvements: there is no improvement in F1 using Wikipedia as a database, and a small improvement in F1 (from 14.7 to 15.3) when using CC, as measured on the test set. Moreover, perplexity in both cases increases (e.g., from 22.3 to 22.8). However, FiD-RAG performs better, improving F1 from 14.7 to 15.5 while maintaining the same perplexity. Nevertheless, these WoW-trained baselines fail to match even the non-augmented no knowledge Transformer trained on WizInt (Table 2, row 2) which has a perplexity of 18.7 and F1 of 16.9. Training DPR+FAISS on WizInt, we also see clear improvements over WoW-trained models, and similar conclusions that FiD-RAG is superior to RAG, with the best approach achieving a perplexity of 17.1 and F1 of 18.0 on the validation set, see Table 9 in the appendix. The impact on the test set however is still fairly minimal, see Table 2. Search Query+FAISS-based models We find that using a search query generator and then using FAISS to retrieve from the database of web documents performs slightly worse than DPR+FAISS-based models. Perplexity is actually no better than the no knowledge model  19.0 for Search Query+FAISS compared to 18.7 for no knowledge. Search Engine-based models The search engine based method provides the best performance in terms of perplexity of all the models tested, with a validation perplexity of 16.4 when trained on WizInt and 16.1 when trained on both Wow and WizInt for the CC+Wikipedia case, see Table 9. While F1 and KF1 metrics are hardly impacted, we do see a similar reduction in perplexity on the test set, see Table 2. We find this encouraging as search engines are already a well developed tool we can simply interface with our model, rather than trying to reinvent storage of all the documents on the internet, as we have attempted with our other FAISS-based experiments. We thus select this method as our main candidate for human evaluations. We perform a human evaluation using crowdworkers. The conversations begin with a random apprentice persona from the WizInt validation set being selected and shown, and the crowdworker is asked to play that role. We ask the crowdworkers to have a natural conversation, where they will also evaluate their partner's responses for conversational attributes, in particular knowledgeability, factual (in)correctness, engagingness and consistency. Screenshots can be found in Figure 7 (in the appendix) which detail further the definitions 8466 Factually Final # Annotated Model Consistent Engaging Knowledgeable Incorrect Rating Responses WizInt Transformer (No Knowledge) 66.5% 69.9% 38.6% 7.1% 3.64 764 Search engine FiD (Bing Search) 76.1% 81.4% 46.5% 5.3% 3.73 757 Table 5: Human Evaluation Results. of those attributes. On each turn of the conversation the crowdworker is asked to check all attribute boxes that apply to the last turn. Each conversation consists of 15 messages (7 from the human, 8 from the bot). At the end of the conversation, an additional question collects an overall engagingness score (out of 5) for their speaking partner. We compared the WizInt BART-Large Transformer (no-knowledge) model, which is a standard Transformer with no retrieval augmentation, to the WizInternet Search engine FiD model, with live Bing search (without using a CC subset). The results are given in Table 5. For each model, around 750 responses were annotated over nearly 100 model conversations. The search engine-based method outperformed the no-knowledge baseline across the board. Not only was the search engine-based model judged to be knowledgeable more often (46.5% vs. 38.6% of the time) and factually incorrect less often (5.3% vs. 7.1%), but it also was measured to be more consistent (76.1% vs. 66.5%) and more engaging (81.4% vs. 69.9% on an utterance level, and 3.73 vs. 3.64 on a conversation level). Success cases In the best case, our augmented models are able to construct appropriate internet search queries, read the corresponding web pages and provide information relevant to the conversation. We show a cherry picked conversations between a human (paper author) and the WizInternet Search engine FiD model (using live Bing search) in Figure 1, and in Figure 8, Figure 9, Figure 10 and Figure 11 in the appendix. In each case, we can compare to a WizInt BART-Large Transformer (no-knowledge) model using the same conversational messages on the human side. We find the search engine model is capable of diverse conversations spanning drink ingredients, TV shows, restaurants and machine learning research. In the TV show and restaurant cases the model is able to surface recommendations and provide details about them, for example the correct address and phone number of a pizza store in Princeton, or the plots of recent TV shows such as The Underground Railroad. Standard BART-Large fine-tuned models on the other hand typically either hallucinate knowledge or else fall back to generic statements. Failure cases Analysis also exposes various kinds of error. Lemon picked conversations between human (paper authors) and the WizInternet Search engine FiD model (using live Bing search) are shown in Figure 12 in the appendix. First, there are generation mistakes despite finding the correct knowledge, for the example where the model incorrectly names Bruno Mars as working on the Cardi B song Bodak Yellow. Bruno Mars did collaborate with Cardi B on other songs, and the model confuses and mixes various pieces of evidence within the given knowledge sources. Second, search query generation mistakes given the context, for example missing out key search terms. Third, selecting the wrong knowledge given earlier context, as in the case where the model associates the wrong authors to a paper. A fourth additional issue is that even if the correct knowledge is available the model may err on the side of not using it and select a more generic response instead, as often happens in the non-augmented model. See for example Figure 8 and Figure 11 in the appendix. This work has studied the problem of siloed knowledge in large language models, whereby they cannot access the knowledge of the world other than through their fixed training set. Developing methods that instead can access the internet as an augmentation to the generation process, we have showed such models can display more knowledge and generate less factually incorrect information during dialogue with humans. Future work should aim to develop improved architectures that can be trained and evaluated on our new task. Going for-ward, in the long-term we require machine learning methods that interact with the world, rather than only having a simple text context  and access to the internet is a natural step in that direction. Large language models bring an impact on the environment in terms of resources required to train and deploy them, and concerns about toxic language, bias and other issues during language generation (Bender et al., 2021). For dialogue in particular, see Xu et al. (2020) for a review of the literature and evaluation of recent methods that try to mitigate these safety issues. The initial pre-training dataset used in this work contains varied and potentially offensive text content, as they were originally procured from the Internet by third parties. However, our fine-tuning task is built with crowdworkers with specific instructions to not use toxic language, a procedure which is shown to yield safer language models (Roller et al., 2021). This work, different to other language generation models, specifically augments the generations with knowledge from the internet. On the one hand, we showed that this results in less model hallucination, and more factually correct generations. Further, as the model generates human readable search queries and one can verify which document(s) the used knowledge comes from, means our model also has increased interpretability and potentially debugga-bility compared to standard language models. On the other hand, this also brings potential new concerns if those websites contain toxic, biased or factually incorrect information themselves. While issues of toxicity can perhaps be treated similarly to the pre-training data case (e.g. safety classi-fiers), fact checking is a separate area with ongoing work, e.g. Hassan et al. (2017); Fan et al. (2020). We further remark however, that the use of internet search engines to augment models, instead of FAISS-based retrieval (Lewis et al., 2020b), means that machine learning models can take advantage of decades of work in search engine safety issue mitigations, rather than having to completely rebuild those tools again.
method method abstain objective method result abstain abstain other abstain abstain method objective abstain abstain abstain objective objective abstain objective abstain other other other other other other other other other other other method abstain abstain method other abstain abstain abstain method method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method method method method abstain abstain method method method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective objective abstain abstain abstain method result objective result result abstain abstain abstain