Active learning promises to alleviate the massive data needs of supervised machine learning: it has successfully improved sample efficiency by an order of magnitude on traditional tasks like topic classification and object recognition. However, we uncover a striking contrast to this promise: across 5 models and 4 datasets on the task of visual question answering, a wide variety of active learning approaches fail to outperform random selection. To understand this discrepancy, we profile 8 active learning methods on a per-example basis, and identify the problem as collective outliers  groups of examples that active learning methods prefer to acquire but models fail to learn (e.g., questions that ask about text in images or require external knowledge). Through systematic ablation experiments and qualitative visualizations, we verify that collective outliers are a general phenomenon responsible for degrading pool-based active learning. Notably, we show that active learning sample efficiency increases significantly as the number of collective outliers in the active learning pool decreases. We conclude with a discussion and prescriptive recommendations for mitigating the effects of these outliers in future work. Today, language-equipped vision systems such as VizWiz, TapTapSee, BeMyEyes, and CamFind are actively being deployed across a broad spectrum of users. 1 As underlying methods improve, these systems will be expected to operate over diverse visual environments and understand myriad language inputs (Bigham et al., 2010; Tellex et al., 2011; Mei et al., 2016; Zhu et al., 2017; Anderson et al., 2018b; Park et al., 2019). Visual Question Answering (VQA), the task of answering questions about 1 Applications can be found at https://vizwiz.org/ , https://taptapsee.com/ , https://www.bemyeyes.com/ , and https://camfindapp.com/ Q: What sport is she playing? visual inputs, is a popular benchmark used to evaluate progress towards such open-ended systems (Agrawal et al., 2015; Krishna et al., 2017; Gordon et al., 2018; Hudson and Manning, 2019). Unfortunately, today's VQA models are data hungry: Their performance scales monotonically with more training data (Lu et al., 2016; Lin and Parikh, 2017), motivating the need for data acquisition mechanisms such as active learning, which maximize performance while minimizing expensive data labeling. While active learning is often key to effective data acquisition when such labeled data is difficult to obtain (Lewis and Catlett, 1994; Tong and Koller, 2001; Culotta and McCallum, 2005; Settles, 2009), we find that 8 modern active learning methods (Gal et al., 2017; Siddhant and Lipton, 2018; Lowell et al., 2019) show little to no improvement in sample efficiency across 5 models on 4 VQA datasets  indeed, in some cases performing worse than randomly selecting data to label. This finding is in stark contrast to the successful application of active learning methods on a variety of traditional tasks, such as topic classification (Siddhant and Lipton, 2018; Lowell et al., 2019), object recognition (Deng et al., 2018), digit classification (Gal et al., 2017), and named entity recognition (Shen et al., 2017). Our negative results hold even when accounting for common active learning ailments: cold starts, correlated sampling, and uncalibrated uncertainty. We mitigate the cold start challenge of needing a representative initial dataset by varying the size of the seed set in our experiments. We account for sampling correlated data within a given batch by including Core-Set selection (Sener and Savarese, 2018) in the set of active learning methods we evaluate. Finally, we use deep Bayesian active learning to calibrate model uncertainty to high-dimensional data (Houlsby et al., 2011; Gal and Ghahramani, 2016; Gal et al., 2017). After concluding that negative results are consistent across all experimental conditions, we investigate active learning's ineffectiveness on VQA as a data problem and identify the existence of collective outliers (Han and Kamber, 2000) as the source of the problem. Leveraging recent advances in model interpretability, we build Dataset Maps (Swayamdipta et al., 2020), which distinguish between collective outliers and useful data that improve validation set performance (see Figure 1). While global outliers deviate from the rest of the data and are often a consequence of labeling error, collective outliers cluster together; they may not individually be identifiable as outliers but collectively deviate from other examples in the dataset. For instance, VQA-2 (Goyal et al., 2017) is riddled with collections of hard questions that require external knowledge to answer (e.g., What is the symbol on the hood often associated with?) or that ask the model to read text in the images (e.g., What is the word on the wall?). Similarly, GQA (Hudson and Manning, 2019) asks underspecified questions (e.g., what is the person wearing? which can have multiple correct answers). Collective outliers are not specific to VQA, but can similarly be found in many open-ended tasks, including visual navigation (Anderson et al., 2018b) (e.g., Go to the grandfather clock requires identifying rare grandfather clocks), and open-domain question answering (Kwiatkowski et al., 2019), amongst others. Using Dataset Maps, we profile active learning methods and show that they prefer acquiring collective outliers that models are unable to learn, explaining their poor improvements in sample efficiency relative to random sampling. Building on this, we use these maps to perform ablations where we identify and remove outliers iteratively from the active learning pool, observing correlated improvements in sample efficiency. This allows us to conclude that collective outliers are, indeed, responsible for the ineffectiveness of active learning for VQA. We end with prescriptive suggestions for future work in building active learning methods robust to these types of outliers. Our work tests the utility of multiple recent active learning methods on the open-ended understanding task of VQA. We draw on the dataset analysis literature to identify collective outliers as the bottleneck hindering active learning methods in this setting. Active Learning. Active learning strategies have been successfully applied to image recognition (Joshi et al., 2009; Sener and Savarese, 2018), information extraction (Scheffer et al., 2001; Finn and Kushmerick, 2003; Jones et al., 2003; Culotta and McCallum, 2005), named entity recognition (Hachey et al., 2005; Shen et al., 2017), semantic parsing (Dong et al., 2018), and text categorization (Lewis and Gale, 1994; Hoi et al., 2006). However, these same methods struggle to outperform a random baseline when applied to the task of VQA (Lin and Parikh, 2017; Jedoui et al., 2019). To study this discrepancy, we systematically apply 8 diverse active learning methods to VQA, including methods that use model uncertainty (Abramson and Fre-und, 2004; Collins et al., 2008; Joshi et al., 2009), Bayesian uncertainty (Gal and Ghahramani, 2016; Kendall and Gal, 2017), disagreement (Houlsby et al., 2011; Gal et al., 2017), and Core-Set selection (Sener and Savarese, 2018). Visual Question Answering. Progress on VQA has been heralded as a marker for progress on general open-ended understanding tasks, resulting in several benchmarks (Agrawal et al., 2015; Malinowski et al., 2015; Ren et al., 2015a; Johnson et al., 2017; Goyal et al., 2017; Krishna et al., 2017; Suhr et al., 2019; Hudson and Manning, 2019) and models (Zhou et al., 2015; Fukui et al., 2016; Lu et al., 2016; Yang et al., 2016; Zhu et al., 2016; Wu et al., 2016; Anderson et al., 2018a; Tan and Bansal, 2019; Chen et al., 2020). To ensure that our negative results are not dataset or model-specific, we sample 4 datasets and 5 representative models, each utilizing unique visual and linguistic features and employing different inductive biases. Interpreting and Analyzing Datasets. Given the prevalence of large datasets in modern machine learning, it is critical to assess dataset properties to remove redundancies (Gururangan et al., 2018; Li and Vasconcelos, 2019) or biases (Torralba and Efros, 2011; Khosla et al., 2012; Bolukbasi et al., 2016), both of which negatively impact sample efficiency. Prior work has used training dynamics to find examples which are frequently forgotten (Krymolowski, 2002; Toneva et al., 2019) versus those that are easy to learn (Bras et al., 2020). This work suggests using two model-specific measures  confidence and prediction variance  as indicators of a training example's learnability (Chang et al., 2017; Swayamdipta et al., 2020). Dataset Maps (Swayamdipta et al., 2020), a recently introduced framework uses these two measures to profile datasets to find learnable examples. Unlike prior datasets analyzed by Dataset Maps that have a small number of global outliers as hard examples, we discover that VQA datasets contain copious amounts of collective outliers, which are difficult or even impossible for models to learn. We adopt the standard pool-based active learning setup from prior work (Lewis and Gale, 1994; Settles, 2009; Gal et al., 2017; Lin and Parikh, 2017), consisting of a model M , initial seed set of labeled examples ( x i , y i )  D seed used to initialize M , an unlabeled pool of data D pool , and an acquisition function A ( x, M ) . We run active learning over a series of acquisition iterations Pool Size # Answers VQA-Sports 5,411 [5k] 20 VQA-Food 4,082 [4k] 20 VQA-2 411,272 [400k] 3130 GQA 943,000 [900k] 1842 Table 1: We evaluate active learning on 4 VQA datasets. T where at each iteration we acquire a batch of B new examples per:  x  D pool to label per  x = arg max x D pool A ( x, M ) . Acquiring an example often refers to using an oracle or human expert to annotate a new example with a correct label. We follow prior work to simulate an oracle using existing datasets, forming D seed from a fixed percentage of the full dataset, and using the remainder as D pool (Gal et al., 2017; Lin and Parikh, 2017; Siddhant and Lipton, 2018). We re-train M after each acquisition iteration. Prior work has noted the impact of seed set size on active learning performance (Lin and Parikh, 2017; Misra et al., 2018; Jedoui et al., 2019). We run multiple active learning evaluations with varying seed set sizes (ranging from 5% to 50% of the full pool size). We keep the size of each acquisition batch B to a constant 10% of the overall pool size. Visual Question Answering (VQA) requires reasoning over two modalities: images and text. Most models use feature backbones (e.g., features from object recognition models pretrained on Ima-geNet, and pretrained word vectors for text). For image features we use grid-based features from ResNet-101 (He et al., 2016), or object-based features from Faster R-CNN (Ren et al., 2015b) fine-tuned on Visual Genome (Anderson et al., 2018a). We evaluate with a representative sample of existing VQA models, including the following: 2 LogReg is a logistic regression model that uses either ResNet-101 or Faster R-CNN image features with mean-pooled GloVe question embeddings (Pennington et al., 2014). Although these models 2 Key implementation details can be found in the appendix. In the interest of full reproducibility and further work in active learning and VQA, we release our code and results here: https://github.com/siddk/vqa-outliers . are not as performant as the subsequent models, logistic regression has been effective on VQA (Suhr et al., 2019), and is pervasive in the active learning literature (Schein and Ungar, 2007; Yang and Loog, 2018; Mussmann and Liang, 2018). LSTM-CNN is a standard model introduced with VQA-1 (Agrawal et al., 2015). We use more performant ResNet-101 features instead of the original VGGNet features as our visual backbone. BUTD (Bottom-Up Top-Down Attention) uses object-based features in tandem with attention over objects (Anderson et al., 2018a). BUTD won the 2017 VQA Challenge (Teney et al., 2018), and has been a consistent baseline for recent work in VQA. LXMERT is a large multi-modal transformer model that uses BUTD's object features and con-textualized BERT (Devlin et al., 2019) language features (Tan and Bansal, 2019). LXMERT is pretrained on a corpus of aligned image-and-textual data spanning MS COCO, Visual Genome, VQA-2, NLVR-2, and GQA (Lin et al., 2014; Krishna et al., 2017; Goyal et al., 2017; Suhr et al., 2019; Hudson and Manning, 2019), initializing a cross-modal representation space conducive to fine-tuning. 3 3.2 Acquisition Functions Several active learning methods have been developed to account for different aspects of the machine learning training pipeline: while some acquire examples with high aleotoric uncertainty (Settles, 2009) (having to do with the natural uncertainty in the data) or epistemic uncertainty (Gal et al., 2017) (having to do with the uncertainty in the modeling/learning process), others attempt to acquire examples that reflect the distribution of data in the pool (Sener and Savarese, 2018). We sample a diverse set of these methods: Random Sampling serves as our baseline passive approach for acquiring examples. 3 Results for LXMERT in Tan and Bansal (2019) are reported after pretraining on training and validation examples from the VQA datasets we use. While this is fair if the goal is optimizing for test performance, this exposure to training and validation examples leaks important information; to remedy this, we obtained a model checkpoint from the LXMERT authors trained without VQA data. This is also why our LXMERT results are lower than the numbers reported in the original paper  however, the general boost provided by cross-modal pretraining holds. Entropy acquires examples with the highest entropy in the model's output (Settles, 2009). MC-Dropout Entropy (Monte-Carlo Dropout with Entropy acquisition) acquires examples with high entropy in the model's output averaged over multiple passes through a neural network with different dropout masks (Gal and Ghahramani, 2016). This process is a consequence of a theoretical casting of dropout as approximate Bayesian inference in deep Gaussian processes. BALD (Bayesian Active Learning by Disagreement) builds upon Monte-Carlo Dropout by proposing a decision theoretic objective; it acquires examples that maximise the decrease in expected posterior entropy (Houlsby et al., 2011; Gal et al., 2017; Siddhant and Lipton, 2018)  capturing disagree-ment across different dropout masks. Core-Set Selection samples examples that capture the diversity of the data pool (Sener and Savarese, 2018; Coleman et al., 2020). It acquires examples to minimize the distance between an example in the unlabeled pool to its closest labeled example. Since Core-Set selection operates over a representation space (and not an output distribution, like prior strategies) and VQA models operate over two modalities, we employ three Core-Set variants: Core-Set (Language) and Core-Set (Vision) operate over their respective representation spaces while Core-Set (Fused) operates over the fused vision and language representation space. We evaluate the 8 active learning strategies across the 5 models described in the previous section. Figures 25 show a representative sample of active learning results across datasets. Due to space constraints, we only visualize 4 active learning strategies  Least-Confidence, BALD, CoreSet-Fused, and the Random Baseline  using 3 models (LSTM-CNN, BUTD, LXMERT). 4 Results and trends are consistent across the different acquisition functions, models and seed set sizes (see the appendix for results with other models, acquisition functions, and seed set sizes). We now go on to provide descriptions of the datasets we evaluate against, and the corresponding results. 4 For LXMERT, running Core-Set selection is prohibitive, so we omit these results; please see Appendix B for more details. One complexity of VQA is the size of the output space and the number of examples present (Agrawal et al., 2015; Goyal et al., 2017); VQA-2 has 400k training examples, and in excess of 3k possible answers (see Table 1). However, prior work in active learning focuses on smaller datasets like the 10-class MNIST dataset (Gal et al., 2017), binary classification (Siddhant and Lipton, 2018), or small-cardinality (  20 classes) text categorization (Lowell et al., 2019). To ensure our results and conclusions are not due to the size of the output space, we build two meaningful, but narrow-domain VQA datasets from subsets of VQA-2. These simplified datasets reduce the complexity of the underlying learning problem and provide a fair comparison to existing active learning literature. VQA-Sports. We generate VQA-Sports by compiling a list of 20 popular sports (e.g., soccer, football, tennis, etc.) in VQA-2, and restricting the set of questions to those with answers in this list. We picked the sports categories by ranking the GloVe vector similarity between the word sports to answers in VQA-2, and selected the 20 most commonly occurring answers. VQA-Food. We generate the VQA-Food dataset similarly, compiling a list of the 20 commonly occurring food categories by GloVe vector similarity to the word food. Results. Figure 2 presents results for VQA-Sports, with an initial seed set restricted to 10% of the total pool (500 examples). The appendix reports similar results on VQA-Food. For LSTM-CNN, Least-Confidence appears to be slightly more sample efficient, while all other strategies perform on par with or worse than random. For BUTD, all methods are on par with random; for LXMERT, they perform worse than random. Generally on VQA-Sports, active learning performance varies, but fails to outperform random acquisition. VQA-2 is the canonical dataset for evaluating VQA models (Goyal et al., 2017). In keeping with prior work (Anderson et al., 2018a; Tan and Bansal, 2019), we filter the training set to only include answers that appear at least 9 times, resulting in 3130 unique answers. Unlike traditional VQA-2 evaluation, which treats the task as a multi-label binary classification problem, we follow prior active learning work on VQA (Lin and Parikh, 2017), which formulates it as a multi-class classification problem, enabling the use of acquisition functions such as uncertainty sampling and BALD. Results. Figures 3 and 4 show results on VQA-2 with different seed set sizes  10% (40k examples) and 50% (200k examples). Active learning performs relatively better with larger seed sets but still underperforms random. Surprisingly, when initialized with 50% of the pool as the seed set, the gain in validation accuracy after acquiring the entire pool of examples (400k examples total) is only 2%. This is an indication that the lack of sample efficiency might be a result of the underlying data, a problem we explore in the next section. GQA was introduced as a means for evaluating compositional reasoning (Hudson and Manning, 2019). Unlike VQA's natural human-written questions, GQA contains synthetic questions of the form what is inside the bottle the glasses are to Underspecification:What is on the shelf? Multi-hop reasoning: What is the vehicle that is driving down the road the box is on the side of? GQAVQA2 External knowledge: What does the symbol on the blanket mean? OCR:What is the first word on the black car? Figure 7: Example groups of collective outliers in the VQA-2 and GQA datasets. the right of?. Results. Figure 5 shows results on GQA using a seed set of 10% of the full pool (90k examples). Despite its notable differences in question structure to VQA-2, active learning still performs on par with or slightly worse than random. The previous section shows that active learning fails to improve over random acquisition on VQA across models and datasets. A simple question remains  why ? One hypothesis is that sample ineffi-ciency stems from the data itself: there is only a 2% gain in validation accuracy when training on half versus the whole dataset. Working from this, we characterize the underlying datasets using Dataset Maps (Swayamdipta et al., 2020) and discover that active learning methods prefer sampling hard-to-learn examples, leading to poor performance. Mapping VQA Datasets. A Dataset Map (Swayamdipta et al., 2020) is a model-specific graph for profiling the learnability of individual training examples. Dataset Maps present holistic pictures of classification datasets relative to the training dynamics of a given model; as a model trains for multiple epochs and sees the same examples repeatedly, the mapping process logs statistics about the confidence assigned to individual predictions. Maps then visualize these statistics against two axes: the y-axis plots the average model confidence assigned to the correct answer over training epochs, while the x-axis plots the spread, or variability, of these values. This introduces a 2D representation of a dataset (viewed through its relationship with individual model) where examples are placed on the map by coarse statistics describing their learnability. We show the Dataset Map for BUTD trained on VQA-2 in Figure 1. For our work, we build this map post-hoc, training on the entire pool as a means for analyzing what active learning is doing  treating it as a diagnostic tool for identifying the root cause why active learning seems to fail for VQA. In an ideal setting, the majority of examples in the training set should lie in the upper half of the graph  i.e., the mean confidence assigned to the correct answer should be relatively high. Examples towards the upper-left side represent the easy-to-learn examples, as the variability in the confidence assigned by the model over time is fairly low. A curious feature of VQA-2 and other VQA datasets is the presence of the 25-30% of examples in the bottom-left of the map (shown in red in Figure 1)  examples that have low confidence and variability. In other words, models are unable to learn a large proportion of training examples. While prior work attributes examples in this quadrant to labeling errors (Swayamdipta et al., 2020), labeling errors in VQA are sparse, and cannot account for the density of such examples in these maps. Interpreting Acquisitions. We profile the acquisitions made by each active learning method, con-textualizing the acquired examples via their placement on the associated Dataset Map. We segregate training examples into four buckets using the map's y-axis: easy (  0 . 75 ), medium (  0 . 50 ), hard (  0 . 25 ), and impossible (  0 . 00 ). Ideally, active learning should be robust to hard-to-learn examples, focusing instead on learnable, high uncertainty examples towards the upper-right portion of the Dataset Map. Instead, we find that active learning methods acquire a large proportion of impossible examples early on and concentrate on the easier examples only after the impossible examples dwindle (see Figure 6). In contrast, the random baseline acquires examples proportional to each bucket's density in the underlying map; acquiring easier examples earlier and performing on par with or better than all others. This leaves two questions: 1) can we characterize these hard examples, and 2) are these examples responsible for the ineffectiveness of active learning on VQA? We first identify hard-to-learn examples as collective outliers and explain why active learning methods prefer to acquire them. Next, we perform ablation experiments, removing these outliers from the active learning pool iteratively, and demonstrate a corresponding boost in sample efficiency relative to random acquisition. Hard Examples are Collective Outliers. Collective outliers are groups of examples that deviate from the rest of the examples but cluster together (Han and Kamber, 2000)  they often present as fundamental subproblems of a broader task. For instance (Figure 7), in VQA-2, we identify clusters of hard-to-learn examples that require optical character recognition (OCR) for reasoning about text (e.g., What is the first word on the black car?); another cluster requires external knowledge to answer (What is the symbol on the hood often associated with?). In GQA, we identify different clusters of collective outliers; one cluster stems from innate underspecification (e.g., what is on the shelf? with multiple objects present on the shelf); another cluster requires multiple reasoning hops difficult for current models (e.g., What is the vehicle that is driving down the road the box is on the side of?). We sample 100 random hard-to-learn examples from both VQA-2 and GQA and find that 100% of the examples belong to one of the two aforementioned collectives. Since hard-to-learn examples constitute 2530% of the data pool, active learning methods cannot avoid them. Uncertainty-based methods (e.g., Least-Confidence, Entropy, Monte-Carlo Dropout) identify them as valid acquisition targets because models lack the capacity to correctly answer these examples, assigning low confidence and high uncertainty. Disagreement-based methods (e.g., BALD) are similar; model confidence is generally low but high variance (lower middle/lower right of the Dataset Maps). Finally, diversity methods (e.g., Core-Set selection) identify these examples as different enough from the existing pool to warrant acquisition, but fail to learn meaningful representations, fueling a vicious cycle wherein they continue to pick these examples. Ablating Outliers. To verify that collective outliers are responsible for the degradation of active learning performance, we re-run our experiments using active learning pools with varying numbers of outliers removed. To remove these outliers, we sort and remove all examples in the data pool using the product of their model confidence and prediction variability (x and y-axis values of the Dataset Maps). We systematically remove examples with a low product value and observe how active learning performance changes (see Figure 8). We observe a 23x improvement in sample efficiency when removing 50% of the entire data pool, consisting mainly of collective outliers (Figure 8c). This improvement decreases if we only remove 25% of the full pool (Figure 8b), and further degrades if we remove only 10% (Figure 8a). This ablation demonstrates that active learning methods are more sample efficient than the random baseline when collective outliers are absent from the unlabelled pool. This paper asks a simple question  why does the modern neural active learning toolkit fail when applied to complex, open ended tasks? While we focus on VQA, collective outliers are abundant in tasks such as natural language inference (Bow-man et al., 2015; Williams et al., 2018) and open-domain question answering (Kwiatkowski et al., 2019), amongst others. More insidious is their na-ture; collective outliers can take multiple forms, requiring external domain knowledge or common-sense reasoning, containing underspecification, or requiring capabilities beyond the scope of a given model (e.g., requiring OCR ability). While we perform ablations in this work removing collective outliers, demonstrating that active learning fails as collective outliers take up larger portions of the dataset, this is only an analytical tool; these outliers are, and will continue to be, pervasive in open-ended datasets  and as such, we will need to develop better tools for learning (and performing active learning) in their presence. Selective Classification. One potential direction for future work is to develop systems that abstain when they encounter collective outliers. Historical artificial intelligence systems, such as SHRDLU (Winograd, 1972) and QUALM (Lehnert, 1977), were designed to flag input sequences that they were not designed to parse. Ideas from those methods can and should be resurrected using modern techniques; for example, recent work suggests that a simple classifier can be trained to identify out-of-domain data inputs, provided a seed out-of-domain dataset (Kamath et al., 2020). Active learning methods can be augmented with a similar classifier, which re-calibrates active learning uncertainty scores with this classifier's predictions. Other work learns to identify novel utterances by learning to intelligently set thresholds in representation space (Karamcheti et al., 2020), a powerful idea especially if combined with other representation-centric active learning methods like Core-Set Sampling (Sener and Savarese, 2018). Another direction for future work to explore is to leverage Dataset Maps to perform more global, holistic reasoning over datasets, to intelligently identify promising examples  in a sense, baking part of the analysis done in this work directly into the active learning algorithms. A possible instantiation of this idea would be in training a discriminator to differentiate between learnable examples (upper half of each Dataset Map) from the unlearnable, collective outliers with low confidence and low variability. Between each active learning acquisition iteration, one can generate an updated Dataset Map, thereby reflecting what models are learning as they obtain new labeled examples. Machine learning systems deployed in real-world settings will inevitably encounter open-world datasets, ones that contain a mixture of learnable and unlearnable inputs. Our work provides a framework to study when models encounter such inputs. Overall, we hope that our experiments serve as a catalyst for future work on evaluating active learning methods with inputs drawn from open-world datasets. All code for data preprocessing, model implementation, and active learning algorithms is made available at https://github.com/siddk/vqa-outliers . Additionally, this repository also contains the full set of results and dataset maps as well. The authors are fully committed to maintaining this repository, in terms of both functionality and ease of use, and will actively monitor both email and Github Issues should there be problems. We thank Kaylee Burns, Eric Mitchell, Stephen Mussman, Dorsa Sadigh, and our anonymous ACL reviewers for their useful feedback on earlier versions of this paper. We are also grateful to Hao Tan for providing us with the LXMERT checkpoint trained without access to VQA datasets, as well as for general LXMERT fine-tuning pointers. Siddharth Karamcheti is graciously supported by the Open Philanthropy Project AI Fellowship. Christopher D. Manning is a CIFAR Fellow.
abstain objective method result result method abstain abstain other abstain abstain result abstain result method method method abstain result abstain abstain abstain abstain result result abstain method abstain method other other other method other other abstain other other other method other method method method method abstain method method abstain method method abstain abstain method method abstain other abstain abstain method abstain abstain abstain abstain other method method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method other abstain abstain other other other other