Event forecasting is a challenging, yet important task, as humans seek to constantly plan for the future. Existing automated forecasting studies rely mostly on structured data , such as time-series or event-based knowledge graphs, to help predict future events. In this work, we aim to formulate a task, construct a dataset, and provide benchmarks for developing methods for event forecasting with large volumes of unstructured text data. To simulate the forecasting scenario on temporal news documents, we formulate the problem as a restricted-domain, multiple-choice, question-answering (QA) task. Unlike existing QA tasks, our task limits accessible information, and thus a model has to make a forecasting judgement. To showcase the usefulness of this task formulation, we introduce FORECASTQA, a question-answering dataset consisting of 10,392 event forecasting questions, which have been collected and verified via crowdsourcing efforts. We present our experiments on FORECASTQA using BERT-based models and find that our best model achieves 61.0% accuracy on the dataset, which still lags behind human performance by about 19%. We hope FORECASTQA will support future research efforts in bridging this gap. 1 1 Introduction Forecasting globally significant events, such as outcomes of policy decisions, civil unrest, or the economic ramifications of global pandemics, is a consequential but arduous problem. In recent years there have been significant advances in applying machine learning ( e.g. , time-series prediction methods) to generate forecasts for various types of events including conflict zones (Schutte, 2017), duration of insurgency (Pilster and Bohmelt, 2014), civil unrest (Ramakrishnan et al., 2014a) and terrorist events (Raghavan et al., 2013). Current automated forecasting methods perform well on problems for which there are sufficient structured data ( e.g. , knowledge graphs), but are not well suited for events for which such data may not exist. Humans, though, can often accurately forecast outcomes by leveraging their judgement, domain knowledge, and prior experience (Tetlock and Gardner, 2016), along with the vast amounts of unstructured text data available to us ( e.g. , news ar-ticles). We are able to identify and retrieve salient facts from the near-endless pool of unstructured information, synthesize those facts into coherent beliefs, and generate probabilistic forecasts. Unfortunately, the process does not scale well in terms of the amount of information that must be processed and the number of events one has to forecast. Here we address the above problem by formalizing a forecasting task, creating a dataset, and providing benchmarks to develop methods for the task. Specifically, we formulate the forecasting problem as a multiple-choice Question Answering (QA) task, where the input is a news corpus, questions, choices and timestamps associated with each question, and the output is one of the given choices per question. Our approach is rooted in the observation that both forecasting and QA follow a similar process: digesting massive amounts of textual data, identifying supporting pieces of evidence from text, and chaining different pieces to generate answers/forecasts. Forecast Question Answering (FORECASTQA) introduces a novel timestamp constraint per question that prohibits the model from accessing new articles published after the timestamp . By doing so, FORECASTQA simulates a forecasting scenario; each question's timestamp is chosen to ensure that the question is about the outcome of a future event. To illustrate this, consider the question,  Will primary schools in Europe admit non-vaccinated children around September 2019?  in Figure 1, and the fact that models only have access to articles before 2019-09-01. With the addition of this timestamp constraint, our query becomes a question about a future event in September, 2019 based on articles from the past; the model is now being tested for its forecasting ability 2 . To answer the question, the model must find pertinent events from past information, resolve the temporal and causal relations between them, and finally make a forecasting judgement based on its interpretation of past information to answer the question. Our task differs from that of other works that require an understanding of temporal relationships (Ning et al., 2020) and temporal commonsense reasoning (Zhou et al., 2019), as our task forces a model to make a forecasting judgement. In support of the proposed FORECASTQA formulation, we construct a dataset of 10,392 yes-no and multiple-choice questions. This data is collected via crowdsourcing based on news articles, where workers are shown articles and asked to come up with yes-no and multiple-choice questions. We also crowdsourced appropriate timestamps for each question. Finally, we design a method based on pre-trained language models to deal with retrieved articles for our task. In our experiments, the methods using retrieved articles slightly outper-2 The ability to predict the outcome of future events based on unstructured text describing past events, without access to an extracted sequence of historical event triples, nor provided a fixed set of possible relations between events; as is the case with human forecasters. Q: Who will drop Japan as a trading partner in August 2019? Choices: South Korea ( answer ), South Africa, Syria, Portugal. Article: Why Japan and South Korea just can't get along. (1/1/19) Apart from the fact of being one another's closest neighbours, the people of South Korea and Japan have a remarkable amount in common. Economically, they are among one another's biggest trading partners. And yet, time and again, relations between Seoul and Tokyo are marked, not by mutual support and co-operation but by anger, reproach and exasperation. Reasoning Process: Seoul is in South Korea, Tokyo is in Japan ( commonsense world knowledge ). Seoul and Tokyo are big trading partners ( language understanding lexical variations ). The relations between Seoul and Tokyo are marked by anger, reproach and exasperation and these relations might cause trading relations to cease ( forecasting skills causal relation we can infer the answer from this part ). form closed-book models, suggesting that our task is still challenging in that finding relevant information for forecasting and making a judgement are not straightforward. Our best attempt achieves 61.0% accuracy on our dataset, a significant performance gap from human performance by 19.3%. Event Forecasting. There are several types of approaches exist to do event forecasting. One approach could learn from highly structured event-coded data such as ICEWS (Boschee et al., 2015) and GDELT (Leetaru and Schrodt, 2013). When these datasets are used for forecasting, they are often represented as a time series (Morstatter et al., 2019; Ramakrishnan et al., 2014b), in which each data point is associated with a timestamp. Another approach is script-learning, in which a model is provided with a chain of events and a subsequent event and is asked to predict the relation between the chain and the future event (Hu et al., 2017; Li et al., 2018; Lv et al., 2019). They require to convert text data into event triples and translate the questions and answer choices into their format, which limits the expressiveness of natural text. However, unlike these datasets and approaches, FORECASTQA does not provide any structured data to a model. The model must learn how to extract, keep track of, and link pertinent events from unstructured text to solve forecasting questions. QA and Temporal Reasoning on Text. There are several approaches for QA using unstructured text. Extractive QA approaches rely on finding answer spans from the text that best answer a question (Rajpurkar et al., 2016, 2018; Yang et al., 2018; Kwiatkowski et al., 2019; Huang et al., 2019). Multiple-Choice QA requires a model to pick the best answer from a set (Talmor et al., 2019; Sap et al., 2019; Zhou et al., 2019), and generative QA prompts the machine to produce its own answer (Khashabi et al., 2020). Our dataset is a type of multiple-choice QA, but it differentiates itself from other QA datasets (all formats) in that the required answer does not exist in the provided text, nor is sufficient evidence provided to be able to answer a question with 100% certainty; a forecast is required. We could convert our questions into alternative query formats such as a text-to-text format, but instead we stick to multiple-choice questions as humans often weigh the benefits of multiple choices when making a forecasting judgement. QA datasets often exist to test certain types of reasoning. One pertinent example of a reasoning type that QA tasks test is the understanding of temporal and casual relations (Jia et al., 2018a,b; Sun et al., 2018; Ning et al., 2020). However, FORECASTQA requires more than just extraction and understanding of relations; a model must be able to extract and understand the relations present in the text with the goal of making a forecasting judgement about an event whose outcome is not found in the text. Another type of reasoning tested in QA tasks is commonsense reasoning (Talmor et al., 2019) and even temporal commonsense reasoning (Zhou et al., 2019). While questions in FORECASTQA often require commonsense to correctly answer, not all do; event outcomes do not always follow common sense. Furthermore, our questions test forecasting abilities, which often includes various types of reasoning in addition to commonsense. FORECASTQA is a question answering task whose goal is to test a machine's forecasting ability . We consider forecasting as the process of anticipating the outcome of future events based on past and present data (Tetlock and Gardner, 2016). We focus on forecasting outcomes of news-based events coming from topics such as politics, sports, economics, etc. Training a machine to make forecasting decisions is inherently difficult, as the ground-truth label of event outcome ( e.g. , whether an event will occur)  so often required for model training  is only obtainable in the future. To make progress in our goal, we devise a way to simulate the forecasting scenario by introducing a novel time constraint , allowing us to validate the machine predic-Will the Will there What will Whatis What kind Whattype What does Who will Who is How many How much Howwill How old Where will Whichcountry W h i c h c o un t r y ' s W h i c h p a r t y Which company Whywill Whenwill Isthe D o e s t h e A r e t h e Will there be electricity in Canada despite hurricane Dorian in September 2019? tions by obtaining desired ground-truth labels. There is also the difficulty of ensuring the quality of question generation via crowdsourcing (neces-sary when building a dataset of scale), due to possible human errors in question formation (Tetlock et al., 2017). We have taken steps to ensure our questions cannot be answered with certainty using past data given the time constraint or commonsense knowledge, but the questions are tractable to answer with an educated guess (see Sec. 4.1). 3 Task Definition. Formally, the input of the FORECASTQA task is a forecasting question Q with a corresponding ending timestamp t Q the last possible date where Q remains a forecasting question. In addition, we have a set of possible choices, C , and a corpus of news articles, A ; the output is a choice C  C . Our task has a novel constraint that any retrieved article A  A must satisfy t A < t Q . In other words, models have access only to articles that are published before t Q . We have ensured that the information required to solve the question deterministically comes out in an article, gold article , published after t Q , i.e., t gold article  t Q . Another way to think of our setup is that we are asking Q on the day before t Q , knowing that the information required to solve Q is not available yet. This for-3 This is in contrast to open-domain QA (machine reading comprehension) (Kwiatkowski et al., 2019) where answers can always be found in some given passages. input of FORECASTQA creation is a news article corpus and the output is yes-no/multiple-choice questions. from existing QA tasks. Challenges in FORECASTQA. Due to the constrained open-domain setting and forecasting properties, testing a model's forecasting ability encompasses the following challenges: information retrieval (IR) on limited sources, understanding of temporal and causal relations between events, and finally a forecasting judgement. Our time constraint limits the accessible articles and also creates more challenges than in standard open-domain QA; effective IR methods are necessary to anticipate what knowledge will be useful for predictions from past information sources. Once useful articles have been retrieved, models should understand these articles and reason over pertinent facts from them. Finally, these models use the gleaned knowledge to infer the outcome of a future event. Unlike in other reading comprehension tasks, models cannot rely on the existence of an answer within the text, but must make an educated guess as to what will happen in the future. While our task does encompass reasoning abilities tested in other datasets, no other tasks investigate these reasoning abilities in the context of predicting future events. More analysis on reasoning types can be found in Sec. 4.2. In this section, we describe how we construct our FORECASTQA dataset and analyze it. The data collection is broken down into three sections: (1) gathering a news corpus, (2) generating question-answer-timestamp triples with distractor choices, and (3) verifying the triples' quality. The data generation process is summarized in Fig. 3. News Corpus Collection. We started by gathering English news articles from LexisNexis 4 . We then curated a list of 21 trustful news sources and filtered articles based on their publishers; we also filtered out non-English articles. Finally, we selected the five-year period of 2015-2019 and filtered out articles outside this period, leaving us with 509,776 articles. This corpus is also used for retrieval in our task setting ( i.e. , constrained open-domain). Q-Answer-timestamp Triple Creation. 5 Once we assembled the news corpus, we built (ques-tion, answer, timestamp) triples to accompany the new corpus as inputs for our task. To generate the needed triples we looked to crowdsourcing via Amazon Mechanical Turk. Our generation task consists of the following steps: (1) we selected a random news article from 2019 from the collected news corpus (these news articles are gold articles and will be hidden for experiments); (2) workers created questions, which if posed before the respective article's publication date would be seen as a forecasting question; (3) they indicated the answer, along with supporting evidence that the question consisted of (to ensure the correctness of the true answer); (4) they were asked to make multiple-choice distractors with their own knowledge and/or access to search engines; and (5) we ensured that a temporal phrase is present in the questions, for example:  After May of 2020... ,  ... in June of 2021?  to provide a temporal context (constraint) for each question, yielding more precise and well-defined forecasting questions. Completion of this task results in the desired triple of: a forecasting question, an answer to the question (with distractor choices), and a timestamp as our temporal constraint. The timestamp is set as the first day of the month in which the gold article was published. To diversify questions in the dataset, we created two kinds of questions: binary yes-no questions and multiple-choice questions with four choices . Multiple-choice questions start with one of the six Ws ( i.e. , who, what, when, where, why, and how) and are more challenging as they require determining the correctness of each choice. Question Quality Verification. We performed a separate crowdsourcing data verification to test and enforce the following criteria: (1) is answering the question a tractable problem given (relevant) 4 https://risk.lexisnexis.com 5 Due to the limited space, for more details of our triple creation guidelines for human annotators, verification steps, and screenshots of our data col-lection/verification AMT interfaces, please refer to Sec. A of the appendix. past articles?, and (2) is the question deterministically answerable given any article adhering to the question's temporal constraint? If a question is too difficult, i.e. , an educated guess to the answer (when given relevant, constraint-adhering articles) is not possible, then we filter the question out. On the other hand, if the questions are answerable with certainty using past articles, or commonsense/world knowledge, then they are not considered to be forecasting questions. The desired response (majority vote from 3 annotators) is a  yes  for criterion (1) and  no  for (2), as that would show that the tuple of question and time constraint simulates the desired forecasting scenario. With the above method, we filtered out 31% of the questions collected in the triple creation step and were left with 5,704 yes-no questions and 4,513 multi-choice questions. More details about the verification step are included in Sec. A of the appendix. To better understand the properties of the questions in FORECASTQA, we examine: 1) a few data statistics 2) types of questions asked, and 3) the types of reasoning required to answer our questions. Summary Statistics. FORECASTQA dataset is composed of 10,392 questions, divided into a 80/10/10 split of train, dev, and test data. Our 10k questions are roughly evenly split between multiple-choice and yes-no binary questions (Ta-ble 2). Over 17K distinct words were used to construct our questions and we have 218 unique time constraints associated with them; time constraints range from 2019-01-11 to 2019-11-12. We include additional statistics in Sec. Types of Questions. To understand the types of questions in FORECASTQA, we examined the popular beginnings of sentences and created a tree-map plot (see Fig. 2). As shown, nearly half the questions start with the word will (44%), a result of over half of the questions being yes-no questions. Reasoning Types. To examine types of reasoning required to answer our questions we sampled 100 questions and manually annotated them with reasoning types. Due to the forecasting nature of our dataset, we are particularly interested in questions containing the forecasting ability and thus spend more time looking into these questions. Our condensed results can be found in Figure 4, and more results from our cataloguing effort can be found in Sec. C of the appendix. Note that most questions contain more than one reasoning type. To evaluate the forecasting capabilities of recent multi-choice / binary QA model architectures on FORECASTQA, we provide a comprehensive benchmarking analysis in this work. We run the experiments in two settings: (1) closed-book and (2) constrained open-domain setup. In the closed-book scenario only Q (question) and C (answer choices) are provided to the model ( Q, C ) , while A (news articles) is provided for setting (2), ( Q, C, A ) 6 . We run these settings to understand the difficulty of both the closed-book and open-domain challenges presented by the questions in FORECASTQA. 6 t Q is always applied to A , we left it out of the notation for simplicity. For both settings, we explore several baseline models, but all follows a general architecture of a text encoder f and an optional context aggregation module g to aggregate information from a set of retrieved articles. Fig. 5 shows the architectures used. We model both yes-no and multiple-choice questions as a binary classification task; a model's prediction is the class with the largest probability. Below we introduce the details of our baselines. Text Encoder. We use pre-trained language model, BERT (Devlin et al., 2019), as a text encoder ( f from above) 7 . f is designed to deal with ( Q, C ) and ( Q, C, A ) inputs, where A is a set of time-stamped articles that are retrieved from A to answer Q . Each input of f is transformed into [ [CLS] Q [SEP] C [SEP] A i ] (for each A i  A , C  C ), or [ [CLS] Q [SEP] C ] (for each C  C ) if articles are not supplied. The [CLS] token is the same as the one commonly used for fine-tuning PTLMs for a classification task, and [SEP] is the special separator token. The embedding of [CLS] is then used for predictions with an MLP layer (the leftmost model architecture in Fig. 5), or as input into a context aggregation module (the middle architecture in Fig. 5) subsequently introduced. Context Aggregation (AGG). Two architectures are used when aggregating information from multiple, time-stamped articles A retrieved for a question. (1) Temporal Aggregation: This aggregator utilizes temporal ordering of the retrieved articles. Articles are sorted by their timestamps and their [CLS] token representation from f are aggregated by a Gated Recurrent Unit (GRU) (Cho et al., 2014) with a MLP head to make final predictions. (2) Set Aggregation: Alternatively, we ignore the temporal ordering of articles and use a maxpooling operation 7 We did not include more recent pre-trained language models ( e.g. , RoBERTa (Liu et al., 2019b), ALBERT (Lan et al., 2020), T5 (Raffel et al., 2020)) or pre-trained QA models like UnifiedQA (Khashabi et al., 2020), as these models are trained using text data published after the earliest timestamp in our dataset (2019-01-01), meaning information leakage could occur (and violates the forecasting setup). We tested more LMs in Sec. E.5 of appendix. on the [CLS] token representations of each article. This pooled representation is passed to an MLP layer to make a prediction. Comparison between these aggregations helps understand the effect of modeling temporal order of evidence. These two aggregation modules are denoted by AGG (GRU) and AGG (Maxpool), respectively. Multi-document Summarization (MDS). Rather than conducting context aggregation of the retrieved articles, we consider an MMR summarizer (Carbonell and Goldstein, 1998) which performs extractive, multi-document summarization of text to generate a summary A summ (rightmost architecture in Fig. 5). The summary article A summ is treated as if it is an A i  A and fed into a text encoder along with Q and C which then produce the [CLS] embedding for making a prediction. We name this method MDS. Integrated Approach. To take the best of both worlds in ( Q, C ) and ( Q, C, A ) settings, we integrate two architectures (the leftmost and middle ones in Fig. 5). We concatenate the last two hidden representations of each architecture before passing the concatenated representation through a shared MLP layer. We use BERTLARGE as f in both architectures, AGG (GRU) for g and call this model BERT LARGE ++ (integrated) in Table 3. Other Baselines. We also consider other baselines: ESIM (Chen et al., 2017b), BI DAF++ (Clark and Gardner, 2018), prepending extracted open event triples (Liu et al., 2019a) to BERT input, and a script learning approach, SAM-Net (Lv et al., 2019). We modify the approaches to fit into our setup. Detailed descriptions of each baseline method are included in Sec. E.3 of appendix. We adopt two types of settings: the closed-book setting ( Q, C ) and the constrained open-domain setting ( Q, C, A ) . In the constrained open-domain setting, we use BM25 (Robertson et al., 1995; Qi et al., 2019) as our IR method 8 to obtain A , 10 retrieved articles. We also explore other IR methods in the later section. Note that we retrieve articles that do not violate the time constraints. We feed the question Q as a query and limit our access to articles in A by t Q . Additionally, we validate the 8 Details of IR methods are described in appendix Sec. E.2. articles instead of retrieved articles (Sec. 6.3). Evaluation Metrics. Because forecasting is uncertain, a system's prediction probabilities indicate its confidence answering the question. In addition to accuracy, we consider Brier score (Brier, 1950), which measures the mean squared error of probabilities assigned to sets of answer choices (outcomes). Formally, Brier = 1 N (cid:80) Ni =1 (cid:80) Cc =1 ( p ic  y ic ) 2 , where p ic is the probability of prediction; y ic is a label indicator for class c of the instance (1 or 0), N is the number of prediction instances, and C is the number of classes (2 or 4). The highest Brier score is 0 (probability 1 for the correct class, probability 0 else), while the worst possible Brier score is 2 (probability 1 for the wrong class, probability 0 else). A confident model gets low Brier scores. To benchmark human performance, seven annotators (computer science graduate students) who were not involved in question generation were asked to answer 150 randomly sampled questions from the test set. We consider two scenarios: 1) annotators are provided with retrieved articles, A ; and 2) annotators can access any article published before the timestamp via Google Search. Moreover, as annotators live in the future with respect to the timestamp of a question, they might already know the actual answer. To avoid the over-estimation Methods GRU Maxpool MDS BERTBASE , TF-IDF 53.2 53.9 51.6 BERTBASE , DPR 53.7 54.6 54.3 BERTBASE , BM25 55.4 54.2 52.0 BERTLARGE , TF-IDF 56.5 55.4 55.0 BERTLARGE , DPR 56.1 59.4 54.6 BERTLARGE , BM25 59.1 58.6 54.7 Table 4: Accuracy with different retrievers: BM25, TF-IDF, and dense passage retrieval (DPR). of accuracy, we asked the annotators to not use their future knowledge. If they felt this is not possible, we asked them to skip the question. On average, 28.3% of questions are skipped. Given this setup, humans achieve 71.2% and 79.4% accuracy respectively, for the two scenarios when taking a majority vote for each question; we also observed good inter-annotator agreement. The two scenarios are referred as  (  )  and  (  )  in Table 3. 6.3 Results and Performance Analysis Results on the Constrained Open-domain Setting. Table 3 shows the results of baseline methods for comparison. We compare pre-trained language models with different context aggregators and other baselines. The integrated model, BERTLARGE ++ shows the best performance in terms of accuracy, while BERTLARGE (closed-book) shows the best Brier score. Unlike the accuracy metric, the Brier score penalizes overand underconfident forecasts (Mellers et al., 2014)  thus the best model under each metric can be different. The marginal differences in performance between the two settings suggest that access to information (text evidence) alone does not solve the forecasting problem. We hypothesize an inability to encode salient relations for forecasting purposes prevents the additional information from proving useful. Among the aggregators in BERTBASE , the GRU aggregator outperforms other aggregators and summarizers. This suggests that utilizing articles' temporal order helps the reasoning. Overall, baselines fall behind human performance by over 10% points given the same retrieved articles. Study of Different IR Methods. We further test several retrieval methods: BM25 (Robertson et al., 1995; Qi et al., 2019), TF-IDF (Chen et al., 2017a), and a pre-trained dense passage retriever (DPR) (Karpukhin et al., 2020). As in Table 4, BERTLARGE with DPR retriever and the Maxpool aggregator shows the best performance than other combinations. However, DPR does not achieve the best accuracy for all methods. This implies that 1) Methods / Metrics GRU Maxpool ACC (  ) Brier (  ) ACC (  ) Brier (  ) w/o timestamps 55.4 0.583 54.2 0.568 Pre-pend timestamps 54.2 0.634 54.8 0.599 Binary timestamp encoding 51.1 0.623 55.6 0.624 Char-RNN timestamp encoding 54.0 0.640 54.3 0.620 Table 5: Study on modeling article timestamps (publica-tion dates) in the constrained open-domain setting. Ablation on Timestamp Modeling. We conduct an ablation study on modeling time information (publication date) of the retrieved articles, as seen in Table 5. We test: a) pre-pending date string as BERT input, b) using binary encodings of dates 9 and concatenate with article encoding before aggregation, and c) using char-RNN (Goyal and Durrett, 2019) for encoding date string before aggregation 10 . We find that using binary encodings of dates improves the accuracy for the maxpool aggregator. However, the GRU aggregator's accuracy decreases when given date information. We conjecture that our modeling for the time information of each article is not strong enough to help forecasting. We leave more sophisticated modeling for future work. Answerability of Questions. To validate that the questions in FORECASTQA are indeed answerable, we convert our setup into a machine reading comprehension (MRC) task  find an answer given an assumed appropriate context. We provide the model with a gold article or the evidence sentence (Sec. 4.1). Since pre-trained models have achieved high performance on MRC tasks (Rajpurkar et al., 2016), we expect adequate performance when provided the correct context. As seen in Table 6, we observe that in closed-book setting, BERT is able to beat out a random baseline, but it still does not 9 https://temporenc.org 10 Details are described in appendix Sec. (a) Varying amounts of data. (b) Different question types. Figure 6: (a) Test accuracy of BERTBASE trained with varying amounts of training data, with human performance (79.1%) shown in orange, and (b) development accuracy breakdown by different types of multichoice questions. perform well; implying our questions are not trivial for BERT, and context is required to answer them correctly. When given the gold article, BERT achieves 76.9% (+22%) and it even performs better (84.4%) given the evidence sentence. This all implies that given the right information, our forecasting questions can be answered correctly. Study of Data Efficiency. To examine how models might perform with less/more training data, we evaluate BERTBASE (closed-book) on the test set, by training it with varying amounts of labeled data. Fig. 6a shows the the resulting learning curve. We observe the accuracy of the model is expected to reach 70%, assuming 100k examples  which is still 9% point lower than human performance. Results on Different Question Types. We test BERTBASE (closed-book) on different question types of multi-choice questions from our development set (Fig. 6b). We find that the accuracy of the model varies across different question types:  how  questions are the most difficult to predict while higher accuracy is achieved on  why  questions. Also for yes-no questions, the method achieves 69.5% on  yes  questions and 62.9%  no  questions, indicating that there is no significant bias towards certain type of binary questions. Error Analysis. We observe 4 main categories of errors produced by the methods in our analysis: (1) retrieving irrelevant articles, (2) incorrect reasoning on relevant evidence, (3) lacking (temporal) common sense, and (4) lacking numerical knowledge. Please refer to Sec. E.7 of appendix for examples and in-depth discussions of these errors. Forecasting is a difficult task that requires every possible advantage to do well. It would be wise to harness this pool of unstructured data for training automatic event forecasting agents. To utilize this form of data for forecasting, we proposed a question-answering task that requires forecasting skills to solve FORECASTQA, and provided the accompanying dataset. Various baseline methods did not perform well, but this is not surprising given the inherent difficulty of forecasting. Our benchmark dataset can benefit future research beyond natural language understanding and hope forecasting performance will be significantly improved.
abstain abstain objective method method result result abstain abstain abstain abstain abstain method abstain objective method method abstain abstain abstain abstain abstain abstain objective abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result other other other other other other other other other other other other method method other other other other other abstain abstain method method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain objective method result result abstain method abstain abstain other abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain method abstain method method method method method abstain abstain method method method method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain result abstain abstain abstain abstain objective abstain result