Charts are commonly used for exploring data and communicating insights. Generating natural language summaries from charts can be very helpful for people in inferring key insights that would otherwise require a lot of cognitive and perceptual efforts. We present Chart-to-text, a large-scale benchmark with two datasets and a total of 44,096 charts covering a wide range of topics and chart types. We explain the dataset construction process and analyze the datasets. We also introduce a number of state-of-the-art neural models as baselines that utilize image captioning and data-to-text generation techniques to tackle two problem variations: one assumes the underlying data table of the chart is available while the other needs to extract data from chart images. Our analysis with automatic and human evaluation shows that while our best models usually generate fluent summaries and yield reasonable BLEU scores, they also suffer from hallucinations and factual errors as well as difficulties in correctly explaining complex patterns and trends in charts. Data visualizations such as bar charts, line charts, and pie charts are very popular for presenting quantitative data. Often people use such charts to get important insights from data and make informed decisions. However, it is well-known that inferring key insights from the charts can be quite challenging and time-consuming, as it may require a lot of cognitive and perceptual efforts (Prez-Echeverra et al., 2018; Whitaker and Jacobbe, 2017). Automatic chart summarization is a task where the goal is to explain a chart and summarize key takeaways from it in natural language. Chart summarization has several key benefits and potential applications. First, chart summaries can help people identify key insights from charts that they might  Equal contribution. Listing order is based on the alphabetical ordering of author surnames. Gold: In 2019, Singapore imported approximately 236.8 billion Singapore dollars worth of machinery and equipment, making it the country's largest import commodity by value. This was followed by the import of mineral fuels and lubricants, valued at 102.7 billion Singapore dollars. TAB-T5: Machinery and equipment was the most valuable commodity for Singapore in 2019, with an import value of 236.8 billion Singapore dollars. Mineral fuels and lubricants were the second most valuable commodity for Singapore, with an import value of 102.7 billion Singapore dollars. have missed otherwise. In a study on a chart corpus, Carberry et al. (2006) found that chart authors often failed to convey key insights from charts in their corresponding textual captions. Thus, automatic summarization could help authors write effective reports and articles on data facts by suggesting explanatory texts. Similarly, readers could benefit from such summaries, as studies have found that captions help readers find important points by explaining visually prominent features in charts (Kim et al., 2021). Chart summarization offers another important benefit of making charts more accessible to people who are visually impaired since they can use screen readers to understand what is being presented in the chart (Ferres et al., 2013). Finally, the generated summaries can be leveraged for indexing documents containing charts to improve information retrieval algorithms (Li et al., 2013). Despite its numerous benefits and applications, the chart summarization problem has not received much attention in the NLP community. Early approaches relied on template-based text generation methods that combine statistical techniques and planning-based architecture (Reiter, 2007) to generate captions from bar and line charts (Fasciano and Lapalme, 1996; Mittal et al., 1998; Green et al., 4005 2004; Demir et al., 2012). Recently, researchers considered data-driven neural models for describing tabular data (Mei et al., 2016; Gong et al., 2019). However, compared to tables, charts serve a different communication goal, and so is the chart-to-text problem. Unlike tables which simply list raw data, charts create visual representation of data that can draw a reader's attention to various prominent features such as trends and outliers (Kim et al., 2021). For example, a line chart may depict an important trend whereas a scatterplot may visually communicate correlations and outliers. Existing table-to-text approaches are not designed to explain such visually salient chart features in summaries. There are two main impediments to addressing the chart summarization task. First, the lack of large-scale datasets makes it difficult to solve the task using data-driven neural models. Second, there are no strong baselines that utilize the latest advances in neural text generation tasks. Obeid and Hoque (2020) made an initial attempt to address this problem with a dataset and a model that utilizes a Transformer (Vaswani et al., 2017) architecture. However, their dataset was built by collecting a small set of charts (8,305) from a single source covering only two types of charts (bar and line). Also, their approach does not exploit the recent advances in large-scale language model pretraining, which has been shown to be very beneficial for many vision and language tasks (Devlin et al., 2019; Touvron et al., 2021). To our knowledge, there is no large-scale benchmark with a wider range of topics from multiple sources, covering many different chart types, and with models that employ large-scale pretraining. In this work, we present a large-scale benchmark for chart-to-text with two datasets consisting of 44,096 charts covering a broad range of topics and a variety of chart types. We introduce two variations of the problem. The first variation assumes that the underlying data table of a chart is available, while the other introduces a more challenging and realistic scenario by assuming that the chart is in image format and the underlying table is not available. These two problem scenarios motivated us to adapt a variety of state-of-the-art models that combine computer vision and natural language generation techniques as strong baselines; see Fig. 1 for a sample model output. Our primary contributions are: ( i ) a new large-scale benchmark covering a wide range of topics and chart types; ( ii ) a set of state-of-the-art neural models which can act as a starting point for other researchers to expand and improve upon; and ( iii ) a series of automatic and human evaluations as well as in-depth qualitative analysis to identify further challenges. Our code and benchmark datasets are publicly available at https://github.com/vis-nlp/Chart-to-text. Chart Summarization Early work (Mittal et al., 1998; Ferres et al., 2013) followed a planning-based architecture (Reiter, 2007) and used templates to generate texts. These systems only describe how to read the chart rather than explain key insights conveyed by the chart. Recently, commercial systems such as Quill and Wordsmith 1 as well as research prototypes, e.g., (Cui et al., 2019) and (Srinivasan et al., 2018) computed statistics ( e.g., extrema, outliers) to present facts from a dataset. Demir et al. (2012) also compute statistics to generates bar chart summaries in a bottomup manner to simultaneously construct the discourse and sentence structures. Recently, Chen et al. (2019) used the ResNet (He et al., 2016) to encode the chart image and an LSTM decoder to create the caption. A key limitation of the above bodies of work is that sentences are generated using predefined templates, which may lack generality and offer little variation in terms of reported insights, grammatical styles and lexical choices compared to data-driven models. Moving beyond template-based summaries, Obeid and Hoque (2020) adapted a transformer-based model on a dataset of 8,305 charts, while Spreafico and Carenini (2020) applied an LSTM based encoder-decoder model on a dataset of 306 chart summaries. Both studies used much smaller datasets and did not consider the computer vision aspects of the problem. Hsu et al. (2021) recently use a CNN+LSTM based image captioning model for scientific figure captioning. In contrast, we focus on the generic chart-to-text problem and train several neural models that combine computer vision and data2text generation. Data2text Generation Data2text models generate a descriptive summary for a table of records. They have been used for various domain-specific tasks such as summarizing sports data (Barzilay and Lapata, 2005; Wiseman et al., 2017), weather-1 Narrative Science Quill; Automated Insights Wordsmith 4006 forecast data (Reiter et al., 2005), recipe generation (Yang et al., 2017) and biography generation (Lebret et al., 2016) as well as open-domain tasks (Parikh et al., 2020; Chen et al., 2020a). Recent methods have primarily used an LSTM-based encoder-decoder architecture (Mei et al., 2016; Le-bret et al., 2016; Wiseman et al., 2017). Gong et al. (2019) found that transformers (Vaswani et al., 2017) yielded more fluent and coherent outputs compared to their LSTM counterparts. Others focused on controlling the structure of the summary using a planning approach (Su et al., 2021) as well as generating facts by preforming logical inference over the given table (Chen et al., 2020a,b). Image Captioning There has been swift progress in image captioning largely due to the availability of large-scale datasets (Agrawal et al., 2019; Chen et al., 2015). Zhang et al. (2021) developed an object detection model to summarize objects in images while Sidorov et al. (2020) utilized texts extracted from images using OCR to generate captions. Unlike images with real-world objects and scenes, charts have marks ( e.g., bars, lines) that map quantitative data. This makes the chart-to-text problem different from image captioning. After searching through various sources including news sites, textbooks, and websites containing data facts, we found two suitable sources with sufficiently large numbers and varieties of charts with textual descriptions as we describe below. Statista Statista (statista.com) is an online platform that regularly publishes charts on a wide range of topics including economics, market and opinion research. We crawled 34,810 publicly accessible webpages in December 2020, yielding a total of 34,811 charts. For each chart, we took a screenshot of the chart image, downloaded the data table, the title, axis labels and the human-written descriptions about the chart. We classified the charts into two groups based on the number of columns in their underlying data tables: Data tables of simple charts have only two columns, whereas complex charts involve at least three columns ( e.g., stacked or group bar charts, line charts with multiple lines). public opinion and demographic trends. The articles are often accompanied by multiple charts along with high-quality descriptions written by professional editors. We scraped 3,999 publicly accessible pages in January 2021, which gave a total of 9,285 charts. Unlike Statista, the Pew reports do not provide the underlying data tables for most of the charts. Among 9,285 charts, only 143 have underlying data tables. For each chart, we downloaded the chart image, the surrounding paragraphs and the alternative text associated with the image (using the alt attribute), if it was available. Like a title, the alt text often gives a very short chart description. Finally, we classified the charts into simple and complex manually since underlying data tables were unavailable. Below we describe two main steps of the data annotation process for each chart: ( i ) identify the relevant summary, and ( ii ) extract data. Additional details of these steps are provided in Appendix A.1. Statista We chose the first part of the text (from the chart icon to the next heading) as the chart summary. This is based on the observation that the first part provides a succinct summary of the chart while the remaining parts often contain background information ( e.g., the history of a company). Extracting data from the Statista charts was relatively straightforward as the underlying data tables were available. However, most charts (32,660 out of 34,811) did not provide x-axis labels. To assign representative labels for them, we first used regular expressions on the cell values of such a column to see if it represents common entities ( e.g., year , location ). Still, there were 7,170 missing labels remaining. We then applied the Wikidata knowledge base (Wik, 2021) to automatically derive an entity type label based on the data values plotted on x-axis. However, sometimes the resulting labels were too generic ( e.g., human , business ). Hence, we manually annotated each label by either accepting the entity type label, if it represents the x-axis accurately, or entering a more specific name. Pew The annotation for Pew was more challenging as often a webpage contains many charts and paragraphs do not explicitly refer to their relevant chart. Also, most charts did not have underlying data tables. To address these challenges, we construct the dataset in three stages (Fig. 2). ( i ) Data extraction from chart images: We first extracted the text from the charts using CRAFT (Baek et al., 2019a,b), a state-of-the-art OCR model. We then extracted the bounding boxes of the detected texts to extract geometric features ( e.g., normalized width and height of the text) and used them to train a gradient boosting classifier that categorizes the recognized text into one of the following categories: title, axis labels, legends, and data labels. Since the visual style and structure vary among chart types, we trained a separate classifier for each chart type. We manually labeled 319 examples (171 bar, 68 line, and 80 pie charts) and split them into train, validation, and test splits with 8:1:1 ratios, respectively. Our models achieved a precision of 95.0% overall and 97.6% for title classification on our test set. We then used our models to predict the text roles for the remaining charts in the Pew dataset. We used the extracted title as the final chart title if there was no associated alt text with the chart image. If the alt text was available, we took the longer one by comparing it with the extracted title. ( ii ) Identification of candidate paragraphs: We observed that relevant paragraphs tend to appear in close proximity to a given chart and share some content with the chart ( e.g., axis labels, data values). We first used this proximity criteria to form a list of candidate paragraphs L c . Specifically, for each chart, we selected the paragraph adjacent to the chart as well as the five paragraphs before and after it as candidates (maximum of 11 in total). Next, we used a heuristic-based approach to automatically select a subset of relevant paragraphs L r  L c . We estimated the relevance score of each paragraph in L c to its corresponding chart as rel = content  proximity , where content takes a weighted sum of the number of tokens matched between the paragraph and the OCR-extracted text (numerical tokens were given a higher weight than Statista Pew Type Simple Complex Simple Complex Bar 24,591 5,616 807 5,497 Line 2,646 902 325 2,129 Area 0 0 29 105 Scatter 0 0 0 68 Pie 409 0 325 0 Table 223 424 0 0 Total 27,869 6,942 1,486 7,799 Table 1: Chart type distribution. lexical tokens as they were better indicators of rel-evance), and proximity is based on the distance between the chart and the paragraph. If rel exceeds a threshold and some minimum number of lexical and numerical tokens are matched between the paragraph and chart, we consider such a paragraph to be relevant to the chart. We set this threshold empirically and chose it to be aggressively high to prioritize precision over recall. We evaluated the efficacy of our approach against a randomly sampled set of 95 charts and 769 surrounding paragraphs and found a recall of 21.1% and a precision of 100%. Given the perfect precision score, we considered the paragraphs in L r to be relevant and to confirm the relevance of the remaining paragraphs, we performed a human study. ( iii ) Selection of relevant paragraphs: We asked crowdworkers on Amazon Mechanical Turk to label how relevant each paragraph is to its chart. A total of 5,478 charts and 13,237 paragraphs were annotated. Each chart received two annotations from two workers. If both workers labeled a paragraph as either completely irrelevant or relevant (partially/completely), we used the label that they agreed upon as the final label. 2 For the remaining 2,888 paragraphs where the workers disagreed, we resolved them through internal annotation. Our chart-to-text datasets contain a diverse range of chart types (Table 1). Bar charts make up the majority of the charts both in Statista (87.9%) and Pew (67.9%) for both simple as well as stacked and group bar charts. The next most common type is line charts (10.2% in Statista and 26.4% in Pew). To analyze the topic distribution, we extracted the topic of each chart using its webpage's metadata ( e.g., breadcrumbs, meta-tags). Our datasets cover a broad range of topics including politics, society and health (see Fig. 9 in Appendix A.3). The topics in Statista are more evenly distributed than the ones in Pew, which is dominated by U.S. Politics & Policy (45.4%). Table 2 presents basic linguistic statistics about the datasets. The summaries in Pew are about twice as long as the those in Statista, in terms of average character, token and sentence count. Unsurprisingly, complex charts generally have longer summaries than their simple counterparts. We further analyzed the semantic content of the summaries using 100 randomly sampled chart-summary pairs from each dataset. Table 3 shows the distribution of sentences across the four main types of semantic content. 3 We notice that statistical and comparative information ( e.g., min, max, avg.) is the most common type of content in both datasets. Summaries in Pew tend to report more insights that require more perceptual and cognitive efforts ( e.g., trends and causal relations) which are arguably more challenging to generate compared to simple statistics. Both datasets contain comparable proportions of sentences covering contextual and domain-specific information. Unlike Statista, Pew summaries rarely explain the chart types and encodings ( e.g., what do the xand yaxes represent). Problem Definition We consider two variations of the chart-to-text problem. In the first variation, we assume that the underlying data table of the chart is available, where the dataset can be represented as a set of 4-element tuples D = 3 Our categorization of content is inspired by a recent study (Lundgard and Satyanarayan, 2022). { C, T, M, S  n }  D  n = 1 with C , T , M and S representing the chart image, data table, metadata and textual summary, respectively. For each cell in the data table T , we have the following information: ( i ) the string value, ( ii ) the row and column positions, and ( iii ) whether it is a header cell or not. The metadata M = ( C title , C type , C labels ) consists of the title, type ( e.g., bar, line) and axis labels. In the second variation, we assume that the data table is not available which makes the problem more challenging as well as realistic because most charts online are in image format and do not have the underlying data tables. For a given input X =  C, T, M  or  C, M  , our goal is to generate a textual description  S which is a good summary of the chart according to a set of evaluation measures. We consider three categories of models to tackle the task. The first category is image captioning models, where the task is formulated as generating a textual description for the given chart image. The second category is data-to-text models, which rely on the underlying data tables of the charts to produce the corresponding descriptions. Finally, we consider a combination of vision and text models, where the models first extract the text using the CRAFT OCR model (Baek et al., 2019b) and then train with a data-to-text setup. We present three categories of models below (hyperparameter settings for all the models are provided in Appendix A.3). We develop over the Show, Attend, and Tell (SAT) model (Xu et al., 2015) to probe the effectiveness of this category of models for our task. Following Xu et al. (2015), we use the ResNet50 (He et al., 2016) as the image encoder and a unidirectional LSTM (Hochreiter and Schmidhuber, 1997) as the decoder for text. As the pretrained ResNet50 model is trained on object detection tasks on ImageNet (Deng et al., 2009), directly applying it to chart images gave poor results in our experiments. Also, we do not have any object labels for the chart images to train the encoder. Hence, we employ the recently proposed self-supervised strategy called Barlow Twins (Zbontar et al., 2021) which tries to make the embedding vectors of distorted versions of an image sample to be similar, while minimizing the redundancy between the components of these vectors. It achieves state-of-the-art results for ImageNet classification with an accuracy gap of only 3.3% from the supervised model. We pretrain a 4009 separate ResNet50 with Barlow Twins for each of our datasets and use it as an encoder in the model. Chart2text (Obeid and Hoque, 2020) is an adapted transformer model for chart-to-text based on the data-to-text model of Gong et al. (2019). It takes a sequence of data records as input with each record being a set of tuples ( e.g., column header, cell value, column index) and embeds them into feature vectors with positional encodings to distinguish orders (Fig. 3a). The model includes an auxiliary training objective (binary labels indicating the presence of the record in the output sequence) on the encoder to maximize the content selection score. It also implements a templating strategy of target text with data variables ( e.g., cells , axis labels ) to alleviate hallucination problems. Since in Pew data tables are not available, we use OCR-generated texts as inputs which are linearized and embedded into feature vectors. The bounding box information of OCR-generated data of each chart is also embedded and concatenated to the table vectors to provide positional information to the model. Field-Infusing Model (Chen et al., 2020a) is inspired by the concept-to-text work (Lebret et al., 2016). The values in a cell are first encoded with an LSTM, which is then concatenated with the embeddings of row index and column heading. These table representations ( h 1 , h 2 in Fig. 3b) are then fed into a 3-layer Transformer encoder-decoder model to generate the target summaries. Additionally, for Pew, we embed the bounding box information of the chart OCR-texts and concatenate it to the LSTM-based field representation as an auxiliary positional information to the model. BART (Lewis et al., 2020) adopts a seq2seq Transformer architecture with denoising pretraining objectives. It is particularly pretrained to be effective for text generation tasks. For our chart-to-text tasks, we flatten the data table row by row and concatenate the title with table content as the input to the encoder (Fig. 3c). In the absence of data tables, we concatenate all the OCR-texts in a top to bottom order and fed it to the model as input. T5 (Raffel et al., 2020) is a unified seq2seq Transformer model that converts various NLP tasks into a text2text generation format. It is first pretrained with a fill-in-the-blank' denoising objective, where 15% of the input tokens are randomly dropped out. The spans of consecutive dropped-out tokens are replaced by a sentinel token. The decoder then has to predict all of the dropped-out token spans, delimited by the same sentinel tokens used in the input. This is different from the pretraining objective of BART where the decoder predicts the entire original sequence (not just the dropped spans). T5 is fine-tuned with several supervised multi-task training objectives ( e.g., machine translation, text summarization). We format the input in the same way as for the BART models. Specifically, we add translate Chart to Text: " to the prefix of the input to mimic the pretraining process (see Fig. 3c). For OCR-based input, we experiment with two T5 model variants. In the first variant, we concatenate all the OCR-extracted sentences from the chart image in a top to bottom order and fed it to the model as input. In the second, we modify the input to accommodate the spatial information of the detected texts. Inspired by Tan and Bansal (2019), we feed the bounding box coordinates of each detected text token into a linear layer to produce positional embeddings which are then added to their corresponding embeddings of the OCR tokens as input. 5 Evaluation 5.1 Automatic Evaluation Measures For automatic evaluation of the summary quality, we utilized five measures. BLEU (Post, 2018) and CIDEr (Vedantam et al., 2015) measure n-gram overlaps between the model generated text and the reference text. CIDEr computes TF-IDF weighted n-gram overlaps. BLEURT (Sel-lam et al., 2020) is a model-based evaluation metric that indicates to what extent the candidate is grammatical and conveys the meaning of the reference. We use BLEURT-base-128. Content Selection (CS) metric measures how well the generated summaries match the gold summaries in terms of selecting records to generate (Wiseman et al., 2017). Since both the BLEURT and CS are calculated at the sentence-level, we average these scores over the whole test set. Finally, for readability and fluency, we measure Perplexity (PPL) using a pre-trained GPT-2 Medium (Radford et al., 2019). Results In general, from the results in Table 4, we notice that large-scale unsupervised pretraining ( i.e.,  -BART",  -T5") helps to boost the performance significantly. In terms of the model variants, the image captioning model has failed to capture 4010 Transformer Encoder Prediction Layer Transformer Decoder Softmax Substitute Variables Quarter|Q3'20|x|line_chart 0/1 0/1 0/1 <sos>     How many users  <eos> How many templateLabel[1][2] <eos> relevant information from charts (low CS score) even though it generates fluent text (low PPL). On Statista, when the data tables are available, Chart2text and Field-Infuse models are able to extract information from the data table, but they struggle to produce texts with good quality. This could be because these models did not use any large-scale pretraining. On the other hand, TAB-BART and TAB-T5 are able to produce well-structured and relevant summaries. The OCR-based models can generally generate fluent summaries but they are slightly less effective in extracting the relevant information since the OCR process introduces some noise in the input data. We also experiment with automatically extracted tables to see how the models perform in the absence of gold data tables. To this end, we extended ChartOCR (Luo et al., 2021), which predicts the raw data values of chart elements, to extract the fully-structured data table. The accuracy of automatic data extraction was 77.31% (see Appendix A.5 for details). We find that similar to OCR-based models, TAB_OCR-based models tend to be less effective in extracting the relevant information compared to their TAB-based counterparts which use ground truth data tables. Pew, on the other hand, is much challenging because it contains many charts with ill-defined structure and the underlying data tables are not available. Unsurprisingly, the performance of all the models has dropped significantly compared to that on Statista. Nonetheless, we can see that without the presence of the underlying data table, the vision+text (OCR-based) models have brought notable improvements over the vision only model. Further breakdown of model performance based on chart types is provided in Appendix A.4.2. We also evaluate the transferability of the models and the datasets, where we first pretrain a model on a source dataset and fine-tune it on the target dataset. In addition to our two datasets (Statista or Pew), we experiment with ToTTo (Parikh et al., 2020) as another source dataset, which is a large-scale open-domain English table-to-text dataset. Our results show that pretraining on other datasets only brings about marginal improvement. Details of this experiment can be found in Appendix A.4.1. To further assess the summary quality we performed a human evaluation on 150 randomly sampled charts from the Statista dataset with four internal annotators who are native speakers of English. For each chart, annotators performed pairwise comparisons between the outputs of TAB-T5, OCR-T5 and the original gold summary (served as a control), resulting in a total of 450 pairwise comparisons (Appendix A.4.3). They compared the summaries based on three criteria: ( i ) Factual correctness : Which summary is more factually 4011 TAB-T5 (1) vs. OCR-T5 (2) Gold (1) vs. TAB-T5 (2) Gold (1) vs. OCR-T5 (2) Summary Factual Coherence Fluency Factual Coherence Fluency Factual Coherence Fluency Summary 1 Win 55.3% 23.3% 20.0% 30.0% 36.7% 22.0% 59.3% 43.3% 28.7% Summary 2 Win 12.0% 11.3% 11.3% 13.3% 16.7% 14.0% 7.33% 15.3% 17.3% Tie 32.7% 65.3% 68.7% 56.7% 46.7% 64.0% 33.3% 41.3% 54.0% p -value (sign test) 1.86e-11 8.77e-3 0.0395 1.31e-3 5.26e-4 0.0668 1.27e-16 4.25e-6 0.0266 Table 5: Human evaluation results for comparing between the outputs of TAB-T5, OCR-T5 and the gold summary. correct ( i.e., facts mentioned are supported by the chart)? ( ii ) Coherence : Which summary is more coherent ( i.e., sentences are well connected)? and ( iii ) Fluency : Which summary is more fluent and grammatically correct? For each criterion, the annotator picked the better one (win) or equally good (tie). Each comparison was performed by one annotator, except the first 150 comparisons for which we had two annotators to measure the agreement. The agreement for these 150 comparisons, excluding ties, was 74.3% (ties were excluded since they do not affect the overall ranking of the summaries). Table 5 shows that the TAB-T5 performed significantly better than OCR-T5 based on all three criteria, especially on factual correctness. This is likely because, without the data table as input, OCR-T5 model often fails to generate factually correct statements from the OCR text. We also observe that while the fluency of the model outputs is comparable to the gold summary, their factual correctness and coherence were significantly worse, especially for the OCR-T5 model. We manually analyzed 200 random samples from Statista and Pew. We chose TAB-T5 and OCR-T5 for Statista and OCR-BART and OCR-T5 models for Pew. This analysis helps us to understand model errors and identify key challenges that existing models face as we describe below. Perceptual and reasoning aspects As mentioned in 1, charts often describe complex patterns and trends which can be perceived by humans easily but they are not necessarily easy to derive through analysis of raw data tables. In Fig. 4b, the OCR-T5 model manages to describe a trend correctly in the first sentence but describes a trend incorrectly in the last sentence. These examples demonstrate the shortcomings of existing models. In order to explain perceptual and reasoning aspects effectively, we need more sophisticated models that better capture prominent visual relationships in charts. In particular, we aim to develop better representations including semantic graph representation of the chart that encodes numerical and logical relationships among chart objects. Hallucinations Sometimes, the model outputs tokens that are irrelevant to the chart. For example, while the model outputs in Fig. 4a,b are quite fluent, they contain hallucination errors. This problem is commonly observed in other data-to-text work as well (Wiseman et al., 2017; Parikh et al., 2020). Factual errors Factually incorrect statements are more common for the OCR-based models ( e.g., in Fig. 4a-b) since they do not take the data table as input, thus fail to associate the data values correctly. In contrast, TAB-T5 which utilizes the data table as input tends to generate less factual errors. This confirms that summarizing charts when the data table is not available is usually more challenging. Computer vision challenges The factual errors illustrate some unique computer vision challenges. First, charts do not always show data values as text labels, thus the OCR models cannot access those values. Even if the data values are labeled, the absence of association between data values ( e.g., Instagram is related to 380.09M in Fig. 4a) leads to factual errors. This problem might be alleviated if the model can extract the data table from a chart image. While there are some initial attempts in this direction ( e.g., Luo et al. (2021); Choi et al. (2019)), more accurate data extraction from charts is necessary. Generalizability The charts in our benchmark cover several different chart types and a wide variety of topics (fig. 9). The charts in the Pew in particular have a wide variety of visual styles in terms of color, layout and typography as they were created over several years by different authors (see examples in fig. 1). Nevertheless, finding more chart-summary pairs with more diverse visual styles is an open challenge. In future, we aim to find more different sources of chart-summaries and perform cross-domain experiments across those different sources to evaluate the generalizability of models. We have presented two large-scale datasets for chart summarization. We also provided several state-of-the-art baselines and measures. Our evaluation highlights the promise of these baselines and also reveals several unique challenges for the chart summarization task. We hope that Chart-to-text will serve as a useful research benchmark for model and metric development and motivate other researchers to explore this relatively new area. The authors would like to thank the anonymous reviewers for their helpful comments. This research was supported by the Natural Sciences & Engineering Research Council (NSERC) of Canada. During the dataset collection and annotation process, we had many ethical issues to take into consideration. To respect the intellectual property of the chart publishers, we only used publicly available charts from resources that provide publication rights of downloaded content for academic purposes. According to the terms of use and publication rights for Statista, 4 users are granted publication rights only to free studies of Statista, so we only used the free publicly available webpages. According to the terms and conditions for Pew, 5 users are allowed to use the content as long as they are attributed to the Center or are not attributed to a different party. To fairly compensate the Mechanical Turk annotators, we compensated the annotators based on the minimum wage in the United States at the time (7.25 US$ per hour) and the estimated time taken for each task (1 minute). Hence, these annotators received 0.10 0.15 US$ for each chart, depending on the number of candidate paragraphs associated with it. Additionally, to protect the privacy of these annotators, all of their annotations were anonymized. To ensure the reproducibility of our experimental results, we have provided the hyperparameter settings and estimated training time in Appendix A.3. We foresee one possible misuse of our models that is to spread misinformation. Currently, our model outputs tend to appear fluent but contain some hallucinations and factual errors, as detailed in 5.3. Hence, if such model outputs are published without being corrected, it may mislead and misinform the general public.
abstain abstain method method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain objective objective other other other other other other other other other other objective other other other other other other other other other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method method abstain method method method abstain abstain abstain abstain method method abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain method abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method result objective other other method abstain method abstain method abstain abstain abstain abstain abstain abstain