Recent neural text generation models have shown significant improvement in generating descriptive text from structured data such as table formats. One of the remaining important challenges is generating more analytical descriptions that can be inferred from facts in a data source. The use of a template-based generator and a pointer-generator is among the potential alternatives for table-to-text generators. In this paper, we propose a framework consisting of a pre-trained model and a copy mechanism. The pre-trained models are fine-tuned to produce fluent text that is enriched with numerical reasoning. However, it still lacks fidelity to the table contents. The copy mechanism is incorporated in the fine-tuning step by using general placeholders to avoid producing hallucinated phrases that are not supported by a table while preserving high fluency. In summary, our contributions are (1) a new dataset for numerical table-to-text generation using pairs of a table and a paragraph of a table description with richer inference from scientific papers, and (2) a table-to-text generation framework enriched with numerical reasoning. Recent data-to-text generation studies have shown significant improvement in generating faithful text aligned with data sources. A copy mechanism has been widely explored to improve faithfulness in various ways. Wiseman et al. (2017) used joint probabilities to let models choose between copying records from data sources or generating from a vocabulary. Puduppully et al. (2019) improved a similar approach by modeling entity representations as a unit of copying. This approach has proven to be effective in generating descriptive text that explicitly mentions facts from sources. However, as introduced by Chen et al. (2020a), humans have the ability to produce more analyti-Model Precision Recall F1 Our full model 89.6 82.2 85.7 Lee et al. (2018) 86.2 83.7 84.9 Table 2: The overall mention detection results on the test set of OntoNotes. cal text with richer inference, including numerical reasoning. Making inferences beyond texts is still an open question due to the limitation of language models in handling numeric operations. In this study, we further encourage research by elaborating numerical tables to initialize the ability to inject reasoning while maintaining high fluency. Our contributions are summarized as follows. We introduce a new dataset for table-to-text generation focusing on numerical reasoning. The dataset consists of textual descriptions of numerical tables from scientific papers. Our dataset is publicly available on https://github.com/titech-nlp/numeric-nlg. We adopt template-guided text generation (Kale and Rastogi, 2020a) for a table-to-text generation task and propose injecting preexecuted numerical operations in the template to guide numerical-reasoning-based text generation. We compare different types of templates for table representations in pre-trained models. We propose a copy mechanism for pre-trained models, that uses general placeholders covering table contents and results of pre-executed numerical operations to avoid fact hallucination. We conduct experiments with current state-of-the-art neural generation models and a simple template-based system to demonstrate the challenges and opportunities for future research on text generation with numerical reasoning. The power of tables in presenting data efficiently further encourages research done by exploring the tables as data sources in natural language tasks, such as table-to-text generation (Liang et al., 2009; Wiseman et al., 2017; Lebret et al., 2016; Parikh et al., 2020), table question answering (Pasupat and Liang, 2015; Wang et al., 2018), and table-based fact verification (Chen et al., 2020b; Gupta et al., 2020). Recent research on the table-to-text generation task is starting to generate text with more reasoning. Murakami et al. (2017) explored stock prices to generate market comments by adding generalization tags of possible arithmetic operations to cover mathematical reasoning. Nie et al. (2018) proposed operation-guided attentions by exploring the results of pre-executed numerical operations. The dataset closest to ours is LOGICNLG, by Chen et al. (2020a), who first introduced logical text generation using open-domain tables with unknown schemas. Different from our target text for generation, which consists of several sentences in a paragraph, they proposed a task of generating only one sentence from selected table contents. We created numericNLG, a new table-to-text dataset focusing on a text generation task with numerical reasoning. We collected table descriptions from scientific papers, that are naturally produced by experts with richer inference. Data Acquisition We constructed a table-to-text dataset based on numerical tables of experimental results, extracted from PDF files of scientific papers on the ACL Anthology website, 1 introduced 1 https://www.aclweb.org/anthology/ by Suadaa et al. (2021). Then, we collected candidates for corresponding descriptions from the source files using PDFMiner. 2 We used table numbers in their captions as keywords for the collection. An example of a table and its description is shown in Figure 1. Data Cleansing and Annotation Extracted table descriptions can be noisy since they may contain only table numbers without any sentences describing table facts. We hired experts in the computer science field to clean and annotate the extracted descriptions in the following steps:  Examine tables and their corresponding descriptions and then recommend only the descriptions that have at least one sentence representing numerical facts in the table. Categorize each sentence of the recommended description into three fact-checking classes: data description, supporting description, and not-related-to-table description. As a final dataset, we used only sentences classified as belonging to the data description category to reduce fact hallucination. Identify a content plan of table description by selecting part of table headers which directly stated or logically inferred in the description, called target header. For example, refer to the table description shown in Figure 1, Our full model is selected as the target header. Table 1 provides a comparison of numericNLG with other related table-to-text datasets. The ROTOWIRE (Wiseman et al., 2017) dataset consists of summaries of NBA basketball games containing several paragraphs, paired with their corresponding box-score tables. Since ROTOWIRE has only 39 record types, each table contains similar record types with limited schemas. Although most of the ROTOWIRE table contents are in numerical values, the summaries contain only a few numerical-reasoning sentences, such as a comparison of scores between two basketball teams. While our dataset consists of closed domain articles as 2 http://pypi.python.org/pypi/pdfminer/ Tables Examples Unit of Desc. with ROTOWIRE, it is of shorter text (a paragraph) and with unlimited table schemas. Chen et al. (2020a) introduced the LOGICNLG dataset to facilitate the study of table-to-text generation tasks with richer inference. The dataset contains unlimited schemas of open-domain tables crawled from Wikipedia, paired with five annotated sentences covering different logical inferences. Although most inferences are numerical reasoning, the table contents are not fully numeric. Similar in motivation to LOGICNLG in generating text that can be logically entailed by facts in tables, numericNLG consists of collections of paragraphs that are naturally produced by human experts in scientific papers, paired with their corresponding numerical tables. Our dataset has fewer tables than LOGICNLG, focusing on numerical-reasoning text in the scientific domain. Due to ROTOWIRE's limited schemas, Wiseman et al. (2017) viewed a table input as a set of records (entity, value, type), where the entity and the type are the extracted row and column names, respectively. Because of the unlimited table schemas in our dataset, by capturing the original table structure in real-world tables, this paper uses the representations which consist of captions, row headers, column headers, cell values, and metrics, called a data table. Using only descriptive facts from the data table as input representations is sufficient to generate descriptive texts that explicitly mention facts in the table. However, since we intend to produce more analytical text with numerical reasoning, we propose adding inferred facts to the input representation by computing a set of arithmetic operations on the data table beforehand, defined as a pre-executed operation table. Data Table We view T as a set of cells with their corresponding row header ( rh ), column header ( ch ), numerical value ( val ), and metric-type ( m ), defined as a data table ( TD ). A data table for the example in Figure 1 consists of rh : ((model, our full model), (model, lee et al. (2018))); ch : (); val : (( 89 . 6 , 82 . 2 , 85 . 7 ), ( 86 . 2 , 83 . 7 , 84 . 9 ); and m : (precision, recall, f1). Since our tables are annotated with a targeted header as a content plan for table descriptions, we mark cells corresponding to the targeted header with a target flag ( tgt ) to highlight the marked cells in text generation. We set tgt = 1 for targeted cells and tgt = 0 for non-targeted cells. In this study, we preprocess the header name by concatenating the row and column headers ( h = [ rh ; ch ] ) and keep information about the header category by extracting overlapping tokens of row and column headers as th . As a result, we define TD = ( h ij , th ij , val ij , m ij , tgt ij ) , where 1  i  n r , 1  j  n c ; n r and n c are the numbers of rows and columns, respectively. Pre-executed Operation Table We provide a table of pre-executed cell operations ( TOP ) by doing mathematical operations only on targeted cells to limit the calculation. In this study, we cover maximum, minimum, and difference operations. Examples of a preprocessed table, data table, and pre-executed operation table are shown in Figure 2. Linearized Table Supporting transfer learning of pre-trained transformers to our table-to-text generation task, we prepare a linearized table PT as an input representation so that it similar to the representation that encoder has seen during pre-training. T is converted to a flat string PT = w 1 , ..., w | PT | , similar to that used in many prior work (Wang et al., 2020; Chen et al., 2020a; Kale and Rastogi, 2020b), where w i denotes the i -th word in paragraph PT with length | PT | . In this study, we adopt the template-based input representation, introduced by Kale and Rastogi (2020a), to handle representation bias between a structured data T and a natural language utterance PT , where PT is generated using a manually defined template. We propose not only covering data table TD in the template but also injecting the pre-executed numerical operations of table T through TOP to guide numerical-reasoning-based text generation. We consider four different methods 3 for converting T into sequences, the last two being our contributions. 3 An example is shown in Table 6 in the appendix. 1. Naive Representation T is simply flattened into a sequence ignoring its table structure by concatenating captions, headers, metrics, and targeted cell values: caption: <table id> <caption> . row name: <rh 1 > . . . <rh nr > . column name: <ch 1 > . . . <ch nc > . metric: <m 1 > , ..., <m nr/nc > . value: <val 1 . 1 > . . . <val nr.nc > . 2. Data-based Template ( TD temp) T is transformed into a natural language sentence by scanning each row of TD with tgt = 1 to fill a manually defined template: <table id> shows <caption> . <m 1 . 1 > of <h 1 . 1 > is <val 1 . 1 > . . . <m nr.nc > of <h nr.nc > is <val nr.nc > . This representation covers the semantics of data in the original table. 3. Reasoning-based Template ( TOP temp) Mathematical operation arguments and results from TOP are injected in this representation to cover the numerical reasoning of data in the original table. This naive representation omits the relation between rows and columns. Note that <table id> is extracted from the caption to support table mentioning in generating table descriptions. We define h op and val op as a header and a value of an operation result respectively, where op = { max, min, diff } . Specific to the difference operation, h diff 1 and h diff 2 refer to the first and second header arguments, respectively. Then, T is represented by concatenating the templatized representation for each row of TOP : <table id> shows <caption> . <h max > has the largest <m max > ( <val max > ) of <th max > . <h min > has the smallest <m min > ( <val max > ) of <th min > . <m diff > of <h diff 1 > is larger/smaller than <h diff 2 >. 4. Data and Reasoning-based Template ( TD + TOP temp) T is converted by combining templatized sentences of TD and TOP . This representation covers both data and their numerical reasoning. The task is to generate text by translating table representation PT into table description Y = y 1 , y 2 , ..., y n . We apply a series of generation models to solve the proposed task. While our focus is primarily on pre-trained models since they have been most widely used for limited data settings, <table_id> shows that <header_max> achieves higher <metric_max> and <metric_max> score. like ours, we also include a template-based generator and a pointer-generator network as baselines. Template-based Generator We design a domain-specific template-based generator covering two types of sentences in producing table descriptions: table referring sentences and data description sentences. Since our task focuses on numerical-reasoning descriptions, we define templatized sentences using maximum records in table TOP : <table id> shows <caption> . we can see that <h max > outperforms other <th max > with <val max > of <m max > . Pointer-Generator Pointer-generator (See et al., 2017) is a sequence-to-sequence model with attention and a copy mechanism. This model copes with the out-of-vocabulary problem in data-to-text generation by jointly copying from source texts and generating from a vocabulary. Fine-tuned GPT2 GPT2 (Radford et al., 2019) is a pre-trained language model with a decoder-only transformer architecture. We fine-tuned the GPT2 model by using table representation PT as a prefix of our input. Specifically, we fed the concatenation of table representation PT and table description Y to the model and generated Y . In the inference phase, we used only PT as the input to generate  Y starting after the last token of PT . Fine-tuned T5 T5 (Raffel et al., 2020) is a pre-trained transformer model with an encoder-decoder architecture, that solves natural language tasks by converting into a text-to-text format. We fine-tuned the T5 model in our dataset by adding a summa-rize prefix to table representation PT producing output  Y . Copy Mechanism Pre-trained language models have proven their effectiveness in handling the open vocabulary problem through subword tokenization. Supported by attention layers of the transformer in their architecture, the models learn to attend to source inputs while generating target texts in subword units. However, pre-trained generators often produce texts that are not aligned to table sources. In this study, we propose strengthening their copying ability by incorporating a copy mechanism into the pre-trained models. Although a copy mechanism based on pointer-generator (See et al., 2017) was used for pre-trained models (Chen et al., 2020c) and is well-known in the community, it cannot maintain the global logical structure of sentences with richer inference. We instead employed a simpler copy mechanism based on placeholders (Murakami et al., 2017) with more specific tags than in Chen et al. (2020a). We further propose a ranking-based placeholder alignment algorithm, as illustrated in Figure 3. First, we align entities and numbers in Y with the data tables TD and pre-executed arithmetic operation results TOP by using string matching. The alignment starts from the first row to the last row of TOP . If no matched token is found, it continues Model BLEU ROUGE-L METEOR BERTSCOREPARENT Template-based 2.82 26.97 15.82 86.88 17.15 Pointer-generator (naive) 2.80 15.26 7.82 76.38 1.40 Fine-tuned GPT2 (naive) 3.06 23.7 18.84 85.12 6.56 Fine-tuned GPT2 ( TD temp) 3.01 22.97 *17.10 *84.68 6.53 Fine-tuned GPT2 ( TOP temp) 4.63 *25.39 18.85 *85.66 7.72 Fine-tuned GPT2 ( TD + TOP temp) 5.05 *25.13 19.14 *85.40 8.05 Fine-tuned GPT2 (naive) + Copy 1.29 *11.66 *6.94 *78.73 *2.45 Fine-tuned GPT2 ( TD temp) + Copy 1.36 *11.23 *6.43 *77.76 *2.10 Fine-tuned GPT2 ( TOP temp) + Copy 1.18 *9.40 *4.42 *73.83 *0.91 Fine-tuned GPT2 ( TD + TOP temp) + Copy 1.22 *9.62 *5.47 *70.87 *1.55 Fine-tuned T5 (naive) 4.25 29.71 18.94 87.64 13.09 Fine-tuned T5 ( TD temp) 5.02 30.25 *20.11 87.68 15.09 Fine-tuned T5 ( TOP temp) 4.99 28.63 18.85 *87.17 12.25 Fine-tuned T5 ( TD + TOP temp) 4.83 29.13 18.46 87.34 12.78 Fine-tuned T5 (naive) + Copy 5.14 *27.40 18.49 *86.37 *12.47 Fine-tuned T5 ( TD temp) + Copy 4.96 *27.08 18.23 *86.12 *11.65 Fine-tuned T5 ( TOP temp) + Copy 5.24 *28.02 18.68 *86.52 *11.96 Fine-tuned T5 ( TD + TOP temp) + Copy 5.45 *28.15 19.16 *86.54 *12.95 Table 2: Experimental results of different models with various types of table representations and proposed copy mechanism. to the rows of TD . We set a higher rank to TOP than TD in the alignment since we focus on logical text generation. Then, we replace the matched tokens with corresponding placeholders 4 in a templatized description Y temp . As depicted in Figure 3, since our full model in sentence Y is matched with the header result of the maximum operation, we replace it with <header max> placeholder. During the fine-tuning phase, instead of directly generating Y , the models learn to produce a templatized description Y temp including placeholders as well as words. In the inference phase, we design a ranking algorithm with a placeholder memory to select the best-replaced tokens for placeholders of a predicted templatized description  Y temp in producing a generated description  Y . We define a set of values in the same row of source tables as a content set and prioritize replacing placeholders in one sentence with the same content set, ensuring sentence coherence. A content set of TD is a tuple of header, metric, and value. For TOP , a content set consists of header, metric, and value of the operation results. Specific to the difference operation, we add the header of the first and second arguments to the content set since the header arguments are important to capture entity comparison in a sentence. example, as shown in Figure 3, after replacing the header max placeholder with the header result from the first row of maximum records of TOP in Step 1, the related placeholders from the same content set ( metric max and value max ) are added to the placeholder memory as higher-ranked candidates in the searching space. The placeholder memory is reset to empty in the following sentence of  Y temp and the alignment starts again from the next content set of table sources. We conducted experiments on the proposed dataset to evaluate the performance of the text generation models and verify the effectiveness of the approach of using different table representations. We used BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) to evaluate the informativeness of generated texts. We computed the BERTSCORE (Zhang et al., 2020) to assess the similarity between the generated texts and the ground-truth table descriptions by using contextualized token embeddings of pre-trained BERT (Devlin et al., 2019), which have been shown to be effective for paraphrase detection. Considering both references and table contents, we also used the PARENT metric, proposed by Dhin-gra et al. (2019). In our experiments, we modified the PARENT calculation by adding noun phrases of table captions as table contents and used only targeted table contents for table sources. We trained a pointer-generator model using the Adagrad optimizer with a batch size of 8 and a learning rate of 0 . 15 . For fine-tuning the GPT2 model, the Adam optimizer set weight decay to 3  10  5 . Following Raffel et al. (2020), the T5 model was fine-tuned with a constant learning rate of 0.001. We trained all models for a maximum of ten epochs with early stopping based on the loss score on the validation set (patience of 3). At the time of decoding, the generated text was produced through a beam search of size 5. 7 Results 7.1 Automatic Evaluation Table 2 shows our experimental results. The fine-tuned T5 models performed better than the others in terms of BLEU, ROUGE-L, METEOR, and BERTSCORE . The slightly lower PARENT of the best fine-tuned T5 model than the template-based generator implies that the fine-tuned T5 model was also comparable in terms of generating related table descriptions. The pointer-generator model had the lowest score since our dataset consists of limited table collections with a broad vocabulary and challenging target texts. Effect of table representation Comparing the performance between table representation types in the pre-trained models, we can see a different tendency between GPT2 and T5. The more similar the table representation used as an input, the higher the score of GPT2. Since GPT2 had only a decoder, the inputs including reasoning-based templates ( TOP and TD + TOP ), which are more similar to our target with numerical reasoning, performed the best for several metrics with more than 1 point improvement. In T5 with an encoder-decoder architecture, on the contrary, there was only a slight margin between different table representations. This indicates that the encoder part of T5 can capture table contexts from various input templates. For variants without a copy mechanism, T5 with only data representation ( TD ) outperformed the other representation types with longer sentences for all metrics. Because of the gap between the encoder and decoder, T5 still had difficulty aligning the information of longer inputs and outputs. Effect of copy mechanism The worst scores of the fine-tuned GPT2+copy models indicate that our proposed copy mechanism failed to learn the templatized target patterns in the fine-tuning step. The decoder-only GPT2 could not handle the sparse distributions of target texts with placeholders. Conversely, the copy-based fine-tuned T5 models achieved a better BLEU score due to their encoder and decoder ability in handling output texts with placeholders. Table 3 shows table descriptions generated by the template-based, pointer-generator, and fine-tuned pre-trained models (GPT2 and T5), using data and reasoning-based templates 5 for our table example in Figure 2. We marked sentences related to table captions in green, correct facts based on table contents in blue, and incorrect facts in red. In this study, since we had a limited training set with a broader vocabulary, the pointer-generator model tended to result in repetitive words and failed to generate well-described descriptions. The pre-trained models, GPT2 and T5, generated more natural descriptions. While several pieces of text generated by GPT2 included numerical facts, they used numbers that were not extracted from table contents. The T5 models produced descriptions that were more related to table contents than GPT2. Considering our lengthy output examples in Table 3, unlike the fine-tuned GPT2 model, which generated longer sentences, the fine-tuned T5 model generated shorter sentences than the references. 6 The length gap between the references and outputs of the fine-tuned T5 model affected the F1-based metrics of ROUGE-L, METEOR, BERTSCORE , and PARENT. Note that BLEU is a precision-based metric that can handle shorter outputs through a brevity penalty (Papineni et al., 2002). Therefore, we assume that BLEU better represents the performance of the fine-tuned T5 model than the other metrics. We conducted a human evaluation 7 to better assess the quality of the generated text. We compared our copy-based fine-tuned T5 model with 5 Examples using other table representations are shown in Table 9 in the appendix. 6 Average token length of references: 80.57, GPT2: 87.39, GPT2+copy: 73.58, T5: 39.81, T5+copy: 41.81. the template-based, pointer-generator, fine-tuned GPT2, and fine-tuned T5 models. We did not compare it against the copy-based fine-tuned GPT2 since GPT2 failed to incorporate our proposed copy mechanism. We used the best table representation with majority metrics for each model on the basis of the experimental results in Table 2. In the first study, we evaluated the correctness of the generated text on the basis of facts in tables. We randomly selected 30 tables in the test set and elicited responses from three graduate students per table. Following Wiseman et al. (2017), the raters were asked to count how many facts in the descriptions were supported by numerical data in the tables and how many were contradicted. Since our task covers numerical-reasoning text, we distinguished descriptive numerical facts from inferred numerical facts. We also measured the level of relevance of the generated text to the table captions by using a four-point Likert scale (highly relevant, relevant, somewhat relevant, and irrelevant). The results are shown in Table 4. The pointer-generator failed to reflect facts due to the wide variety of our table schemas. While the fine-tuned GPT2 model generated sentences with a larger number of descriptive and inferred facts than the others on average, most of the facts were contradictive. The fine-tuned T5 model generated fewer sentences than GPT2, with the average number of inferred facts being larger than that of descriptive facts. Our model based on the fine-tuned T5 model with a copy mechanism reduced the ratio of contradictive facts for both descriptive and inferred facts. Following earlier work (Puduppully et al., 2019), we also evaluated text fluency in terms of grammaticality, coherence, and conciseness by using best-worst scaling (BWS) (Louviere and Woodworth, 1991; Louviere et al., 2015). We divided the outputs of the five models into ten pairs of descriptions. We presented workers with two descriptions and asked them to decide which one is best for each fluency category. The score of each model was calculated by using the MaxDiff approach (Orme, 2009): the number of times a description was chosen as the best minus the number of times it was chosen as the worst. Scores range from  100 (absolutely worst) to 100 (absolutely best). We elicited judgments with Amazon Mechanical Turk for the 30 descriptions, rated by 3 participants. The results are shown in Table 5. Most of the pre-trained models achieved better scores than the others. The fine-tuned GPT2 model achieved the highest score in terms of grammaticality and coherence. The fine-tuned T5 model achieved the highest score in terms of conciseness. Adding a copy mechanism to the T5 slightly decreased the grammaticality and conciseness but improved the coherence. We proposed numericNLG, a new dataset for table-to-text generation using a table and its corresponding description from scientific papers, focusing on numerical-reasoning texts. Even though our proposed dataset is not a large-scale table collection, we provided pairs of a table and its rich inference description, that are naturally written by experts in scientific papers, supporting further research on table-to-text generation with numerical reasoning. We conducted experiments with fine-tuned pre-trained models by using several types of table linearization as input representations, comparing with a template-based generator and pointer-generator. The experiments showed that transfer-learning of pre-trained language models leads to an improvement in our settings, that resulted in more fluent text while it still lacked fidelity to table contents. We then proposed incorporating a copy mechanism by using general placeholders to avoid the production of hallucinated phrases, that are not supported by tables while preserving high fluency. Even though our proposed copy mechanism failed to learn to generate better outputs in the decoder-only pre-trained models, we showed that a copy-based pre-trained model with an encoder-decoder architecture leads to a better BLEU score and improves correctness. Lya Hulliyyatus Suadaa is supported by the Indonesian Endowment Fund for Education (LPDP) and the Okumura-Takamura-Funakoshi Laboratory, Tokyo Institute of Technology. This work is partially supported by JST PRESTO (Grant Number JPMJPR1655). We thank the anonymous reviewers for their helpful discussion on this work and comments on the previous draft of the paper.
abstain abstain abstain objective abstain abstain abstain objective abstain abstain abstain abstain abstain result abstain abstain method objective objective objective other abstain method objective objective other other other other other objective abstain method abstain other abstain method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain method abstain abstain method other abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain other abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain objective objective method result objective objective other other other