In data-to-text (D2T) generation, training on in-domain data leads to overfitting to the data representation and repeating training data noise. We examine how to avoid finetuning pretrained language models (PLMs) on D2T generation datasets while still taking advantage of surface realization capabilities of PLMs. Inspired by pipeline approaches, we propose to generate text by transforming single-item descriptions with a sequence of modules trained on general-domain text-based operations: ordering, aggregation, and paragraph compression. We train PLMs for performing these operations on a synthetic corpus WIKIFLUENT which we build from English Wikipedia. Our experiments on two major triple-to-text datasetsWebNLG and E2Eshow that our approach enables D2T generation from RDF triples in zero-shot settings. 1 1 Introduction The aim of data-to-text (D2T) generation is to produce natural language descriptions of structured data (Gatt and Krahmer, 2018; Reiter and Dale, 1997). Although pipelines of rule-based D2T generation modules are still used in practice (Dale, 2020), end-to-end approaches based on PLMs recently showed superior benchmark performance (Ke et al., 2021; Chen et al., 2020a; Ferreira et al., 2020; Kale and Rastogi, 2020b; Ribeiro et al., 2020), surpassing pipeline systems (Ferreira et al., 2019) in both automatic and human evaluation metrics. Finetuning PLMs on human-written references is widely accepted as a standard approach for adapting PLMs to the D2T generation objective and achieving good performance on a given benchmark (Agarwal et al., 2021; Ke et al., 2021). However, finetuning a model on the domain-specific data leads to overfitting to the particular benchmark, decreasing performance on out-of-domain 1 Our code and data is available at https://github. data (Laha et al., 2019). Gathering a large set of references for a particular domain is also costly and time-consuming as it usually requires collecting human-written references through crowdsourcing (Duek et al., 2020). These problems can be partially mitigated using few-shot approaches (Chen et al., 2020b; Ke et al., 2021; Su et al., 2021a), which operate with only several dozens or hundreds of annotated examples, but the robustness of these approaches is questionableselecting a representative set of examples which would improve performance is difficult (Chang et al., 2021a), and the limited sample is often noisy, increasing the chance of hallucinations and omissions (Duek et al., 2019; Harkous et al., 2020; Rebuffel et al., 2022). In this paper, we present a zero-shot alternative to the traditional finetuning paradigm by formulating the D2T generation from RDF triples as a sequence of general-domain operations over text in natural language. We start by transforming individual triples to text using trivial templates, which 3914 we subsequently order, aggregate, and compress on the paragraph level to produce the resulting description of the data. In constrast to traditional pipeline systems, all our pipeline modules are built upon PLMs and operate over sentences in natural language. The modules are trained on our new WIKIFLUENT corpus, which contains 934k examples of first paragraphs from the English Wikipedia, each supplied with a synthesized set of simple template-like sentences which together convey the meaning of the original paragraph. Our approach allows generating natural language descriptions from RDF triples with a minimum amount of domain-specific rules or knowledge and without using training data from the D2T datasets. Although our approach is primarily a probe into the territory of zero-shot approaches and cannot yet match the quality of state-of-the-art models, we show that it can yield large improvements upon simple baselines and match older supervised systems on automatic metrics for text fluency. Moreover, the semantic accuracy metrics and our manual error analysis suggest that our approach offers a way to prevent omissions and hallucinations common in few-shot approaches. Our contributions are the following: (1) We propose an alternative D2T generation approach based on general-domain text-to-text operations (ordering, aggregation, and paragraph compression). (2) We introduce a synthetic WIKIFLUENT corpus containing 934k sentences based on English Wikipedia, providing training data for the operations in (1). (3) We apply our system on two D2T datasets and evaluate its performance both automatically and manually, including the contribution of individual pipeline modules. (4) We release our code, data, pretrained models, and system outputs to ease future research. 1 2 Related Work D2T Generation with PLMs Large neural language models pretrained on self-supervised tasks (Lewis et al., 2020; Liu et al., 2019; Devlin et al., 2019) have recently gained a lot of traction in D2T generation research (Ferreira et al., 2020; Kasner and Duek, 2020b). Following Chen et al. (2020b), other works adopted PLMs for few-shot D2T generation (Chang et al., 2021b; Su et al., 2021a). Kale and Rastogi (2020b) and Ribeiro et al. (2020) showed that PLMs using linearized representations of data can outperform graph neural networks on graph-to-text datasets, recently surpassed again by graph-based models (Ke et al., 2021; Chen et al., 2020a). Although the models make use of general-domain pretraining tasks, all of them are eventually finetuned on domain-specific data. Pipeline-based D2T Generation Until the recent surge of end-to-end approaches (Duek et al., 2020), using several modules connected in a pipeline was a major approach for D2T generation (Gatt and Krahmer, 2018; Reiter, 2007; Reiter and Dale, 1997). Our approach is inspired by the pipeline approaches, in particular the pipelines utilizing neural modules (Ferreira et al., 2019). In contrast with these approaches, our pipeline works with unstructured data in natural language and it operates in zero-shot setting, i.e. without using any training data from target D2T datasets. Laha et al. (2019) introduce a three-step pipeline for zero-shot D2T generation similar to ours. Unlike the approach we describe here, they use a semiautomatic template generation system, 2 their sentence fusion is rule-based, and they do not address content planning. Content Planning in D2T Generation Content planning, i.e. the task of ordering input facts and aggregating them into individual sentences, is one of the steps of the traditional D2T pipeline (Gatt and Krahmer, 2018). As shown by Moryossef et al. (2019a,b) and confirmed by other works (Pudup-pully et al., 2019; Zhao et al., 2020; Trisedya et al., 2020; Su et al., 2021b), including a content plan improves the quality of outputs in neural D2T pipelines. Unlike the aforementioned planners, which use predicates or keys from D2T datasets for representing the data items, our planner is trained on ordering sentences in natural language. Sentence Ordering Sentence ordering is the task of organizing a set of natural language sentences to increase the coherence of a text (Barzilay et al., 2001; Lapata, 2003). Several neural methods for this task were proposed, using either interactions between pairs of sentences (Chen et al., 2016; Li and Jurafsky, 2017), global interactions (Gong et al., 2016; Wang and Wan, 2019), or combination of both (Cui et al., 2020). We base our ordering module (5.2) on the recent work of Calizzano et al. 2 As we describe in 5.1, we opted for a simpler way for generating the templates to showcase the results of our approach independently of the template generator quality. Aggregating Input into Sentences Typically, multiple pieces of input information need to be merged into a single sentence. Previous works (Wiseman et al., 2018; Shao et al., 2019; Shen et al., 2020; Xu et al., 2021) capture the segments which correspond to individual parts of the input as latent variables. Unlike these works, we adopt a simpler scenario using an already ordered sequence of facts (see 3.1), into which we selectively insert delimiters to mark sentence boundaries. Paragraph Compression We introduce paragraph compression (PC) as a new task and the final step in our D2T generation pipeline. This task combines several standard natural-language tasks including sentence fusion, rephrasing, and coreference resolution. Unlike text summarization or simplification (Zhang et al., 2020; Jiang et al., 2020), we aim to convey the complete semantics of the text without omitting any facts. In contrast to sentence fusion (Geva et al., 2019; Barzilay and McKeown, 2005) or sentence compression (Filip-pova and Altun, 2013), we operate in the context of multiple sentences in a paragraph. The task is the central focus of our WIKIFLUENT corpus (4). In this section, we provide the formal description of our proposed approach. We focus on the task of producing a natural language description Y for a set of n RDF triples X  t x 1 , . . . , x n u . Each triple x i  t s i , p i , o i u consists of subject s i , predicate p i , and object o i . triples X on the input, we: (1) transform the triples into facts , which are sentences in natural language, (2) sort the facts using an ordering module, (3) insert sentence delimiters between the sorted facts using an aggregation module, (4) input the ordered sequence of facts with delimiters into a paragraph compression module, which generates the final description Y . The individual steps are described in the following sections: transforming individual triples to text (3.1), ordering (3.2), aggregation (3.3), and paragraph compression (3.4). The first step in our pipeline involves transforming each of the input triples x i P X into a fact f i P F using a transformation T : X  F . We define a fact f i as a single sentence in natural language describing x i . The transformation serves two purposes: (a) preparing the data for the subsequent text-to-text operations, (b) introducing in-domain knowledge about the semantics of individual predicates. This step can be realized e.g. using a simple template for each predicate (cf. 5.1). We assume that the default order of triples X is random and the same applies for the respective facts F . Note, however, that that F is a indeed set of meaningful sentences. We can use this to our advantage and apply a sentence ordering model to maximize the coherency of the paragraph resulting from their concatenation. An example outcome of such operation may be grouping together facts mentioning birth date and birth place of a person, followed by their occupation (see Figure 1). The ordering module allows downstream modules to only focus on operations over neighboring sentences. Formally, we apply the ordering model O p F q to get an ordered sequence of facts: F o  t f o 1 , . . . , f o n u , where o 1: n is a permutation of indices. We describe our ordering model in 5.2. Some facts will be typically mentioned together in a single sentence. Considering the previous example, occupation is likely to be mentioned separately, while birth date and birth place are likely to be mentioned together. Using an ordered sequence of facts as input, we can apply an aggregation model to decide which facts should be merged into a single sentence. Formally, the aggregation model takes a sequence of ordered facts F o as input and produces a sequence of sentence delimiters A p F o q  t  o 1 ,  o 2 , . . . ,  o n  1 u ;  i P t 0 , 1 u . The output  i  1 means that the neighboring facts should be mentioned separately, i.e. the neighboring sentences should not be fused. Conversely,  i  0 means that the facts should be aggregated and their corresponding sentences should be fused. We describe our aggregation model in 5.3. The paragraph compression (PC) model is a generative model which outputs the final text description. It has two main objectives: (a) fusing related sentences, i.e., sentences i and j in between which  i  0 , and (b) rephrasing the text to improve its fluency, e.g. fixing disfluencies in the templates, replacing noun phrases with refering expressions, etc. The goal of the task is to preserve the semantics of the text which is an already ordered sequence of sentences, so the edits will typically be minor. Formally, the model takes as input the ordered sequence of facts with delimiters F a  t f o 1 ,  o 1 , f o 2 , . . . ,  o n  1 , f o n u and produces the final text Y . We describe our PC model in 5.4. 4 WIKIFLUENT Corpus Here we descibe the process of building a large-scale synthetic corpus WIKIFLUENT . The corpus provides training data for the neural models which we use in our implementation of the ordering, aggregation, and paragraph compression modules (cf. 5). Our goal is to cover a broad range of domains while capturing the sentence style in D2T generation with respect to both the input facts and the target descriptions. In other words, we aim to build a corpus in which (1) the input is a set of simple, template-like sentences, (2) the output is a fluent text in natural language preserving the semantics of the input. As we describe below in detail, we achieve that by using human-written paragraphs in English Wikipedia and applying split-and-rephrase and coreference resolution models to obtain synthetic source texts. The process is illustrated in Figure 2; corpus statistics are included in Appendix A. 4.1 Data Source For building the WIKIFLUENT corpus, we extracted 934k first paragraphs of articles from a Wikipedia dump 3 using WikiExtractor (Attardi, 2015). Wikipedia is commonly used for large-scale pretraining of D2T generation models (Jin et al., 2020; Chen et al., 2020a). Although it is not bias-free, it provides more balanced sample of natural language use than typical D2T generation datasets. We used the first paragraphs of Wikipedia entries, which contain mostly concise, fact-based descriptions. We selected paragraphs with length between 3 enwiki-20210401-pages-articles-multistream The Westmeath Examiner is a weekly newspaper in Westmeath, Ireland. It is located in Westmeath, Ireland. The Westmeath Examiner is a weekly newspaper. original paragraph The Westmeath Examiner is a weekly newspaper. It was founded in 1882. It was founded in 1882. split-and-rephrase coreference replacement The Westmeath Examiner is located in Westmeath, Ireland. The Westmeath Examiner was founded in 1882. processed paragraph split successful pronounsresolved Figure 2: The building process of the WIKIFLUENT corpus. We apply a split-and-rephrase model on each sentence in the paragraph and resolve coreferences in the split sentences. The result is a set of simple sentences which together convey the same meaning as the original paragraph. The synthesized sentences are used as input into our models, the original human-written texts are used as ground truth . 30-430 characters; filtering out lists, disambiguations, and repeated and malformed paragraphs. To balance the length of inputs, we selected 250k examples each from 4 equally sized length ranges (30-130 characters, etc.). 4.2 Split-and-Rephrase To generate a set of simple sentences, we divide each paragraph into sentences using NLTK (Bird, 2006) and apply a split-and-rephrase model on each sentence. Split-and-rephrase is a task of splitting a complex sentence into a meaning preserving sequence of shorter sentences (Narayan et al., 2017). The process is illustrated in the upper part of Figure 2. We train our split-and-rephrase model on the large-scale WikiSplit corpus by Botha et al. (2018), containing human-made sentence splits from Wikipedia edit history. Following the same setup as for a paragraph compression model (3.4), we train BART-base (Lewis et al., 2020) on the WikiSplit dataset in a sequence-to-sequence setting. Next, we apply the trained split-and-rephrase model on each sentence in our Wikipedia-based corpus, uniformly randomly choosing between 0-2 recursive calls to ensure that the splits are not deterministic. If the sentence cannot be meaningfully split, the model tends to duplicate the sentence on the output; in that case, we use only the original sentence and do not proceed with the splitting. 3917 4.3 Coreference Replacement As the next step, we concatenate the split sentences and apply a coreference resolution model (Gardner et al., 2018; Lee et al., 2018) in order to replace referring expressions with their antencendents (e.g., pronouns with noun phrases). The motivation for this step is to match the style of the facts (see 3.1), which do not use pronouns since each fact describes a single triple only. Note that this procedure replaces the referring expressions only in the synthesized sentences (which are used as input) and keeps them in the original paragraphs (which are used as ground truth). As a consequence, the paragraph compression module is implicitly trained to generate referring expressions in the final description. 4.4 Filtering To ensure that the generated sentences convey the same semantics as the original paragraph, we use a pretrained RoBERTa model 4 (Liu et al., 2019) trained on the MultiNLI dataset (Williams et al., 2018) for checking the semantic accuracy of the generated text. Following Duek and Kasner (2020), we test if the original paragraph entails each of the synthesized sentences (checking for omissions), and if the set of concatenated synthesized sentences entails the original paragraph (checking for hallucinations). In a filtered version of the WIKIFLUENT corpus, we include only the examples without omissions or hallucinations (as computed by the model), reducing it to 714k examples (approximately 75% of the original size). 5 Implementation In this section, we describe how we implement our pipeline modules (3) using simple template transformations (5.1) and neural models trained on the WIKIFLUENT dataset (5.2-5.4). 5 5.1 Templates We transform triples into facts (3.1) using a single-triple template t i for each predicate. For example, if p i  instrument , then T p p i q   s i plays o i  (cf. Table 1). We follow previous work in which simple hand-crafted templates have been used as an efficient way of introducing domain knowledge (Kale and Rastogi, 2020a; Kasner and Duek, 2020a). Compared to more complex rule-based 4 https://huggingface.co/roberta-large-mnli 5 Our training setup details are included in Appendix C. dataset predicate template WebNLG instrument < s > plays < o >. template generation engines (Laha et al., 2019; Hei-dari et al., 2021; Mehta et al., 2021), the approach may produce less fluent outputs, but it minimizes manual workload and makes it easier to control the quality of the input for the subsequent steps. For our ordering model (3.2), we use the Simple Pointer model from Calizzano et al. (2021). The model is based on a pretrained BART-base extended with a pointer network from Wang and Wan (2019). We provide a short description of the model here; for details please refer to Calizzano et al. (2021). In the encoding phase, facts F are concatenated and tokenized. Each fact is surrounded by special tokens denoting the beginning ( <s> ) and the end ( </s> ) of the fact. The sequence is processed by the BART encoder, generating a sequence of encoder states E for each end token </s> representing the preceding fact. The decoding proceeds autoregressively. To bootstrap the decoding process, the pair of tokens <s></s> is fed into the decoder, producing the decoder state d 1 . The pointer network (attend-ing to d 1 and E ), selects the first ordered fact f o 1 , which is fed into the decoder in the next step ( d 2  <s> f o 1 </s> ). The process is repeated until the all the facts are decoded in a particular order. The pointer network computes the probability of a fact to be on the j -th position, using the encoder output E and the decoder output state d j . The network is based on the scaled dot product attention, where d j is the query and encoder outputs E i are the keys: Q  d j WQK  EWKP j  softmax  QKT ? b  . 3918 A dam is a barrier obstructing flowing water. A dam is a barrier. 3-stage 2-stage 1-stage A dam obstructs flowing water. src tgt a agg ord PC + PC+agg PC+ord+agg b b a b b a a Figure 3: An example illustrating how the individual modules are trained and subsequently applied as the parts of the pipeline. See 5.2 for description of the ordering model ( ORD ), 5.3 for the aggregation model ( AGG ), and 5.4 and 6 for the paragraph compression model (PC, PC+ AGG , PC+ ORD + AGG ). Here WQ and WKP R b  b , b is the dimension of BART hidden states, and P j P R n ` 1 is the probability distribution for the j -th position (i.e., P ji is the probability that fact f i is on the j -th position). We train the model using the synthesized simple sentences in the WIKIFLUENT corpus, randomly shuffling the order of the sentences and training the model to restore their original order. 5.3 Aggregation Model We base our aggregation model (3.3) on RoBERTa-large (Liu et al., 2019) with a token classification head. 6 Similarly to the ordering model (5.2), we input the sequence of (now ordered) facts F o into the model, separating each pair of facts f o i with a special token </s> (used by the model as a separator). Subsequently, the token classification layer classifies each separator </s> i position into two classes t 0 , 1 u corresponding to the delimiter  i . We ignore the outputs for the non-separator tokens while computing cross-entropy loss. We create the training examples using the synthesized sentences in the WIKIFLUENT corpus, in which we set  i  0 for the sentences i, i ` 1 which were originally aggregated (i.e., are the result of splitting a single sentence) and  i  1 otherwise. 5.4 Paragraph Compression Model We adopt BART-base for our paragraph compression model. We finetune the model on the WIKIFLUENT corpus, concatenating the synthesized sentences on the input. We add delimiters between the sentences i and i ` 1 where  i  1 using a special token <sep> , which we add to the model vocabulary. As shown in Keskar et al. (2019), including control codes for training the model can steer the model towards producing certain outputs. Here we expect that the model will learn to fuse the sentences between which there are no delimiters 6 https://huggingface.co/transformers/model_ doc/roberta.html#robertafortokenclassification on the input. We evaluate how the model learns to respect the order and aggregation markers in 7.3. 6 Experiments We train our pipeline modules on the WIKIFLUENT corpus as described in 5. Next, we use these modules without finetuning for generating descriptions for RDF triples on two English D2T datasets, WebNLG and E2E. Datasets The datasets differ in domain, size, textual style, and number of predicates (see Appendix A for details):  WebNLG (Gardent et al., 2017; Ferreira et al., 2020) contains RDF triples from DBPedia (Auer et al., 2007) and their crowdsourced descriptions. We use version 1.4 of the dataset for comparison to prior work. We hand-crafted templates for all 354 predicates, including unseen predicates in the test set. 7  E2E (Novikova et al., 2017; Duek et al., 2020) contains restaurant recommendations in the form of attribute-value pairs. We use the cleaned version of the dataset (Duek et al., 2019). Following previous work, we transform the attribute-value pairs into RDF triples (using the restaurant name as a subject) and then apply the same setup as for WebNLG. We created a template for each of the 8 attributes manually. Pipeline versions In order to evaluate individual components of our pipeline, we train three versions of the paragraph compression model (see 5.4). The models share the same architecture and targets, but differ in their inputs:  PC  the model takes as an input ordered facts with delimiters (as described in 3.4),  PC+ AGG  the model takes as an input ordered facts without delimiters (i.e., the aggregation is left implicitly to the model),  PC+ ORD + AGG  the model takes as an input facts in random order and without delimiters 7 See Appendix B for details on template creation. 3919 (i.e., both ordering and aggregation are left implicitly to the model). Correspondingly, we test three versions of the pipeline in our ablation study (see Figure 3):  3STAGE  a full version of the pipeline consisting of the ordering model ( ORD ), the aggregation model ( AGG ) and the PC model (following the full pipeline from 3),  2STAGE  a pipeline consisting of the ORD model and the PC+ AGG model,  1STAGE  a single stage consisting of the PC+ ORD + AGG model. We evaluate all versions of the pipeline with PC models trained on the full and filtered versions of the WIKIFLUENT dataset (see 4). 7 Evaluation and Discussion Our main aim is the evaluation of our pipeline on the downstream task of D2T generation. We evaluate outputs from the {1,2,3}STAGE variants of our pipeline using automatic metrics (7.1), and we perform a detailed manual error analysis of the model outputs (7.2). We also evaluate the performance of the content planning modules and the ability of the PC module to follow the content plan (7.3). In 7.4, we include an intrinsic evaluation of our modules on the WIKIFLUENT test set. 7.1 Automatic Metrics Following prior work, we use BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) to evaluate the outputs against the human references. 8 We also evaluate the number of omission and hallucination errors (i.e., facts missing or added, respectively) using a metric from Duek and Kasner (2020) based on a RoBERTa model (Liu et al., 2019) pretrained on natural language inference (NLI). 9 We include a diverse set of baselines for comparison. For WebNLG (see Table 3), we compare our systems with the results of:  UPF-FORGe and MELBOURNE  systems (grammar-based and supervised, respectively) from the first run of WebNLG Challenge (Gar-dent et al., 2017),  Ke et al. (2021)  a state-of-the-art system with 8 We use the implementation from https://github. com/tuetschek/e2e-metrics . 9 We additionally evaluated the outputs on the E2E dataset using the provided pattern-based slot error script. See Appendix D for details. a structure-aware encoder and task-specific pretraining,  Laha et al. (2019)  the only other (to our knowledge) zero-shot D2T generation system applied to WebNLG. For E2E (see Table 4), we compare our systems with the results of:  TGEN (Duek and Jurccek, 2015)  the baseline system for the E2E Challenge (Duek et al., 2020),  Harkous et al. (2020)  a state-of-the-art supervised system on cleaned E2E data. For both datasets, COPY denotes the baseline of copying the facts without further processing. The automatic evaluation shows that our systems consistently outperform the COPY baseline (e.g.,  12 BLEU points for E2E), which is already strong thanks to our manually curated set of templates. 10 Automatic scores also suggest that our systems are comparable with some older supervised systems. Nevertheless, our systems still un-derperform the state-of-the-art supervised systems. For this reason, we further focus on manual error analysis in 7.2 to pinpoint the current shortcomings of our approach. The 2STAGE system is generally on par with the 3STAGE system or better, which indicates that explicit aggregation using the AGG model may not be necessary. However, an advantage of having a separate aggregation module is the possibility to control the aggregation step explicitly. The models using the filtered version of the corpus generally produce better results, although they also bring in a larger number of omissions. 7.2 Manual Error Analysis Since automatic performance metrics do not provide insights into specific weaknesses of the system (van Miltenburg et al., 2021), we manually examined 100 outputs of the models. We counted the number of errors: factual (hallucinations, omissions, incorrect fact merging, redundancies) and grammatical. The results are summarized in Table 5. The 1STAGE model (which has to order the facts implicitly) tends to repeat the facts in the text (es-pecially in E2E) and produces frequent hallucinations. These problems are largely eliminated with the 2STAGE and 3STAGE models, which produce 10 On WebNLG, our COPY baseline achieves 37.18 BLEU points, compared to 24.80 BLEU points of the full system of Laha et al. (2019), which uses automatic template generation. 3920 Input (Allen Forrest; background; solo singer), (Allen Forrest; genre; Pop music), (Allen Forrest; birthPlace; Dothan, Alabama) Templ. Allen Forrest is a solo singer. Allen Forrest performs Pop music. Allen Forrest was born in Dothan, Alabama. Model Allen Forrest is a solo singer who performs Pop music. He was born in Dothan, Alabama. Human Born in Dothan, Alabama, Allen Forrest has a background as a solo singer and was a pop artist. Input name[Wildwood], eatType[restaurant], food[French], area[riverside], near[Raja Indian Cuisine] Templ. Wildwood is a restaurant. Wildwood serves French food. Wildwood is in the riverside. Wildwood is near Raja Indian Cuisine. Model Wildwood is a restaurant serving French food. It is in the riverside near Raja Indian Cuisine. Human A amazing French restaurant is called the Wildwood. The restaurant is near the Raja Indian Cuisine in riverside. They love kids. Table 2: Example outputs of our model (3STAGE , filtered). See Appendix E for more examples. B M O H UPF-FORGe  38.65 39.00 0.075 0.101 MELBOURNE  45.13 37.00 0.237 0.202 Ke et al. (2021) : 66.14 47.25 -Laha et al. (2019) : 24.80 34.90 --COPY 37.18 38.77 0.000 0.000 full 3STAGE 42.92 39.07 0.051 0.148 2STAGE 42.90 39.28 0.043 0.125 1STAGE 39.08 38.94 0.071 0.204 filtered 3STAGE 43.19 39.13 0.152 0.073 2STAGE 43.49 39.32 0.146 0.096 1STAGE 42.99 38.81 0.202 0.093 Table 3: Automatic metrics on WebNLG. B = BLEU, M = METEOR, O = omissions / # facts, H = hallucinations / # examples. The systems marked with asterisk (*) are trained on the WebNLG dataset. Results for the systems marked with : are taken from the respective works. almost no hallucinations or omissions. However, the outputs on WebNLG for all systems suffer from semantic errors resulting from merging of unrelated facts. This mostly happens with unrelated predicates connected to the same subject/object (e.g. X was born in Y, X worked as Z expressed as X worked as Z in Y; see Appendix E for examples). This behavior is the main obstacle to ensure factual consistency of the output. As a possible remedy, we propose explicitly controlling the semantics of sentence fusion (Ben-David et al., 2020), e.g. using a variant of constrained decoding (Balakrishnan et al., 2019; Wang et al., 2021). On the E2E data, which has a simpler triple structure (all predicates share the same subject), the outputs are generally consistent and the 2STAGE and 3STAGE models exhibit almost no semantic errors. Grammar errors and disfluencies stem mainly from over-eager paragraph compression or from artifacts in our templates and are relatively minor (e.g., missing is in serves French food and B M O H TGEN  40.73 37.76 0.016 0.083 Harkous et al. (2020)  43.60 39.00 --COPY 24.19 34.89 0.000 0.000 full 3STAGE 36.04 36.95 0.001 0.001 2STAGE 35.84 36.91 0.001 0.001 1STAGE 30.81 36.01 0.009 0.122 filtered 3STAGE 35.88 36.95 0.001 0.001 2STAGE 36.01 36.99 0.001 0.001 1STAGE 34.08 36.32 0.012 0.050 Table 4: Automatic metrics on E2E. B = BLEU, M = METEOR, O = omissions / # facts, H = hallucinations / # examples. The systems marked with asterisk (*) are trained on the E2E dataset. The results for Harkous et al. (2020) are taken from their work. WebNLG E2E H I O R G H I O R G f u ll 3STAGE 3 39 2 2 16 0 1 0 0 17 2STAGE 8 36 1 5 16 1 1 0 1 23 1STAGE 28 27 6 10 20 17 0 1 79 45 fi lt e r e d 3STAGE 2 37 2 1 15 0 0 0 0 17 2STAGE 5 32 1 2 14 0 0 0 0 11 1STAGE 8 40 6 6 16 11 2 1 41 22 Table 5: Number of manually annotated errors on 100 examples: H = hallucinations, I = incorrect fact merging, O = omissions, R = redundancies, G = grammar errors or disfluencies. family-friendly). Following Su et al. (2021b) and Zhao et al. (2020), we report the accuracy and BLEU-2 score of our ordering model on WebNLG against the human-generated plans from Ferreira et al. (2018). The results are listed in Table 6 and compared against a RANDOM baseline (random ordering) and prior work. The results show that although our approach again lags behind state-of-the-art supervised ap-3921 B-2 Acc Transformer (Ferreira et al., 2019) : 52.20 0.35 Step-by-step (Moryossef et al., 2019b) : 70.80 0.47 PLANENC (Zhao et al., 2020) : 80.10 0.62 Plan-then-generate (Su et al., 2021b) : 84.97 0.72 RANDOM 47.00 0.29 Ours (BART+ptr) 59.10 0.48 Table 6: Evaluation of our zero-shot ordering model based on Calizzano et al. (2021). proaches, it can outperform both the random baseline and the Transformer-based approach from Ferreira et al. (2019) while not using any in-domain examples. We also evaluate the accuracy of our aggregation model , using triples ordered according to the plans from Ferreira et al. (2018) as input. The accuracy is 0.33 per example and 0.62 per sentence boundary (random baseline is 0.23 and 0.50, re-spectively). The results show that although our approach is better than the random baseline, there is still room for improvement. Finally, we manually evaluate how the PC model follows the content plan (i.e., keeping the predefined order and aggregating the sentences according to the delimiters) using 100 randomly cho-sen examples with more than 1 triple on WebNLG and E2E. We find that the model follows the content plan in 95% and 100% of cases, respectively. The incorrect cases include a fact not properly mentioned or an extra boundary between sentences without a separator. We can thus conclude that the pretraining task successfully teaches the PC model to follow a given content plan. Aside from the main D2T generation results, we also provide an intrinsic evaluation of our pipeline modules on the WIKIFLUENT test sets. We evaluated the ordering, aggregation, and paragraph compression modules trained on the full WIKIFLUENT corpus. The results for both full and filtered test sets are summarized in Table 7. The PC model achieves high scores, which follows from the fact that we provide it with ground truth content plans (i.e., the ordering and aggregation plan corresponding to the original paragraph). Accuracy of the ordering and aggregation modules is comparable to their performance on D2T datasets. Our experiments outline several possible future research directions. Automatic generation of facts without using hand-crafted templates (cf. 5.1) could allow applying zero-shot generation systems to datasets with a large number of predicates, such as ToTTo (Parikh et al., 2020). The task of paragraph compression could be used as a task-specific pretraining (Gururangan et al., 2020) for more efficient finetuning of D2T models, e.g., with a small amount of clean data. Consistency checks may be introduced in the pipeline to control the output from the modules, and individual modules may be improved by using more efficient model architectures. More research is also needed regarding the main shortcoming of our approach, i.e., the semantic errors stemming from merging of facts in improper ways. As we suggested in 7.2, explicitly controlling the semantics of sentence fusion could help to mitigate this issue, while still keeping the advantages of a zero-shot approach. We presented an approach for zero-shot D2T generation. The approach uses a pipeline of PLMs trained on general-domain lexical operations over natural language. The pipeline builds upon traditional approaches and consists of three interpretable intermediate steps. By avoiding noisy human-written references from the D2T datasets, our models produce more semantically consitent output. We believe that training models for zero-shot D2T generation using large cross-domain corpora will help to build D2T generation systems with good performance across various domains. We study zero-shot D2T generation with the focus on generating descriptions for RDF triples. Although the task of D2T generation has numerous applications, using neural models for D2T generation (especially in the zero-shot context) is still limited to experimental settings (Dale, 2020). Similarly to other recent approaches for D2T generation, our approach relies on PLMs, which are known to reflect the biases in their pretraining corpus (Ben-der et al., 2021; Rogers, 2021). Our system may therefore rely on spurious correlations for verbalizing e.g. gender or occupation of the entities. Since we cannot guarantee the factual correctness of the outputs of our system, the outputs should be used with caution. On the flip side, our approach helps to reduce the number of omissions and hallucinations stemming from noise in human-written references. Our work thus contributes to the general aim of D2T generation in conveying the data semantics accurately and without relying on implicit world knowledge. This research was supported by Charles University projects GAUK 140320, SVV 260575 and PRIMUS/19/SCI/10, an Apple NLU Research Grant for Heriot-Watt University and Charles University, and by the European Research Council (Grant agreement No. 101039303 NG-NLG). It used resources provided by the LINDAT/CLARIAH-CZ Research Infrastructure (Czech Ministry of Education, Youth and Sports project No. LM2018101).
abstain method objective method abstain abstain other abstain other abstain abstain other method result method objective method result abstain objective method objective abstain objective abstain abstain abstain abstain method objective abstain method abstain abstain method abstain abstain result abstain abstain result objective abstain abstain abstain abstain method method abstain method abstain method method abstain abstain abstain abstain method abstain method abstain abstain method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method method other abstain method abstain method abstain abstain abstain abstain abstain other abstain abstain abstain other method abstain other abstain method abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method method method abstain abstain abstain method method abstain other other