We present AGGGEN (pronounced again' ), a data-to-text model which re-introduces two explicit sentence planning stages into neural data-to-text systems: input ordering and input aggregation. In contrast to previous work using sentence planning, our model is still end-to-end: AGGGEN performs sentence planning at the same time as generating text by learning latent alignments (via semantic facts) between input representation and target text. Experiments on the WebNLG and E2E challenge data show that by using fact-based alignments our approach is more interpretable, expressive, robust to noise, and easier to control, while retaining the advantages of end-to-end systems in terms of fluency. Our code is available at https://github.com/XinnuoXu/ AggGen . Recent neural data-to-text systems generate text end-to-end (E2E) by learning an implicit mapping between input representations (e.g. RDF triples) and target texts. While this can lead to increased fluency, E2E methods often produce repetitions, hallucination and/or omission of important content for data-to-text (Du  sek et al., 2020) as well as other natural language generation (NLG) tasks (Cao et al., 2018; Rohrbach et al., 2018). Traditional NLG systems, on the other hand, tightly control which content gets generated, as well as its ordering and aggregation. This process is called sentence planning (Reiter and Dale, 2000; Duboue and McKeown, 2001, 2002; Konstas and Lapata, 2013; Gatt and Krahmer, 2018). Figure 1 shows two different ways to arrange and combine the representations in the input, resulting in widely different generated target texts. In this work, we combine advances of both paradigms into a single system by reintroducing William Anders dateOfRetirement 1969-09-01 Apollo 8 commander Frank Borman William Anders member_of Apollo 8 Apollo 8 backup_pilot Buzz Aldrin Apollo 8 operator NASA Input DBpedia Triples William Anders served as a crew member on Apollo 8 operated by nasa. The backup pilot was Buzz Aldrin. Frank Borman was also an Apollo 8 commander. William Anders retired on September 1st, 1969. William Anders retired on 1969-09-01. He was a crew member of nasa 's Apollo 8. Frank Borman was also a commander with Buzz Aldrin as the backup pilot. operator backup_pilot commander dateOfRetirement dateOfRetirement operator backup_pilot commander dateOfRetirement commander member_of backup_pilot operator member_of member_of William Anders, who retired on September 1st, 1969, was a crew member on Apollo 8 and served under commander Frank Borman. Apollo 8 was operated by NASA with Buzz Aldrin as backup pilot. Human-authored Text Sentence Plan 1: Generated Target Text 1: Sentence Plan 2: Generated Target Text 2: Figure 1: Two different sentence plans with their corresponding generated target texts from our model on the WebNLG dataset. sentence planning into neural architectures. We call our system AGGGEN (pronounced again' ). AGGGEN jointly learns to generate and plan at the same time. Crucially, our sentence plans are interpretable latent states using semantic facts 1 (ob-tained via Semantic Role Labelling (SRL)) that align the target text with parts of the input representation. In contrast, the plan used in other neural plan-based approaches is usually limited in terms of its interpretability, control, and expressivity. For example, in (Moryossef et al., 2019b; Zhao et al., 2020) the sentence plan is created independently, incurring error propagation; Wiseman et al. (2018) use latent segmentation that limits interpretability; Shao et al. (2019) sample from a latent variable, not allowing for explicit control; and Shen et al. (2020) aggregate multiple input representations which limits expressiveness. 1 Each fact roughly captures who did what to whom. target text, using a separate inference algorithm based on dynamic programming. Crucially, this enables us to directly evaluate and inspect the model's planning and alignment performance by comparing to manually aligned reference texts. We demonstrate this for two data-to-text generation tasks: the E2E NLG (Novikova et al., 2017) and the WebNLG Challenge (Gardent et al., 2017a). We work with a triple-based semantic representation where a triple consists of a subject, a predicate and an object. 2 For instance, in the last triple in Figure 1, Apollo 8 , operator and NASA are the subject, predicate and object respectively. Our contributions are as follows:  We present a novel interpretable architecture for jointly learning to plan and generate based on modelling ordering and aggregation by aligning facts in the target text to input representations with an HMM and Transformer encoder-decoder. We show that our method generates output with higher factual correctness than vanilla encoder-decoder models without semantic information. We also introduce an intrinsic evaluation framework for inspecting sentence planning with a rigorous human evaluation procedure to assess factual correctness in terms of alignment, aggregation and ordering performance. Factual correctness is one of the main issues for data-to-text generation: How to generate text according to the facts specified in the input triples without adding, deleting or replacing information? The prevailing sequence-to-sequence (seq2seq) architectures typically address this issue via rerank-ing (Wen et al., 2015a; Dusek and Jurccek, 2016; Juraska et al., 2018) or some sophisticated training techniques (Nie et al., 2019; Kedzie and McKeown, 2019; Qader et al., 2019). For applications where structured inputs are present, neural graph encoders (Marcheggiani and Perez-Beltrachini, 2018; Rao et al., 2019; Gao et al., 2020) or decoding of explicit graph references (Logan et al., 2019) are applied for higher accuracy. Recently, large-scale pretraining has achieved SoTA results on WebNLG by fine-tuning T5 (Kale and Rastogi, 2020). Several works aim to improve accuracy and controllability by dividing the end-to-end architecture into sentence planning and surface realisation . Castro Ferreira et al. (2019) feature a pipeline with multiple planning stages and Elder et al. (2019) introduce a symbolic intermediate representation in multi-stage neural generation. Moryossef et al. (2019b,a) use pattern matching to approximate the required planning annotation (entity mentions, their order and sentence splits). Zhao et al. (2020) use a planning stage in a graph-based model  the graph is first reordered into a plan; the decoder conditions on both the input graph encoder and the linearized plan. Similarly, Fan et al. (2019) use a pipeline approach for story generation via SRL-based sketches. However, all of these pipeline-based approaches either require additional manual annotation or depend on a parser for the intermediate steps. Other works, in contrast, learn planning and realisation jointly. For example, Su et al. (2018) introduce a hierarchical decoding model generating different parts of speech at different levels, while filling in slots between previously generated tokens. Puduppully et al. (2019) include a jointly trained content selection and ordering module that is applied before the main text generation step.The model is trained by maximizing the log-likelihood of the gold content plan and the gold output text. Li and Rush (2020) utilize posterior regularization in a structured variational framework to induce which input items are being described by each token of the generated text. Wiseman et al. (2018) aim for better semantic control by using a Hidden Semi-Markov Model (HSMM) for splitting target sentences into short phrases corresponding to templates, which are then concatenated to produce the outputs. However it trades the controllability for fluency. Similarly, Shen et al. (2020) explicitly segment target text into fragment units, while aligning them with their corresponding input. Shao et al. (2019) use a Hierarchical Variational Model to aggregate input items into a sequence of local latent variables and realize sentences conditioned on the aggregations. The aggregation strategy is controlled by sampling from a global latent variable. In contrast to these previous works, we achieve input ordering and aggregation, input-output alignment and text generation control via interpretable states , while preserving fluency. We jointly learn to generate and plan by aligning facts in the target text with parts of the input representation. We model this alignment using a Hidden Markov Model (HMM) that follows a hierarchical structure comprising two sets of latent states, corresponding to ordering and aggregation. The model is trained end-to-end and all intermediate steps are learned in a unified framework. Let x = { x 1 , x 2 , . . . , x J } be a collection of J input triples and y their natural language description (human written target text). We first segment y into a sequence of T facts y 1: T = y 1 , y 2 , . . . , y T , where each fact roughly captures who did what to whom in one event. We follow the approach of Xu et al. (2020), where facts correspond to predicates and their arguments as identified by SRL (See Appendix B for more details). For example: William Anders, who retired in 1969, was a crew member on Apollo 8. Each fact y t consists of a sequence of tokens y 1 t , y 2 t , . . . , y N t t . Unlike the text itself, the planning information, i.e. input aggregation and ordering, is not directly observable due to the absence of labelled datasets. AGGGEN therefore utilises an HMM probabilistic model which assumes that there is an underlying hidden process that can be modeled by a first-order Markov chain. At each time step, a latent variable (in our case input triples) is responsible for emitting an observed variable (in our case a fact text segment). The HMM specifies a joint distribution on the observations and the latent variables. Here, a latent state z t emits a fact y t , representing the group of input triples that is verbalized in y t . We write the joint likelihood as: p ( z 1: T , y 1: T | x ) = p ( z 1: T | x ) p ( y 1: T | z 1: T , x ) = (cid:34) p ( z 1 | x ) T (cid:89) t =2 p ( z t | z t  1 , x ) (cid:35) (cid:34) T (cid:89) t =1 p ( y t | z t , x ) (cid:35) . i.e., it is a product of the probabilities of each latent state transition ( transition distribution ) and the probability of the observations given their respective latent state ( emission distribution ). Latent State. A latent state z t represents the input triples that are verbalized in the observed fact y t . It is not guaranteed that one fact always verbalizes only one triple (see bottom example in Figure 1). Thus, we represent state z t as a sequence of latent variables o 1 t , . . . , o L t t , where L t is the number of triples verbalized in y t . Figure 2 shows the structure of the model. Let o lt  Q = { 1 , . . . , K } be a set of possible latent variables, then KL t is the size of the search space for z t . If o lt maps to unique triples, the search space becomes intractable for a large value of K . To make the problem tractable, we decrease K by representing triples by their predicate. Q thus stands for the collection of all predicates appearing in the corpus. To reduce the search space for z t further, we limit L t < L , where L = 3 . 3 Transition Distribution. distribution between latent variables (T1 in Figure 2) is a K  K matrix of probabilities, where each row sums to 1. We define this matrix as p (cid:16) o lt | o ( l  1) t , x (cid:17) = softmax ( AB (cid:12) M ( q )) (1) where (cid:12) denotes the Hadamard product. A  RK  m and B  R m  K are matrices of predicate embeddings with dimension m . q = { q 1 , q 2 , . . . , q J } is the set of predicates of the input triples x , and each q j  Q is the predicate of the triple x j . M ( q ) is a K  K masking matrix, where M ij = 1 if i  q and j  q , otherwise 3 By aligning the triples to facts using a rule-based aligner (see Section 5), we found that the chance of aggregating more than three triples to a fact is under 0.01% in the training set of both WebNLG and E2E datasets. M ij = 0 . We apply row-wise softmax over the resulting matrix to obtain probabilities. The probability of generating the latent state z t (T2 in Figure 2) can be written as the joint distribution of the latent variables o 1 t , . . . , o L t t . Assuming a first-order Markov chain, we get: p ( z t | x ) = p (cid:16) o 0 t , o 1 t , o 2 t , . . . , o L t t | x (cid:17) = p (cid:0) o 0 t | x (cid:1) (cid:34) L t (cid:89) l =1 p (cid:16) o lt | o ( l  1) t , x (cid:17)(cid:35) , where o 0 t is a marked start-state. On top of the generation probability of the latent states p ( z t | x ) and p ( z t  1 | x ) , we define the transition distribution between two latent states (T3 in Figure 2) as: p ( z t | z t  1 , x ) = p (cid:16) o 0( t  1) , . . . , o L t  1 ( t  1) | x (cid:17)  p (cid:16) o 1 t | o L t  1 ( t  1) , x (cid:17)  p (cid:16) o 0 t , . . . , o L t t | x (cid:17) , where o L t  1 ( t  1) denotes the last latent variable in latent state z t  1 , while o t 1 denotes the first latent variable (other than the start-state) in latent state z t . We use two sets of parameters { A in , B in } and { A out , B out } to describe the transition distribution between latent variables within and across latent states, respectively. Emission Distribution. The emission distribution p ( y t | z t , x ) (T4 in Figure 2) describes the generation of fact y t conditioned on latent state z t and input triples x . We define the probability of generating a fact as the product over token-level probabilities, p ( y t | z t , x ) = p ( y 1 t | z t , x ) N t (cid:89) i =2 p ( y it | y 1:( i  1) t , z t , x ) . The first and last token of a fact are marked fact-start and fact-end tokens. We adopt Transformer (Vaswani et al., 2017) as the model's encoder and decoder. Each triple is linearized into a list of tokens following the order: subject, predicate, and object. In order to represent individual triples, we insert special [SEP] tokens at the end of each triple. A special [CLS] token is inserted before all input triples, representing the beginning of the entire input. An example where the encoder produces a contextual embedding for the tokens of two input triples is shown in Figure 6 in Appendix E. At time step t , the decoder generates fact y t token-by-token autoregressively, conditioned on both the contextually-encoded input and the latent state z t . To guarantee that the generation of y t conditions only on the input triples whose predicate is in z t , we mask out the contextual embeddings of tokens from other unrelated triples for the encoder-decoder attention in all Transformer layers. Autoregressive Decoding. Autoregressive Hidden Markov Model (AR-HMM) introduces extra links into HMM to capture long-term correlations between observed variables, i.e., output tokens. Following Wiseman et al. (2018), we use AR-HMM for decoding, therefore allowing the interdependence between tokens to generate more fluent and natural text descriptions. Each token distribution depends on all the previously generated tokens, i.e., we define the token-level probabilities as p ( y it | y 1: N t 1:( t  1) , y 1:( i  1) t , z t , x ) instead of p ( y it | y 1:( i  1) t , z t , x ) . During training, at each time step t , we teacher-force the generation of the fact y t by feeding the ground-truth history, y 1:( t  1) , to the word-level Transformer decoder. However, since only y t depends on the current hidden state z t , we only calculate the loss over y t . We apply the backward algorithm (Rabiner, 1989) to learn the parameters introduced in Section 3.2, where we maximize p ( y | x ) , i.e., the marginal likelihood of the observed facts y given input triples x , over all the latent states z and o on the entire dataset using dynamic programming. Following Murphy (2012), and given that the latent state at time t is C , we define a conditional likelihood of future evidence as:  t ( C ) (cid:44) p ( y t +1: T | z t = C, x ) , (2) where C denotes a group of predicates that are associated with the emission of y . The size of C ranges from 1 to L and each component is from the collection of predicates Q (see Section 3.2). Then, the backward recurrences are:  t  1 (cid:0) C (cid:48) (cid:1) = p (cid:0) y t : T | z t  1 = C (cid:48) , x (cid:1) = (cid:88) C  t ( C ) p ( y t | z t = C, x ) p (cid:0) z t = C | z t  1 = C (cid:48) , x (cid:1) with the base case  T ( C ) = 1 . In Equation 2, the size of the search space for C is (cid:80) L  =1 K  , where K = | Q | , i.e., the number of unique predicates appearing in the dataset. The 0 1 0 0 1 Fact1 Fact2 Input Ordering Predicates of Input triples Input Aggregation Text Generation Sentence Planning Figure 3: The inference process (Section 3.4) problem can still be intractable due to high K , despite the simplifications explained in Section 3.2 (cf. predicates). To tackle this issue and reduce the search space of C , we: (1) only explore permutations of C that include predicates appearing on the input; (2) introduce a heuristic based on the overlap of tokens between a triple and a fact if a certain fact mentions most tokens appearing in the predicate and object of a triple we hard-align it to this triple. 4 As a result, we discard the permutations that do not include the aligned predicates. After the joint learning process, the model is able to plan, i.e., order and aggregate the input triples in the most likely way, and then generate a text description following the planning results. Therefore, the joint prediction of ( y,  z ) is defined as: ( y,  z ) = arg max ( y (cid:48) , z (cid:48) ) , z (cid:48) {  z ( i ) } p (cid:0) y (cid:48) , z (cid:48) | x (cid:1) = arg max ( y (cid:48) , z (cid:48) ) , z (cid:48) {  z ( i ) } p ( y (cid:48) | z (cid:48) , x ) p ( z (cid:48) | x ) , (3) where {  z ( i ) } denotes a set of planning results,  y is the text description, and  z is the planning result that  y is generated from. The entire inference process (see Figure 3) includes three steps: input ordering, input aggregation, and text generation. The first two steps are responsible for the generation of {  z ( i ) } together with their probabilities { p ( z ( i ) | x ) } , while the last step is for the text generation p ( y (cid:48) |  z ( i ) , x ) . Planning: Input Ordering. The aim is to find the top-k most likely orderings of predicates appearing in the input triples. In order to make the search process more efficient, we apply left-to-right beam-4 This heuristic is using the rule-based aligner introduced in Section 5 with a threshold to rule out alignments in which the triples are not covered over 50%, since our model emphasises more on precision. Thus, not all triples are aligned to a fact . search 5 based on the transition distribution introduced in Equation 1. Specifically, we use a transition distribution between latent variables within latent states, calculated with predicate embeddings A in and B in (see Section 3.2). To guarantee that the generated sequence does not suffer from omission and duplication of predicates, we constantly update the masking matrix M ( q ) by removing generated predicates from the set q . The planning process stops when q is empty. Planning: Input Aggregation. The goal is to find the topn most likely aggregations for each result of the Input Ordering step. To implement this process efficiently, we introduce a binary state for each predicate in the sequence: 0 indicates wait and 1 indicates emit (green squares in Figure 3). Then we list all possible combinations 6 of the binary states for the Input Ordering result. For each combination, the aggregation algorithm proceeds left-to-right over the predicates and groups those labelled as emit with all immediately preceding predicates labelled as wait. In turn, we rank all the combinations with the transition distribution introduced in Equation 1. In contrast to the Input Ordering step, we use the transition distribution between latent variables across latent states, calculated with predicate embeddings A out and B out . That is, we do not take into account transitions between two consecutive predicates if they belong to the same group. Instead, we only consider consecutive predicates across two connected groups, i.e., the last predicate of the previous group with the first predicate of the following group. Text Generation. The final step generates a text description conditioned on the input triples and the planning result (obtained from the Input Aggregation step). We use beam search and the planning-conditioned generation process described in Section 3.2 (Emission Distribution). While the jointly learnt model is capable of fully automatic generation including the planning step (see Section 3.4), the discrete latent space allows direct access to manually control the planning component, which is useful in settings which require 5 We use beam search since Viterbi decoding aims at getting z  = arg max z ( z 1: T | y 1: T ) , but y 1: T is not available at this stage. 6 We assume that each fact is comprised of L triples at most. To match this assumption, we discard combinations containing a group that aggregates more than L predicates. increased human supervision and is a unique feature of our architecture. The plans (latent variables) can be controlled in two ways: (1) hyperparameter. Our code offers a hyperparameter that can be tuned to control the level of aggregation: no aggregation, aggregate one, two triples, etc. The model can predict the most likely plan based on the input triples and the hyperparameter and generate a corresponding text description; (2) the model can directly adopt human-written plans, e.g. using the notation [eatType][near customer-rating] , which translates to: first generate eatType' as an independent fact and then aggregate the predicates near' and customer-rating' in the following fact and generate their joint description. We tested our approach on two widely used data-to-text tasks: the E2E NLG (Novikova et al., 2017) and WebNLG 7 (Gardent et al., 2017a). Compared to E2E, WebNLG is smaller, but contains more predicates and has a larger vocabulary. Statistics with examples can be found in Appendix C. We followed the original training-development-test data split for both datasets. Generation Evaluation focuses on evaluating the generated text with respect to its similarity to human-authored reference sentences. To compare to previous work, we adopt their associated metrics to evaluate each task. The E2E task is evaluated using BLEU (Papineni et al., 2002), NIST (Dod-dington, 2002), ROUGE-L (Lin, 2004), METEOR (Lavie and Agarwal, 2007), and CIDEr (Vedantam et al., 2015). WebNLG is evaluated in terms of BLEU, METEOR, and TER (Snover et al., 2006). Factual Correctness Evaluation tests if the generated text corresponds to the input triples (Wen et al., 2015b; Reed et al., 2018; Dusek et al., 2020). We evaluated on the E2E test set using automatic slot error rate (SER), 8 i.e., an estimation of the occurrence of the input attributes (predicates) and their values in the outputs, implemented by Dusek et al. 7 Since we propose exploring sentence planning and increasing the controllability of the generation model and do not aim for a zero-shot setup, we only focus on the seen category in WebNLG. 8 SER is based on regular expression matching. Since only the format of E2E data allows such patterns for evaluation, we only evaluate factual correctness on the E2E task. To evaluate the contributions of the planning component, we choose the vanilla Transformer model (Vaswani et al., 2017) as our baseline, trained on pairs of linearized input triples and target texts. In addition, we choose two types of previous works for comparison: (1) best-performing models reported on the WebNLG 2017 (seen) and E2E dataset, i.e. T5 (Kale and Rastogi, 2020), PlanEnc (Zhao et al., 2020), ADAPT (Gardent et al., 2017b), and TGen (Dusek and Jurccek, 2016); (2) models with explicit planning, i.e. TILB-PIPE (Gardent et al., 2017b), NTemp+AR (Wiseman et al., 2018) and Shen et al. (2020). To make our HMM-based approach converge faster, we initialized its encoder and decoder with the baseline model parameters and fine-tuned them during training of the transition distributions. Encoder and decoder parameters were chosen based on validation results of the baseline model for each task (see Appendix D for details). Table 1 shows the generation results on the WebNLG seen category (Gardent et al., 2017b). Our model outperforms TILB-PIPE and Transformer, but performs worse than T5, PlanEnc and ADAPT. However, unlike these three models, our approach does not rely on large-scale pretraining, extra annotation, or heavy pre-processing using external resources. Table 2 shows the results when training and testing on the original E2E set. AGGGEN outperforms NTemp+AR and is comparable with Shen et al. (2020), but performs slightly Model BLEU NIST MET R-L CIDer Add Miss Wrong SER TGen (cid:7) 66.41 8.5565 45.07 69.17 2.2253 00.14 04.11 00.03 04.27 NTemp+AR (cid:7) 59.80 7.5600 38.75 65.01 1.9500     Shen et al. (2020) (cid:7) 65.10  45.50 68.20 2.2410     Transformer 68.23 8.6765 44.31 69.88 2.2153 00.30 04.67 00.20 05.16 AGGGEN 64.14 8.3509 45.13 66.62 2.1953 00.32 01.66 00.71 02.70 AGGGEN  OD 58.90 7.9100 43.21 62.12 1.9656 01.65 02.99 03.01 07.65 AGGGEN  AG 44.00 6.0890 43.75 58.24 0.8202 08.74 00.45 00.92 10.11 Table 2: Evaluation of Generation (middle) and Factual correctness (right) trained/tested on the original E2E data (Section 5 for metrics description). However, the results in Table 3 demonstrate that our model does outperform the baselines on most surface metrics if trained on the noisy original E2E training set and tested on clean E2E data (Dusek et al., 2019). This suggests that the previous performance drop was due to text references in the original dataset that did not verbalize all triples or added information not present in the triples that may have down-voted the fact-correct generations. 9 This also shows that AGGGEN produces correct outputs even when trained on a noisy dataset. Since constructing high-quality data-to-text training sets is expensive and labor-intensive, this robustness towards noise is important. The results for factual correctness evaluated using SER on the original E2E test set are shown in Table 2. The SER of AGGGEN is the best among all models. Especially, the high Miss scores for TGen and Transformer demonstrate the high chance of information omission in vanilla seq2seq-based generators. In contrast, AGGGEN shows much better coverage over the input triples while keeping a low level of hallucination (low Add 9 We also trained and tested models on the cleaned E2E data. The full results (including the factual correctness evaluation) are shown in Table 8 in Appendix F: there is a similar trend as in results in Table 3, compared to Transformer. and Wrong scores). To explore the effect of input planning on text generation, we introduced two model variants: AGGGEN  OD , where we replaced the Input Ordering with randomly shuffling the input triples before input aggregation, and AGGGEN  AG , where the Input Ordering result was passed directly to the text generation and the text decoder generated a fact for each input triple individually. The generation evaluation results on both datasets (Table 1 and Table 2) show that AGGGEN outperforms AGGGEN  OD and AGGGEN  AG substantially, which means both Input Ordering and Input Aggregation are critical. Table 2 shows that the factual correctness results for the ablative variants are much worse than full AGGGEN , indicating that planning is essential for factual correctness. An exception is the lower number of missed slots in AGGGEN  AG . This is expected since AGGGEN  AG generates a textual fact for each triple individually, which decreases the possibility of omissions at the cost of much lower flu-ency. This strategy also leads to a steep increase in added information. Additionally, AGGGEN  AG performs even worse on the E2E dataset than on the WebNLG set. This result is also expected, since input aggregation is more pronounced in the E2E dataset with a higher number of facts and input triples per sentence (cf. Appendix C). We manually examined a sample of 100 outputs (50 from each dataset) with respect to their factual correctness and fluency. For factual correctness, we follow the definition of SER and check whether there are hallucinations, substitutions or omissions in generated texts. For fluency, we check whether the generated texts suffer from grammar mistakes, redundancy, or contain unfinished sentences. Fig-the cricketers is a chinese restaurant near all bar one in the city centre . it is children friendly and has an average customer rating . william anders birthplace british hong kong apollo 8 backup_pilot buzz aldrin apollo 8 crewmembers frank borman apollo 8 operator nasa william anders was born in british hong kong and served as a crew member on apollo 8. frank borman was a crewman aboard the nasa operated apollo 8 mission. the backup pilot was buzz aldrin. william anders retired on september 1st , 1969 . william anders (born in british hong kong) was a crew member of nasa's apol 8 alongside frank borman. william anders retired on september 1st, 1969 . [birthPlace] [crew_member] [operator crewMembers] [backup_pilot] [Retirement] [eatType priceRange] [food customerrating] [familyFriendly area near] the cricketers area city centre the cricketers customerrating average the cricketers eattype restaurant the cricketers familyfriendly yes the cricketers food chinese the cricketers near all bar one the cricketers pricerange high I npu t s T r a n s A gg G e n the cricketers is a chinese restaurant with a high price range. it has an average customer rating and is children friendly near all bar one in the city centre. william anders retirement 1969-09-01 william anders crew_member apollo 8 I npu t s T r a n s A gg G e n Figure 4: Examples of input and system-generated target text for E2E (top) and WebNLG (bottom). ure 4 shows two examples of generated texts from Transformer and AGGGEN (more examples, including target texts generated by AGGGEN  OD and AGGGEN  AG , are shown in Table 6 and Table 7 in Appendix A). We observe that, in general, the seq2seq Transformer model tends to compress more triples into one fluent fact, whereas AGGGEN aggregates triples in more but smaller groups, and generates a shorter/simpler fact for each group. Therefore, the texts generated by Transformer are more compressed, while AGGGEN 's generations are longer with more sentences. However, the planning ensures that all input triples will still be mentioned. Thus, AGGGEN generates texts with higher factual correctness without trading off fluency. 10 6 Intrinsic Evaluation of Planning We now directly inspect the performance of the planning component by taking advantage of the readability of SRL-aligned facts. In particular, we investigate: (1) Sentence planning performance. We study the agreement between model's planning and reference planning for the same set of input triples; (2) Alignment performance  we use AGGGEN as an aligner and examine its ability to align segmented facts to the corresponding input triples. Since both studies require ground-truth triple-to-fact alignments, which are not part of the WebNLG and E2E data, we first introduce a human annotation process in Section 6.1. 10 The number of fluent generations for Transformer and AGGGEN among the examined 100 examples are 96 and 95 respectively. The numbers for AGGGEN  OD and AGGGEN  AG are 86 and 74, which indicates that both Input Ordering and Input Aggregation are critical for generating fluent texts. We asked crowd workers on Amazon Mechanical Turk to align input triples to their fact-based text snippets to derive a  reference plan  for each target text. 11 Each worker was given a set of input triples and a corresponding reference text description, segmented into a sequence of facts. The workers were then asked to select the triples that are verbalised in each fact. 12 We sampled 100 inputs from the WebNLG 13 test set for annotation. Each input was paired with three reference target texts from WebNLG. To guarantee the correctness of the annotation, three different workers annotated each input-reference pair. We only consider the alignments where all three annotators agree. Using Fleiss Kappa (Fleiss, 1971) over the facts aligned by each judge to each triple, we obtained an average agreement of 0.767 for the 300 input-reference pairs, which is considered high agreement. We now check the agreement between the model-generated and reference plans based on the top-1 Input Aggregation result (see Section 3.4). We introduce two metrics:  Normalized Mutual Information (NMI) (Strehl and Ghosh, 2002) to evaluate aggregation. We represent each plan as a set of clusters of triples, where a cluster contains the triples sharing the same fact verbalization. Using NMI we measure mutual information between two clusters, normalized into the 0-1 range, where 0 and 1 denote no mutual information and perfect correlation, respectively. Kendall's tau (  ) (Kendall, 1945) is a ranking based measure which we use to evaluate both ordering and aggregation. We represent each plan as a ranking of the input triples, where the rank of each triple is the position of its associated fact verbalization in the target text. measures rank correlation, ranging from -1 (strong disagreement) to 1 (strong agreement). In the crowdsourced annotation (Section 6.1), each set of input triples contains three reference texts with annotated plans. We fist evaluate the correspondence among these three reference plans by 11 The evaluation requires human annotations, since anchor-based automatic alignments are not accurate enough (86%) for the referred plan annotation. See Table 5 (RB) for details. 12 The annotation guidelines and an example annotation task are shown in Figure 7 in Appendix G. 13 We chose WebNLG over E2E for its domain and predicate diversity. calculating NMI and  between one plan and the remaining two. In the top row of Table 4, the high average and maximum NMI indicate that the reference texts' authors tend to aggregate input triples in similar ways. On the other hand, the low average  shows that they are likely to order the aggregated groups differently. Then, for each set of input triples, we measure NMI and  of the top-1 Input Aggregation result (model's plan) against each of the corresponding reference plans and compute average and maximum values (bottom row in Table 4). Compared to the strong agreement among reference plans on the input aggregation, the agreement between model's and reference plans is slightly weaker. Our model has slightly lower agreement on aggregation (NMI), but if we consider aggregation and ordering jointly (  ), the agreement between our model's plans and reference plans is comparable to the agreement among reference plans. In this study, we use the HMM model as an aligner and assess its ability to align input triples with their fact verbalizations on the human-annotated set. Given the sequence of observed variables, a trained HMM-based model is able to find the most likely sequence of hidden states z  = arg max z ( z 1: T | y 1: T ) using Viterbi decoding. Similarly, given a set of input triples and a factoid segmented text, we use Viterbi with our model to align each fact with the corresponding input triple(s). We then evaluate the accuracy of the model-produced alignments against the crowdsourced alignments. The alignment evaluation results are shown in Table 5. We compare the Viterbi (Vtb) alignments with the ones calculated by a rule-based aligner (RB) that aligns each triple to the fact with the greatest word overlap. The precision of the Viterbi aligner is higher than the rule-based aligner. However, the Viterbi aligner tends to miss triples, which leads to a lower recall. Since HMMs are locally optimal, the model cannot guarantee to annotate input triples once and only once. We show that explicit sentence planning, i.e., input ordering and aggregation, helps substantially to produce output which is both semantically correct as well as naturally sounding. Crucially, this also enables us to directly evaluate and inspect both the model's planning and alignment performance by comparing to manually aligned reference texts. Our system outperforms vanilla seq2seq models when considering semantic accuracy and word-overlap based metrics. Experiment results also show that AGGGEN is robust to noisy training data. We plan to extend this work in three directions: Other Generation Models. We plan to plug other text generators, e.g. pre-training based approaches (Lewis et al., 2020; Kale and Rastogi, 2020), into AGGGEN to enhance their interpretability and controllability via sentence planning and generation. Zero/Few-shot scenarios. Kale and Rastogi (2020)'s work on low-resource NLG uses a pre-trained language model with a schema-guided representation and hand-written templates to guide the representation in unseen domains and slots. These techniques can be plugged into AGGGEN , which allows us to examine the effectiveness of the explicit sentence planning in zero/few-shot scenarios. Including Content Selection. In this work, we concentrate on the problem of faithful surface realization based on E2E and WebNLG data, which both operate under the assumption that all input predicates have to be realized in the output. In contrast, more challenging tasks such as RotoWire (Wiseman et al., 2017), include content selection before sentence planning. In the future, we plan to include a content selection step to further extend AGGGEN 's usability. This research received funding from the EPSRC project AISec (EP/T026952/1), Charles University project PRIMUS/19/SCI/10, a Royal Society research grant (RGS/R1/201482), a Carnegie Trust incentive grant (RIG009861). This research also received funding from Apple to support research at Heriot-Watt University and Charles University. We thank Alessandro Suglia, Jindrich Helcl, and Henrique Ferrolho for their suggestions. We thank the anonymous reviewers for their helpful comments.
method objective result other abstain abstain abstain abstain abstain method objective objective objective objective objective objective objective objective method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain objective result abstain other other other other other other other other other other other other other other other other other other other objective method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other method other method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method other abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain abstain abstain result method result abstain objective abstain abstain abstain abstain abstain abstain abstain objective other other other other