Data-to-text generation focuses on generating fluent natural language responses from structured meaning representations (MRs). Such representations are compositional and it is costly to collect responses for all possible combinations of atomic meaning schemata, thereby necessitating few-shot generalization to novel MRs. In this work, we systematically study the compositional generalization of the state-of-the-art T5 models in few-shot data-to-text tasks. We show that T5 models fail to generalize to unseen MRs, and we propose a template-based input representation that considerably improves the model's generalization capability. To further improve the model's performance, we propose an approach based on self-training using fine-tuned BLEURT for pseudo-response selection. On the commonly-used SGD and Weather benchmarks, the proposed self-training approach improves tree accuracy by 46%+ and reduces the slot error rates by 73%+ over the strong T5 baselines in few-shot settings. 1 1 Introduction Data-to-text generation (Duek et al., 2020; Shen et al., 2020) is a critical component in today's task-oriented dialog systems for producing fluent natural language responses to users' requests. The task takes structured meaning representations (MRs) as input for natural language text response generation. Such representations are compositional, which allows for the combination of atomic meaning units in various ways to express the rich semantics encoded in languages. Recently, large pre-trained language models (LMs) have shown impressive results on many language understanding and generation  Work performed during an internship at Google. 1 Our code and data is available at github.com/googleresearch/google-research/tree/master/compgen_d2t 250 500 750 1000 Few-shot train split size 20 40 60 80 T r ee A cc u r a c y Semantic repr. tasks (Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2020), however it remains unclear how well these LMs generalize compositionally to novel semantic representations. There have been many studies revealing that large LMs often memorize the patterns from training data, while generalizing poorly to novel patterns. Compositionality in languages (Banarescu et al., 2013; Konstas et al., 2017) further aggravates such issues as the number of novel structural combinations exponentially increases with the number of atomic semantic units. In recent years, we have seen progress on benchmarking and measuring compositional generalization for languages (An-dreas, 2019), from perspectives including specialized architectures (Lake, 2019; Rao et al., 2019) and learning strategies (Andreas, 2020; Akyrek et al., 2021). However, most of these works study the generalization for NLU tasks like question answering (Keysers et al., 2020) and semantic pars-4205 Query: Is it jacket weather? ing (Kim and Linzen, 2020). To the best of our knowledge, compositional generalization for natural language generation is still an under-explored problem, which is the focus of this work. To answer the question of whether pre-trained LMs still suffer from lack of compositional generalization, we start with an empirical evaluation of T5 (Raffel et al., 2020), the state-of-the-art model on data-to-text generation tasks (Kale and Rastogi, 2020b). In our study, we use the Weather dataset (Balakrishnan et al., 2019) consisting of tree-structured compositional MRs along with tree-structured output responses (see Figure 2 for (a) naive MR and (c) target response). For evaluation, we compute the tree accuracy (Balakrishnan et al., 2019) which measures exact match between input and generated tree-structures. In this study we observe 47% 80% (across different few-shot train splits) drop in the tree accuracy when evaluated on validation splits containing unseen tree-structures in comparison to splits containing seen tree-structures (Figure 1). Furthermore, simply increasing the model size from T5small to T5large does not close the generalization gap (Table 2), af-firming our hypothesis that even strong seq-to-seq LMs fail to generalize compositionally. Inspired by Kale and Rastogi (2020a), we examine whether template-guided MRs are effective over naive MRs for tackling compositional generalization in data-to-text tasks. We introduce a simple template engine that traverses the compositional MR in a top-down manner and converts it to a text representation (Figure 2(b)). We hypothesize that such a template-guided setup reduces the change in representation between LM pre-training and finetuning. With template-guided MRs, we report up to 2 x increase in the tree accuracy over naive MRs on the validation split with unseen structures, demonstrating improved model generalization. We also propose to self-train the generation model to further boost performance by mitigating data sparsity in the low-data regime without requiring additional manual annotation. Concretely, we augment the limited labeled MRs with unlabeled novel MRs to iteratively bootstrap the model. To fil-ter out noisy pseudo responses during self-training, we repurpose BLEURT (Sellam et al., 2020), a learned metric, to be a quality estimator. We synthetically generate datasets for finetuning BLEURT with the goal of identifying hallucinations, missing slot-values, and ungrammatical responses. In sum, our overall approach improves the tree accuracy on unseen structures of the FewShotWeather dataset by 12 . 3% 46 . 4% over strong T5 baselines. On unseen schemata of the FewShotSGD dataset, we reduce the slot error rate by 54 . 4% 73 . 0% . In this section, we are interested in investigating the following with respect to data-to-text tasks: (Q1) Do current state-of-the-art generation models compositionally generalize? (Q2) What is an effective semantic representation for tackling compositional generalization? Problem Setup Data-to-text generation is the task of generating natural language text y from meaning representation (MR) x . In the context of task-oriented dialog systems, the choice of MR ranges from a flat list of slot-value pairs (Duek et al., 2018) to a more expressive tree structure. Balakrishnan et al. (2019) defines tree-structured MRs consisting of arguments, dialog acts, and discourse relations, which we use in this work. They report significant gains in the naturalness of the generated responses with tree-structured MRs on the Weather domain dataset. Figure 2 (a) visualizes an instantiation of such a tree-structured MR where the argument LOCATION is made up of a sub-argument ( CITY ), the dialog act RECOMMEND consists of three arguments ( ATTIRE _ NOT , LOCATION , DATE _ TIME ), and the discourse relation JUSTIFY captures the relationship between two dialog acts ( RECOMMEND , INFORM ). We consider linearized versions of tree-structured MR x and output response y . Generating the tree structure in the output enables us to compute the tree accuracy which helps to assess the structural correctness of the predicted response. FewShotWeather Dataset Due to the compositional nature of MRs, it is costly to collect responses for all combinations of discourse relations, dialog acts and arguments. In order to keep data labeling costs under control, we simulate a more realistic few-shot (or limited labeled data) setup. In the original Weather dataset, we have 25 , 390 training examples spanning 4 , 690 unique tree-structured MRs. An unique tree-structured MR is defined as a novel composition of discourse relations, dialog acts and argument names. Basically, they constitute non-terminals of a tree (Figure 2(a) without terminals or argument values like extremely humid, light rain, today, Palo Alto, jacket, and cold). For the Weather dataset (Balakrishnan et al., 2019), we construct 4 few-shot splits: 1shot-250, 1shot-500, 1shot-750, and 1shot-1000, where 1shot-X denotes training split to include one example per unique tree-structured MR and in total X unique tree-structured MRs. Further, all X examples in 1shotX are included while constructing 1shot-Y splits, where X < Y . We also make sure each discourse relation, dialog act and argument name is represented at least once in our few-shot splits. However, all combinations of these may not exist, thus allowing us to simulate structural shifts and evaluate compositional generalization. Based upon these splits, we construct two evaluation sets: seen tree-structures (overlapping with tree-structured MRs from 1shot-250) and unseen tree-structures (disjoint with tree-structured MRs from 1shot-1000) (see Section 4.1 for more details). Henceforth, all of the above splits constitute the FewShotWeather dataset. We release these splits for future studies. To answer (Q2), we use linearized tree structures as input to the T5 model ( naive representation ). However, T5 based models are pre-trained on normal text as input, thereby creating a representation discrepancy between pre-training and fine-tuning. To alleviate this discrepancy, we introduce a simple template engine that recursively traverses the compositional MR in a top-down manner to generate a structure-aware text representation ( template guided representation ). to convert naive representation (Figure 2(a)) to template guided representation (Figure 2(b)) are listed in Table 1. Each template, consisting of a name and a body, is invoked if a node in the MR (e.g., DG_INFORM) matches its name. A template can also invoke other templates or some utility functions. For example, template 3 could invoke templates 4 or 5 based on the returned value of the utility function IsSet($condition) (namely, whether the argument $condition is set or not). Such a template engine requires developing only a linear number of templates with respect to the number of meaning units to convert a compositional MR to a text representation, without writing a template for each unique MR (4,690 unique MRs in the dataset). In our study, we fine-tune the T5-small model using different few-shot train splits and report tree accuracy on validation splits. We observe that current state-of-the-art generation models undergo a significant drop in performance when evaluated on unseen tree structures. Specifically, with naive input representation, we observe 47% 80% (across different few-shot train splits) drop in tree accuracy, thus, providing evidence to answer (Q1) that the current model does not generalize to novel MRs. On experimentation with template guided MRs and 1shot-250 train split, the tree accuracy on validation unseen split increases from 8.77 to 26.3 ( 2 x increase over naive MRs), thus, answering (Q2) favorably (Figure 1). However, across different few-shot train splits, template-guided MRs still undergo a significant 41% 65% drop in tree accuracy on the unseen split compared to the seen split. Recent studies (Kaplan et al., 2020; Tay et al., 2021) show that model scale can affect the performance on several pre-training and downstream tasks. To understand how model scale affects the generalization to unseen structures, we consider three T5 variants: T5-small (77M), T5-base (120M), and T5-large (800M). We fine-tune each of these models on the full training data (16,816 examples corresponding to 1000 unique tree-structured MRs from 1shot-1000 split) and convincingly answer (Q3): Increasing the model (and dataset) size does not close the performance gap between seen and unseen splits (Table 2). Surprisingly, we observe that the T5-small model performs similarly or better than its larger counterparts. We use T5-small for the remaining experiments. As discussed earlier, the compositional nature of MRs makes it difficult to collect responses for all combinations. However, with access to data simulators (Rastogi et al., 2020), it is feasible to automatically generate large amounts of unlabeled MRs. Given limited labeled MRs, S = { x i , y i } ni =1 , and assuming access to unlabeled MRs, U = { x i } mi =1 , we investigate self-training (Scudder, 1965), a semi-supervised learning approach to effectively use U to improve compositional generalization. Self-training starts from a model trained on labeled data S , iteratively applies the current model to generate pseudo-labels on unlabeled data U , and then re-trains the current model on the augmented version of S and (subset of) U . For self-training to be effective, one needs to carefully select confident pseudo labels to alleviate the risk of reinforcing the model's mistakes (He et al., 2020). This issue gets further exacerbated in the context of generation tasks, where neural models are prone to hallucinate additional content not supported by the input (Maynez et al., 2020). With recent developments in learned evaluation metrics that penalize the model for hallucination, fluency, etc., we pose the question: Can we repurpose those metrics to assess the quality of pseudo-responses during self-training? Formally, given a pair of template guided MR (source) and model predicted response (candidate), we want a model that estimates the response quality by looking for hallucinations, fluency, coverage of argument value-pairs. Ideally, to learn such a model we require a large amount of positive and negative text pairs. To alleviate this requirement, we propose synthesizing the examples using the limited labeled task dataset. Furthermore, we initialize our quality estimation model using a pre-trained BLEURT (Sellam et al., 2020), which is shown to be sample efficient and robust to data shifts as a learned evaluation metric. Soruce (text-to-text input): there will be light freezing fog with a temperature high of 74 low of 61 at next friday Positive candidate (target response): next friday will have a high of 74 , a low of 61 , and a light freezing fog Negative candidates: [retrieving similar examples] next friday will be cloudy with a high of 74 , a low of 61 , and thunderstorms and rain [pairing with reference] there will be light freezing fog with a temperature high of 74 low of 61 at next friday [swapping words] next friday will of have a high of will 74 , a low of 61 , and a light freezing fog [repeating phrases] next friday will have a high of 74 , a low of 61 of 61 , and a light freezing fog [dropping phrases] next friday will have a high of 74 , a low of 61 , and a light freezing fog [flipping digits] next friday will have a high of 78 , a low of 61 , and a light freezing fog Once we have a fine-tuned BLEURT model, we use it to select pseudo-responses using a selection threshold for self-training. We synthetically generate the dataset for finetuning BLEURT using the labeled dataset available for each of our experiments. Template guided inputs and ground truth target responses are paired as positive examples (rating: 1 . 0 ). We use the following transformations on the target responses to create negative examples (rating: 0 . 0 ): Retrieving similar examples: For every input x , we rank all other inputs from the dataset using the BLEU score and select top-k examples below a certain threshold ( 90 . 0 ). Target responses corresponding to these top-k examples are paired with x to construct negative examples. Intuitively, these responses partially overlap with input x in terms of the content and inform a fine-tuned model to handle hallucinations. Pairing with reference: Template guided inputs need not be grammatically correct. Pairing the input x with itself as a response provides grammatically incorrect negative examples. Swapping, repeating and dropping phrases, flipping digits: Using these methods, we prepare a fine-tuned BLEURT for structurally inconsistent behaviors of the NLG system. Figure 3 visualizes an instantiation of different transformations to construct negative examples. FewShotWeather The original Weather dataset (Balakrishnan et al., 2019) has 25 , 390 training examples. Each example consists of a user query, the tree-structured MR, the tree-structured annotated response and metadata. As discussed in Section 2, we create new canonical subsets for compositional generalization experiments, FewShotWeather with 1shot-250 (approx. 1% of original training data), 1shot-500, 1shot-750, and 1shot-1000 splits. We repurpose all the remaining 24 k training examples as unlabeled examples for self-training. Our evaluation splits have 1 , 087 / 1 , 121 (val/test) examples with seen tree-structures, and 1 , 095 / 1 , 170 (val/test) examples with novel tree-structures. We report tree accuracy and BLEU-4 (Papineni et al., 2002) for the FewShotWeather dataset. FewShotSGD The original multi-domain Schema Guided Dialogue (SGD) dataset (Rastogi et al., 2020) has 160 k examples spanning across 20 domains (e.g., Banks, Travel, Weather, etc.). For each of these domains, there are different services with a total of 45 different schemata. Schema here refers to the combination of intents and slots, which change with services and domains. Further, not all domains and services are observed during training. Therefore, we use this dataset to study generalization to unseen schemata. Specifically, we use the few-shot variant of the dataset, FewShotSGD, as introduced by Kale and Rastogi (2020a). The FewShotSGD benchmark consists of k -shot splits (5/10/20/40), where k denotes the number of dialogues selected per train domain. The few-shot train splits have 558/1,075/2,140/4,312 (5/10/20/40-shot) examples. Evaluation splits have 13,748/10,216 (val/test) examples with seen schema, and 10,386/26,568 (val/test) examples with novel schema. Following Kale and Rastogi (2020a), we report BLEU-4 and slot error rate (SER) (Duek and Jurcicek, 2019). SER measures the fraction of examples where at least one slot was incorrectly copied from the input (lower SER is better). For each of the experiments we fine-tune the off-the shelf T5.1.1.small checkpoint 2 . It has 6 layers each in encoder and decoder with a total of 77 M parameters. We set the maximum sequence length to 512 , batch size to 16 and a constant learning rate of 0 . 001 for Adafactor optimizer (Shazeer and Stern, 2018). All models are fine-tuned on a 4x4 TPU slice, each taking around 2-3 hours to fin-ish 5000 steps. We evaluate models after every 200 steps and retain the checkpoint yielding best tree accuracy (for FewShotWeather) or BLEU (for FewShotSGD) on the held-out validation seen split. During inference, we set the beam size to 4 and length penalty  = 0 . 6 . While constructing the fine-tuning dataset for BLEURT, we generate up to 4 different negative candidates for each of the 6 transformations. We upsample the positive examples to be half the total number of negative examples and retain random 10% of total examples for validation set. For finetuning the BLEURT model, we start with publicly available BLEURT-20-D12 (Sellam et al., 2020). We set the maximum sequence length to 512 , batch size to 32 , a learning rate 1e-6, and fine-tune for 100 k steps. We use the held-out validation set to select the best checkpoint for self-training. In this section, we compare the performance of BLEURT based pseudo-response selection strategy with that of vanilla self-training. For each experiment, we randomly sample an equal number of examples for vanilla self-training and the BLEURT model to explicitly control for the sample complexity. We run 3 iterations of the self-training unless explicitly specified and set the BLEURT score selection threshold to 0 . 99 . We study the performance on a dataset (FewShotWeather) with tree-structured outputs as well as show the generality of our method on a dataset (FewShotSGD) without explicit tree-structured outputs. Note that naive T5 fine-tuning with template guided input representation constitutes a strong baseline for few-shot experiments as shown by Kale and Rastogi (2020a). We include results from this baseline under None pseudo-response selection strategy as it does not involve self-training. Unseen tree structures (FewShotWeather) Table 3 reports the performance of different methods as a function of the number of labeled examples. We observe that the performance for all methods improves with more training data. Across all few-shot splits, we observe that BLEURT based self-training improves over vanilla self-training both in terms of tree accuracy and BLEU. Empirically, we see that relative gains in tree accuracy (over the 4210 Model Self-No. of FewShotWeather training training Seen structures Unseen structures iteration examples BLEU  Tree Acc.  BLEU  Tree Acc.  Baseline 250 69 . 16 73 . 68 50 . 40 29 . 83 Vanilla 1 + 14 , 742 69 . 25 73 . 77 51 . 87 31 . 37 2 + 4 , 170 68 . 72 73 . 06 51 . 92 31 . 11 BLEURT-250 1 + 14 , 742 69 . 64 83 . 85 52 . 10 41 . 03 2 + 4 , 170 69 . 59 84 . 12 52 . 34 43 . 68 BLEURT-1000 1 + 14 , 021 70 . 95 84 . 83 52 . 13 45 . 47 2 + 4 , 772 70.47 85.64 53.08 47.44 Table 4: Model performance over multiple self-training iterations with FewShotWeather 1shot-250 train split. BLEURT-X denotes BLEURT model fine-tuned using 1shot-X train split. We observe that BLEURT model fine-tuned with larger datasets further enhances the self-training performance, especially on unseen structures. T5-small baseline) from vanilla self-training are comparable on both unseen and seen splits (e.g., 7 . 15% v.s. 6 . 72% , 1shot-500). On the other hand, BLEURT based self-training significantly improves the relative performance on the unseen split in comparison to seen splits (e.g., 18 . 72% vs. 10 . 5% , 1shot-500), thus showcasing the effectiveness of selecting quality pseudo-responses for improving performance on unseen tree-structures. Unseen schema (FewShotSGD) Table 3 reports the performance on the FewShotSGD dataset. Similar to results on the FewShotWeather dataset, we observe that the performance improves with more training data. Further, the performance of the baseline T5-small model is comparable to seen and unseen schemata. These gains can be attributed to the benefits of using template guided MRs. In comparison to vanilla self-training, BLEURT based approach improves the overall performance across all few-shot splits on both seen and unseen schema. For example, with 5-shot experiments, BLEURT based selection strategy reduces the SER on unseen schema from 19.93 to 5.39 ( 73% improvement) in comparison to the baseline T5 model. On the other hand, vanilla self-training reduces the SER only by 3.97 ( 20% ), thus showcasing the effectiveness of the proposed approach in filtering pseudo-responses with missing slot-value pairs. These results confirm that BLEURT based self-training is a generic method and can be plugged in to existing methods to improve the few-shot generalization capabilities of existing SOTA generation models. Performance with respect to self-training iterations We iteratively self-train the model starting from a T5-small baseline and continue adding unlabeled examples up to 3 iterations. From Table 4 and 9, we see that model performance improves across the self-training iterations. However, the number of additional examples added decreases over iterations, thus suggesting that 2 3 iterations might be enough to obtain benefits from self-training. Quality of fine-tuned BLEURT models For all our experiments, we use the few-shot labeled datasets for fine-tuning the BLEURT model. To investigate self-training performance with a BLEURT model fine-tuned on a large dataset, we set up an experiment on the FewShotWeather dataset, where we fine-tune the BLEURT model on a 1shot-1000 train split (BLEURT-1000) and use it for self-training with 1shot-250. From Table 4, we see that self-training with BLEURT-1000 performs significantly better than BLEURT-250, especially on unseen structures, thereby confirming the intuition that self-training is sensitive to the quality of the BLEURT model. Aside from automatic metrics-based evaluation, we also perform a human evaluation study by asking annotators to assess the quality of the generated responses from different models. For each example, human annotators are shown user query, generated response and the ground truth response. They are asked to provide ratings on a scale of 1 (bad), 2 (slightly bad) to 3 (good) along two dimensions: grammaticality , naturalness , rating on a scale of 0 (less) to 1 (adequate) for informativeness , and binary rating for accuracy . Similar to (Balakrish-nan et al., 2019), grammaticality evaluates the response for subject-verb agreement, repetitions, and grammatical completeness. Naturalness measures whether the response sounds coherent and natural by the response itself. Informativeness measures whether the response contains the right amount 4211 Model Gram Nat Info Acc FewShotWeather (Seen split) Baseline 2.59 2.55 0.81 0.94 BLEURT 2.66 1 2.63 1 0.80 0.93 Full 2.66 1 2.61 0.80 0.95 FewShotWeather (Unseen split) Baseline 2.43 2.41 0.75 0.79 BLEURT 2.50 1 2.46 1 0.76 0.80 Full 2.53 1 2.50 1 0.79 1 0.86 1 , 2 FewShotSGD (Seen split) Baseline 2.72 2.66 2 0.79 0.76 BLEURT 2.69 2.59 0.81 0.88 1 Full 2.83 1 , 2 2.74 1 , 2 0.81 0.94 1 , 2 FewShotSGD (Unseen split) Baseline 2.70 2.61 0.77 0.72 BLEURT 2.67 2.60 0.79 0.86 1 Full 2.83 1 , 2 2.73 1 , 2 0.82 1 , 2 0.94 1 , 2 Table 5: Human evaluation results comparing different models. of relevant information to the user query and accuracy evaluates the response for hallucinations (incorrectly added slots), missing slots by comparing it against the reference. For each evaluation split (seen/unseen), we randomly select 200 examples and collect ratings from 3 different annotators. For the FewShotWeather/SGD datasets, we consider models trained with 1shot-250/5-shot splits and compare them with models fine-tuned on the full dataset. In total, we collect 7 , 200 annotations, each with 3 ratings. Table 5 reports results for human evaluation study. FewShotWeather Similar to automatic metrics, we see a drop in human ratings on the unseen split (compared to seen split), confirming the model's lack of generalization to novel MRs. On both the evaluation splits, our approach outperforms the baseline model with significant results on grammaticality and naturalness ratings. Moreover, the responses from the self-trained model are comparable (in terms of the human ratings) with that of the model fine-tuned with the full dataset, demonstrating the effectiveness of our approach. FewShotSGD Apart from generating natural responses, model responses must be factually grounded in the input data and address user queries. On FewShotSGD, we see that our approach significantly improves informativeness and accuracy rating over the baseline model. Surprisingly, we see a drop on naturalness when evaluated on seen schemata. In Table 6 (and Tables 7, 8 in Appendix A) we visualize the sample responses generated using different models for unseen test splits. We consider three models: T5-small baseline, BLEURT based self-training, and model trained with full data. For the FewShotWeather/ FewShotSGD datasets, we consider models trained with 1shot-250/ 5-shot train splits. We see that the baseline model fails to generate responses that are coherent and factually grounded in the input. They hallucinate to generate novel concepts like cloudy hail , drop relevant details like cafe located in Emeryville , and are repetitive in nature. We also report the BLEURT score along with human ratings per sample and see that they are reflective of the response quality. Data-to-Text Generation While early research focused on rule-based methods (Reiter and Dale, 2000), more recent work has relied heavily on neural methods (Wen et al., 2015; Marcheggiani and Perez-Beltrachini, 2018). Some recent works (Kale and Rastogi (2020b), Peng et al. (2020), Kale and Roy (2020)) showed that transfer learning from pre-trained language models can improve generalization capabilities and sample efficiency. In other lines of work, Ferreira et al. (2019); Moryossef et al. (2019) find that pipelined neural approaches with explicit planning steps can outperform their end-to-end counterparts, while Kale and Rastogi (2020a) and Du et al. (2020) showed the benefits of schema and template guided input representations. Inspired by Kale and Rastogi (2020a) we propose a simple and generic way to produce text-to-text representation, and study how it impacts compositional generalization. Self-training for NLG He et al. (2020) revisits the problem of self-training for NLG. They found that noise (from perturbing the input space) helps in self-training and propose a noisy version of self-training by augmenting vanilla training with the inputs from a reconstruction model. Building on this idea, the contemporary work (Heidari et al., 2021) on few-shot data-to-text generation proposes to self-train the model and shows efficacy 4212 Fields BLEURT Gram Nat Info Acc Input or output response User query --What will the temperature be tomorrow morning Template --There will be temperatures between 76 and 80 tomorrow morning there will be partly cloudy tomorrow morning Reference --The temperature for tomorrow morning will be between 76 and 80 fahrenheit along with partly cloudy skies Predictions Baseline -0.002 2.17 1.67 0.67 1.0 Expect partly cloudy skies and tomorrow morning. on the Weather dataset. Another contemporary work (Li et al., 2021) proposes to use constrained decoding to generate valid pseudo-responses for self-training and show convincing benefits. However, our work focuses on compositional generalization, rather than the pure few-shot learning setup. We propose a BLEURT-based self-training method, which is more generic than pseudo-response selection methods that rely on output structures. We systematically study the problem of compositional generalization for data-to-text generation and show that existing state-of-the-art generation models do not generalize to unseen structures. We propose a simple and generic way to produce template guided text representation for response generation, and demonstrate its effectiveness on both seen and unseen structures. Further, we introduce a generic self-training approach that leverages fine-tuned BLEURT for pseudo response selection and show significant improvements over vanilla self-training on existing few-shot data-to-text generation benchmarks. While our method requires only a small number of templates to start with, we still need to manually generate them for every unseen MR. Automatically generating templates by priming GPT-style models is an interesting line of future work. Furthermore, the effectiveness of our self-training method is highly dependent on the quality of the underlying BLEURT model (see Table 4). Given BLEURT based quality estimator is a learned model, it may be susceptible to data distribution shifts. We leave such analysis to future work. Another interesting future direction is to investigate the effectiveness of our approach to languages other than English. To study compositional generalization for data-to-text tasks, we introduce data splits based on the already existing, publicly available, and widely used compositional weather dataset (Balakrishnan et al., 2019). We release our data splits to facilitate the development of new methods and consistent evaluation of them in comparison with exist-4213 ing works. In terms of use-case scenarios, we focus on task-oriented dialogue generation by using large pre-trained language models. These models are known to exhibit and potentially amplify social biases found in the training data, such as gender biases (Dinan et al., 2020), and are capable of generating toxic or otherwise unsafe content (Wei-dinger et al., 2021). Our method helps these models generate higher quality responses than considered baselines when evaluated in terms of grammaticality, naturalness, informativeness, and accuracy. However, our work does not explicitly focus on mitigating social biases, unsafe content, or other potential ethical or social harms that might result from dialogue generation. Therefore, we caution against the deployment of our system in environments where any such biases can negatively impact the individuals interacting with our system without further assessment of the safety of this system in that environment.
abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective method result abstain abstain method result abstain method method abstain objective objective objective objective abstain objective result abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain method abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain method abstain abstain method abstain other abstain abstain abstain abstain method abstain abstain abstain abstain other method method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result result result abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain result other other other abstain other other other other other method objective result objective result method result abstain abstain objective method abstain method abstain method abstain abstain