  Data-to-text (D2T) generation aims to transform structured data into natural
language text. Data-to-text pre-training has proved to be powerful in enhancing
D2T generation and yields impressive performances. However, previous
pre-training methods either oversimplified structured data into a sequence
without considering input structures or designed training objectives tailored
for a specific data structure (e.g., table or knowledge graph). In this paper,
we unify different types of structured data (i.e., table, key-value data,
knowledge graph) into the graph format and cast different data-to-text
generation tasks as graph-to-text generation. To effectively exploit the
structural information of the input graph, we propose a structure-enhanced
pre-training method for D2T generation by designing a structure-enhanced
Transformer. Concretely, we devise a position matrix for the Transformer,
encoding relative positional information of connected nodes in the input graph.
In addition, we propose a new attention matrix to incorporate graph structures
into the original Transformer by taking the available explicit connectivity
structure into account. Extensive experiments on six benchmark datasets show
the effectiveness of our model. Our source codes are available at
https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/unid2t.

Computation and Language;Artificial Intelligence