  The information in tables can be an important complement to text, making
table-based question answering (QA) systems of great value. The intrinsic
complexity of handling tables often adds an extra burden to both model design
and data annotation. In this paper, we aim to develop a simple table-based QA
model with minimal annotation effort. Motivated by the fact that table-based QA
requires both alignment between questions and tables and the ability to perform
complicated reasoning over multiple table elements, we propose an omnivorous
pretraining approach that consumes both natural and synthetic data to endow
models with these respective abilities. Specifically, given freely available
tables, we leverage retrieval to pair them with relevant natural sentences for
mask-based pretraining, and synthesize NL questions by converting SQL sampled
from tables for pretraining with a QA loss. We perform extensive experiments in
both few-shot and full settings, and the results clearly demonstrate the
superiority of our model OmniTab, with the best multitasking approach achieving
an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively,
also establishing a new state-of-the-art on WikiTableQuestions. Detailed
ablations and analyses reveal different characteristics of natural and
synthetic data, shedding light on future directions in omnivorous pretraining.
Code, pretraining data, and pretrained models are available at
https://github.com/jzbjyb/OmniTab.

Computation and Language