The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities.
This work proposes the problem of fact verification with semi-structured data source such as tables. Specifically, the authors created a new dataset TabFact and evaluated two baseline models with different variations. They applied two criteria and different rewards for workers to collect two subsets of different levels (“simple” and “complex”). They also applied a negative rewriting strategy to avoid exploitable cues or patterns in the annotations. They evaluated two baselines models: (1) latent program algorithm (LPA), which makes use of simple string match entity linking and systematic search and trains a neural network discriminator, and (2) Table-BERT, which linearize the table into a sequence through concatenation or template, and treat it as a classfication problem. Both showed reasonable accuracy (~68%), but still below human performance (92.1%) on a held out test set.
I would like to see the paper accepted because:
(1) it proposes an interesting task (table fact verification) with a clean dataset, and the experiments evaluated the ability of the current neural network models, such as BERT, or hybrid models, such as the LPA baseline, to perform (symbolic) reasoning;
(2) special care is done to ensure the dataset doesn't contain simple cues or patterns, which is a common pitfall in dataset collection, and the dataset is also validated through two reasonable baseline models. 
Some weakness and concerns are:
(1) some error analysis of the baseline models are missing. For example, what types questions are hard/simple for table-BERT and what are hard/simple for LPA, and some comparison between them. This helps point out where the difficulty come from, for example, whether the difficulty is language understanding or symbolic reasoning.
(2) "To focus on statement verification against the table, we do not feed the caption to the model and simply mask the phrases in the statements which links to the caption with placeholders."
I am not sure this is the right thing to do since even the human annotators requires the caption to understand the context, why not also feed the caption into the model? 
(3) Although the Figure 2 showed that the higher order operations are indeed used in a majority of questions, which measures the breadth of the reasoning, it is unclear about the depths of the required reasoning, i.e., how many operations / steps are required to achieve the correct answer. It would help if the average number of steps / operations required for answering the questions are shown. 
If the above concerns are addressed, I will be willing to raise my score.
Minor comments:
The paper, especially the Appendix, requires some proof reading, for example, I believe the caption for Figure 5 in Appendix A is misplaced.
"... they are explicitly banned bring ..." -> "banned from bringing"
“As illustrated in Figure 6, the title only acts as a placeholder in the statements to make it sound more natural.” -> From the example, it seems the name of the player is kept unchanged in the sentence, which is different from a placeholder.
Suggestions:
I understand this might require some works, but it would be really helpful to add more comments and maybe examples for the functions shown in Figure 5 (you might need to split the table into two pages or have another table for examples), so that it can be used by works that follows the LPA approach.
“During the annotation, the workers are explicitly guided to modify the words, phrases or sentence structures but retain the sentence style/length to prevent from artificial cues”
Avoiding simple cues or patterns are important, so it will be good if more details can be shared (for example, the instructions/guidelines you showed to the workers). 
==================================
Post rebuttal update:
Thanks for the update to the paper. I think this work provides a good dataset and some reasonable baselines, thus should be accepted. 
However, I am still a bit concerned about the categorization of questions and the analysis on reasoning depth, because they are all based on the programs generated by LPA, while systematic search's recall is just 77% and the programs potentially contains a lot of spurious ones. So the categorization of the questions in Figure 11 and the number of reasoning steps in Figure 12 has to be taken with a grain of salt. It is worth annotating a few hundred, say 100-300 questions, manually with the ground truth programs for question distribution analysis or reasoning depth analysis, and confirm the numbers estimated using programs from LPA. Those manually annotated examples would also be a great addition to the dataset to aid the researchers for better analysis. A good example is the WikiTableQuestions dataset https://github.com/ppasupat/WikiTableQuestions/releases, which contains 300 manually annotated examples (annotated-all.examples), and they are used to analyze the distribution of the questions in the paper.