Neural conditional text generation systems have achieved significant progress in recent years, showing the ability to produce highly fluent text. However, the inherent lack of controllability in these systems allows them to hallucinate factually incorrect phrases that are unfaithful to the source, making them often unsuitable for many real world systems that require high degrees of precision. In this work, we propose a novel confidence oriented decoder that assigns a confidence score to each target position. This score is learned in training using a variational Bayes objective, and can be leveraged at inference time using a calibration technique to promote more faithful generation. Experiments on a structured data-to-text datasetWikiBio (Lebret et al., 2016) -show that our approach is more faithful to the source than existing state-of-the-art approaches, according to both automatic metrics and human evaluation.
This paper presents a method for conditional text generation that has higher factual precision, minimizing hallucination of facts. The method involves predicting confidence of generation at each time step and using this confidence measure to skip tokens during generation and calibrate output probabilities in test time. Their method achieves SoTA performance on automatically measured precision and human evaluated "faithfulness." However their method does see a drop in recall (automatic metric and human evaluation).
Comments and issues,
- The intuitive explanation for the confidence score is a little confusing. In Section 4, page 3, you say that "If a token is likely a content word (i.e. when its generation probability by the encoder-decoder is much higher than the unconditioned language model), but the attention score is low, then the token might not be predicted based on the source, and could be hallucination." However, this doesn't seem like an airtight conclusion. Isn't it possible that the base-LM and enc-dec model have similar probabilities for a content word with the enc-dec attention being low? This seems possible given your observation that low attention to the source is what may be causing content hallucination. This same thing is essentially restated in section 4.1 "we expect P(y_t |y_<t, x) to be higher than P(y_t | y_<t) for content words so the confidence score will largely depend on the attention score", which seems more tangled up since P(y_t |y_<t, x) inherently depends on the attention score. This is all clarified when you explain the alteration made to the base-LM. I would recommend rewording/rearranging some of the earlier explanation for the efficacy of the confidence score since it seems that the alteration to the base-LM is an essential part of the explanation. 
- Need some explanation for Equation 6. I don't really get the intuition behind it.
- The presented results are pretty good! However, I would like to see some numbers on average score across a few runs.
- It would also be good to see results on one more dataset like E2E.
- Provide a little more detail on human evaluation, you don't even mention if the evaluation was done with crowd-workers or another pool of people like grad students. How many annotators? What is the inter-annotator agreement? What was the prompt/structure? Human evaluation of models is notoriously difficult, more details would give some more weight to the results.
I think this is a well written paper with thought out experiments. I recommend it be accepted to ICLR. I'd also be curious to see some future work that improves, or at least maintains recall, while keeping the higher precision.
Minor requests/recommendations: 
- Include more examples of generations. Could be an appendix.