This paper presents a new methodology for modeling the local semantic distribution of responses to a given query in the human-conversation corpus, and on this basis, explores a specified adversarial learning mechanism for training Neural Response Generation (NRG) models to build conversational agents. The proposed mechanism aims to address the training instability problem and improve the quality of generated results of Generative Adversarial Nets (GAN) in their utilizations in the response generation scenario. Our investigation begins with the thorough discussions upon the objective function brought by general GAN architectures to NRG models, and the training instability problem is proved to be ascribed to the special local distributions of conversational corpora. Consequently, an energy function is employed to estimate the status of a local area restricted by the query and its responses in the semantic space, and the mathematical approximation of this energy-based distribution is finally found. Building on this foundation, a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response generation models, named as LocalGAN. Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.
POST-REBUTTAL FEEDBACK
I share the same concerns as that of reviewer 2 in the response to the rebuttal. Hence, my score remains unchanged.
SUMMARY OF REVIEW
This paper motivates the need to "contextualize" responses based on the query to bring about stable training in NRG and consequently proposes localGAN to realize this. On the overall, I like the motivation and the proposed approach of this paper. The experimental results also look convincing.
On the flip side, the technical formulation and theoretical results are not presented rigorously and important technical details are missing, as discussed below. As a result, clarifications from the authors are needed to ensure the correctness of their formulation. The authors also need to improve the presentation and proof of the theoretical results; the correctness has to be checked again. In my opinion, these theoretical results do not improve my current assessment of the paper and can be removed to cut down to 8 pages. If the authors like to keep them, they need to revise them based on my concerns above.
It would be good to show some sample queries and corresponding "meaningful" responses produced by their proposed LocalGAN that are not considered safe responses which are produced by the other tested methods.
DETAILED COMMENTS
For Lemmas 1 and 2 and Theorem 1, the authors need to present them rigorously by specifying the exact math expressions since they have not defined what it means by sufficient, approximates, small enough, and estimate properly. This will also eliminate any discrepancy in their interpretations. For example, the authors have used Taylor series expansion to approximate the expectation of F(q,r) in equation 19 (instead of bounding it). Hence, one can claim that Lemma 2 does not hold and hence Theorem 1 does not hold as well.
In Section 4.3, the described mechanism is confusing to me: 
(a) Are the authors saying that it is performed sequentially from foundation to phase-1, followed by phase-2? Or are the authors saying that these three phases are expected behaviors occurring during the optimization in equation 17? 
(b) For the foundation phase, is the DBM pre-trained, that is, prior to optimization in equation 17?
(c) Are there multiple response clusters, that is, one for each q? If so, the second RELU term in the minimizing criterion in equation 17 does not seem to properly reflect this.
(d) How are the response cluster centers r_c exactly determined? The authors vaguely say that they are modeled from training data. Is it one center per response cluster? Are the cluster centers also optimized in equation 17, besides the generator's weights? Can the authors provide the argument under the min operator in equation 17? It is confusing to leave out r_c from the subscript of alpha.
I would have preferred that the authors specify the expression of the evaluation metrics to be self-contained.
In Fig. 2, how exactly do the authors measure stability? If the entropy rapidly increases like that of LocalGAN and Adver-REGS, are they considered stable?
Minor issues
Page 1: Despite of?
Page 3: inequation?
Page 3: Equation 4 and 5?
Equation 6: The first summation should just be over q, unless there are multiple sets of R_q per q.
tilde{R}_q is not used in equation 6.
Page 4: a limited samples?
Page 5: defined in 3.2?
Page 5: To be consistent, mathbb should be applied to E.
Page 9: valid this aspect?
Page 9: from the the?