Deep Generative Models with Learnable Knowledge Constraints
The broad set of deep generative models (DGMs) has achieved remarkable advances. However, it is often difficult to incorporate rich structured domain knowledge with the end-to-end DGMs. Posterior regularization (PR) offers a principled framework to impose structured constraints on probabilistic models, but has limited applicability to the diverse DGMs that can lack a Bayesian formulation or even explicit density evaluation. PR also requires constraints to be fully specified a priori, which is impractical or suboptimal for complex knowledge with learnable uncertain parts. In this paper, we establish mathematical correspondence between PR and reinforcement learning (RL), and, based on the connection, expand PR to learn constraints as the extrinsic reward in RL. The resulting algorithm is modelagnostic to apply to any DGMs, and is flexible to adapt arbitrary constraints with the model jointly. Experiments on human image generation and templated sentence generation show models with learned knowledge constraints by our algorithm greatly improve over base generative models.
Summary: The paper proposes a way to incorporate constraints into the learning of generative models through posterior regularization. In doing so, the paper draws connections between posterior regularization and policy optimization. One of the key contributions of this paper is that the constraints are modeled as extrinsic rewards and learned through inverse reinforcement learning.  The paper studies an interesting and very practical problem and the contributions are substantial. The writing could definitely be made clearer for Sections 3 and 4, where the overloaded notation is often hard to follow.  I have the following questions: 1. Since generative models can be easily sampled from, why not introduce an additional term E_p[f(x)] and minimize L(theta) - E_p[f(x)]. The only explanation I find for introducing q in the paper is  “for efficient optimization”. I don’t see theoretically or empirically any reason why the optimization will suffer without q.         When f is learned as well, this reduces to energy-based GAN setting discussed in 210-221. I would be curious to see theoretical and/or empirical evidence in support of the  approach proposed in the paper.  2. L169-171 needs more clarification. For a self contained paper, the link between q_phi* and q_pi* seems like a key detail that’s missing from the paper.  In lines 149-156, the paper establishes a link between q* and q_pi*. In Eq. (8) however, they substitute q_phi with the expression for q*. What’s the justification?  3. Bayesian optimization over the latent space of a generative model is another approach to generate objects with desired properties. It has successfully been applied for generating molecules [1, 2].  Would be good to include a discussion on the merits of either approaches.  With regards to the experiments, it’s hard to comment on the significance of the results since the paper claims that the baselines from prior work are not directly comparable (which I agree). I believe energy-based GANs (as mentioned in pt.1 above)  and potentially Bayesian optimization techniques (if applicable) would have been good baselines to include.  [1] R. Gómez-Bombarelli, D. Duvenaud, J. M. Hernández-Lobato, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. arXiv preprint arXiv:1610.02415, 2016.   [2] M. J. Kusner, B. Paige, and J. M. Hernández-Lobato. Grammar variational autoencoder. arXiv preprint arXiv:1703.01925, 2017.   --- --- Thanks for the response. I would highly encourage the authors to include results for the energy based models in the final version, since I believe that to be the closest baseline to this work.