Faithful Inversion of Generative Models for Effective Amortized Inference
Inference amortization methods share information across multiple posteriorinference problems, allowing each to be carried out more efficiently. Generally, they require the inversion of the dependency structure in the generative model, as the modeller must learn a mapping from observations to distributions approximating the posterior. Previous approaches have involved inverting the dependency structure in a heuristic way that fails to capture these dependencies correctly, thereby limiting the achievable accuracy of the resulting approximations. We introduce an algorithm for faithfully, and minimally, inverting the graphical model structure of any generative model. Such inverses have two crucial properties: a) they do not encode any independence assertions that are absent from the model and b) they are local maxima for the number of true independencies encoded. We prove the correctness of our approach and empirically show that the resulting minimally faithful inverses lead to better inference amortization than existing heuristic approaches.
The authors present a novel way to automatically generate the dependency structure for constructing inference network. They frame the problem as a graphical-model-inversion problem and propose to use variable elimination and the min-fill heuristic.  The proposed method generates a faithful and minimal structure for constructing inference network.  The authors show that the generated inference network outperforms the existing heuristically designed network and the fully connected network. This submission is well-written. I enjoying reading this paper.  I have the following questions related to this work.  (1) About the parametric form of variational factors in continuous cases Since the true posterior distribution is intractable, why do we use the same (distribution) parametric form in the inference network as the one in the model? Does the choice of the parametric form play a role in choosing the dependency structure of inference network? If the parametric form is limited, does a fully connected inference network have more expressive power?   (2) For a deep hierarchical Bayesian network, do skip connections play a role in training? Figure 5 shows that skip connections could speed up training at the first epochs.   Why is the minimality important? How about a faithful structure with skip connections from observations to latent variables? For example, we can construct a dependency structure by adding these skip connections into a faithful and minimal structure.