Published as a conference paper at ICLR 2018 VARIATIONAL MESSAGE PASSING WITH STRUCTURED INFERENCE NETWORKS
Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE. Finally, we derive a variational message passing algorithm to perform efficient naturalgradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods.
This paper presents a variational inference algorithm for models that contain
deep neural network components and probabilistic graphical model (PGM)
components.
The algorithm implements natural-gradient message-passing where the messages
automatically reduce to stochastic gradients for the non-conjugate neural
network components. The authors demonstrate the algorithm on a Gaussian mixture
model and linear dynamical system where they show that the proposed algorithm
outperforms previous algorithms. Overall, I think that the paper proposes some
interesting ideas, however, in its current form I do not think that the novelty
of the contributions are clearly presented and that they are not thoroughly
evaluated in the experiments.
The authors propose a new variational inference algorithm that handles models
with deep neural networks and PGM components. However, it appears that the
authors rely heavily on the work of (Khan & Lin, 2017) that actually provides
the algorithm. As far as I can tell this paper fits inference networks into
the algorithm proposed in (Khan & Lin, 2017) which boils down to i) using an
inference network to generate potentials for a conditionally-conjugate
distribution and ii) introducing new PGM parameters to decouple the inference
network from the model parameters. These ideas are a clever solution to work
inference networks into the message-passing algorithm of (Khan & Lin, 2017),
but I think the authors may be overselling these ideas as a brand new algorithm.
I think if the authors sold the paper as an alternative to (Johnson, et al., 2016)
that doesn't suffer from the implicit gradient problem the paper would fit into
the existing literature better.
Another concern that I have is that there are a lot of conditiona-conjugacy
assumptions baked into the algorithm that the authors only mention at the end
of the presentation of their algorithm. Additionally, the authors briefly state
that they can handle non-conjugate distributions in the model by just using
conjugate distributions in the variational approximation. Though one could do
this, the authors do not adequately show that one should, or that one can do this
without suffering a lot of error in the posterior approximation. I think that
without an experiment the small section on non-conjugacy should be removed.
Finally, I found the experimental evaluation to not thoroughly demonstrate the
advantages and disadvantages of the proposed algorithm. The algorithm was applied
to the two models originally considered in (Johnson, et al., 2016) and the
proposed algorithm was shown to attain lower mean-square errors for the two
models. The experiments do not however demonstrate why the algorithm is
performing better. For instance, is the (Johnson, et al., 2016) algorithm
suffering from the implicit gradient? It also would have been great to have
considered a model that the (Johnson, et. al., 2016) algorithm would not work
well on or could not be applied to show the added applicability of the proposed
algorithm.
I also have some minor comments on the paper:
- There are a lot of typos.
- The first two sentences of the abstract do not really contribute anything
  to the paper. What is a powerful model? What is a powerful algorithm?
- DNN was used in Section 2 without being defined.
- Using p() as an approximate distribution in Section 3 is confusing notation
  because p() was used for the distributions in the model.
- How is the covariance matrix parameterized that the inference network produces?
- The phrases "first term of the inference network" are not clear. Just use The
  DNN term and the PGM term of the inference networks, and better still throw
  in a reference to Eq. (4).
- The term "deterministic parameters" was used and never introduced.
- At the bottom of page 5 the extension to the non-conjugate case should be
  presented somewhere (probably the appendix) since the fact that you can do
  this is a part of your algorithm that's important.