Autoencoders are powerful generative models for complex data, such as images. However, standard models like the variational autoencoder (VAE) typically have unimodal Gaussian decoders, which cannot effectively represent the possible semantic variations in the space of images. To address this problem, we present a new probabilistic generative model called the Hierarchical Bayes Autoencoder (HBAE). The HBAE contains a multimodal decoder in the form of an energybased model (EBM), instead of the commonly adopted unimodal Gaussian distribution. The HBAE can be trained using variational inference, similar to a VAE, to recover latent codes conditioned on inputs. For the decoder, we use an adversarial approximation where a conditional generator is trained to match the EBM distribution. During inference time, the HBAE consists of two sampling steps: first a latent code for the input is sampled, and then this code is passed to the conditional generator to output a stochastic reconstruction. The HBAE is also capable of modeling sets, by inferring a latent code for a set of examples, and sampling set members through the multimodal decoder. In both single image and set cases, the decoder generates plausible variations consistent with the input data, and generates realistic unconditional samples. To the best our knowledge, Set-HBAE is the first model that is able to generate complex image sets.
The article proposes to replace the conditional distribution p(x|z) in an VAE by a general graphical model parameterized by a deep neural network. Using a series of approximations it arrives at a triplet of models, where the graphical model is split into a model representing the likelihood and a model that generates samples from the distribution. the third model is the usual autoencoder. The core idea is that the latent space of the VAE serves as bottleneck so that the graphical model develops a multi-modal distribution. At the same time, the use of a neural network allows for a much better modeling than using the typical local Gaussian distribution as is used in most VAE-approaches. Experimental results show that reconstructions and markov-chains starting from reconstructions produce decent results. An extension to sets is proposed as well as a task showing performance for semi-supervised learning.
-------------------------------------------
The text above serves mainly as a  summary of what the paper sets out to do. However, we have to check whether the paper is actually achieving this goal. In my view, it does not, and it suffers from severe problems on the theoretical side. Furthermore, important evaluations that are standard within the VAE framework are missing: samples from the true underlying distribution. In the remainder of this review, I will try to substantiate the claims regarding theory which lead me to an overall decision to reject the article in its current form.
Let us start from a birds-eye perspective: the model consists of three parts:
the Encoder E, which i will refer to as q(z|x)
the Energy-function, also called E(x|z) and with -E referred to as D later on.
a generator function G(z,z') that is supposed to sample from the distribution p(x|z) = 1/Z(z) exp(-E(x|z)) using entropy generated by z'~N(0,I). I will refer to this distribution as r(x|z)
This split is smart: using a general graphical model to model p(x|z) makes it very hard to sample from the distribution and it is difficult to estimate the normalization constant Z. The paper refers to Zhai et.al (2016) to find eq(3). I really like (3) because the inequality is tight when G is producing samples from p(x|z). To be more exact, we have that the estimator for the normalization constant is log(Z) - KL(r(x|z) || p(x|z) ). This reveals the first issue: the generator must have an r(x|z) > 0 for all x except a countable infinite subset. This is difficult to achieve with dimensionality of z and z' smaller than dimensionality of x and G being deterministic. Therefore, KL(r(x|z) || p(x|z) ) is already infinite. This is a technicality that i will ignore, since we can always add a minimal amount of noise on the output of G, even though this paper does not.
The real problem starts with eq(4). Since r(x|z) is implicit, there is no way to compute its entropy, which is then just discarded. The paper argues that this is no problem, because they intend to somehow bound the entropy. We will look at this point, later. 
The issue with bounding is that for this to be meaningful, we also need a bound on the entropy of p(x|z), because otherwise there is no way for r(x|z) to achieve a small KL-divergence. If we now look at (4), we can compute the optimal r without bounded entropy. The optimal r is the distribution that by (5) maximizes -E(x|z) for a given z. this result is not a multi-modal distribution but a delta-peak on the optimum and certainly not p(x|z). so the bound is super important and technically we need a bound on log|det(d/dz' G(z,z'))|. 
This is difficult from a theoretical view-point as well as from a modeling view-point because we know that we have to have nearly discontinuous transformations to create sharp edges. Further, to transform z'~N(0,I), into a multi-modal distribution, there must be a function g(z')->w so that w is multi-modal. This is difficult to achieve with bounded determinant.
Looking at the experiments, let's see whether the determinant is actually bounded. The paper claims that bounding can be achieved using batch-normalization. First this requires a reference on why this should be the case and show that this holds as implemented for the resnet block and second, this requires an estimate for how large the entropy can still be. if the entropy can be very large, we can not claim to train a VAE, because the error going from (3) to (4) is large. Secondly, we have to ensure that both E and G have bounded entropy, because otherwise KL(r(x|z) || p(x|z) ) will be large and there is no way to that the approximation in (3) is good, at which point E can not be claimed as doing maximum-likelihood on the data.
As shown in Figure 3, the Generator has a Conv-Block and the initial layer is a linear layer. Similarly the discriminator (or E) has a linear output layer. Therefore, the entropy is not bounded as the paper does not claim to have any entropy normalization here. The paper does not say what the activation functions are, but if the final layer of E is linear or Relu, we can add yet another problem to the list (next to the unbounded Jacobian): the VAE framework stops making sense. If we look at eq (6) or (7), the final divergence term (multiplied by beta) hinges on the fact that it provides a suitable error contribution. However, if the final layer of D is linear, we can just scale that layer arbitrarily large until the error contribution of the divergence vanishes. 
Lastly, the Energy-function does not even necessarily model a probability function. Remember, we need that Z(z)=\int exp(-E(x|z)) dx <\infty. However, the definition by Figure (3) is E(x|z)=-h(x)^T(z) and the output layer of h is linear. so we can find weights such that h(x)=c \forall x. In which case 
Z(z)=\int exp(-c^Tz) dx = exp(-c^Tz) \int 1 dx 
which is only bounded if x is bounded, which I can not see since the generator is not bounded as well. This is only one of many counter-examples.
A few other small things:
- For z from a normal distribution the bottleneck discussion in 2.1 last paragraph does not apply, because for a dataset with N samples it is always possible to partition the real-space of Z into N partitions with probability 1/N each. This even works with 1-dimensional normal distributions, independently of the dimensionality of x. So a multi-modal distribution for p(x|z) is really not necessary if encoder and decoder are strong enough to memorize the dataset.
- below (5) it is claimed that this is similar to a WGAN, but WGAN have a difficult constraint to be 1-Lipschitz to make it work (which proves to be extremely difficult in practice as it translates to a similar inequality constraint on the jacobian)
- In Fig(5) it is unclear to me how a markov-chain is defined using the model. I assume this is done by taking samples from z', generating a new x and encoding this to obtain a new z. If z' models different features from z (assuming z' is multi-modal) we would assume that the z would remain the same in each point, so no exploration of p(z) takes place. However, we can already see in the lat rows that the pictures look more strange, so some mixing is taking place but it is not clear how that relates to p(z). Therefore i would really like to see images generated by sampling z~N(0,I) and z'~N(0,I) to see whether the generated images look reasonable. 
- Moreover, to show that the Generator actually produces images that are likely under p(x|z), we would also need samples generated from the true underlying distribution p(x,z), e.g. using hamilton monte-carlo.
- Finally, i have to notice that the model, while called hierarchival VAE, does not provide a way to compute z' from the input. is that than really an autoencoder?