Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poorer sample quality. Recent works have proposed hybrid VAE-GAN frameworks which integrate a GAN-based synthetic likelihood to the VAE objective to address both the mode collapse and sample quality issues, with limited success. This is because the VAE objective forces a trade-off between the data log-likelihood and divergence to the latent prior. The synthetic likelihood ratio term also shows instability during training. We propose a novel objective with a "Best-of-Many-Samples" reconstruction cost and a stable direct estimate of the synthetic likelihood. This enables our hybrid VAE-GAN framework to achieve high data log-likelihood and low divergence to the latent prior at the same time and shows significant improvement over both hybrid VAE-GANS and plain GANs in mode coverage and quality.
“Best of Many Samples” Distribution matching
Summary:
This paper proposes to a novel VAE-GAN hybrid which, during training, draws multiple samples from the reparameterized latent distribution for each inferred q(z|x), and only backpropagates reconstruction error for the resulting G(z) which has the lowest reconstruction. The authors appear to use the AAE method instead of analytically matching KL(p||q) for enforcing that the latents q(z|x) match the prior. The authors present results on MoG toy datasets, CIFAR-10, and CelebA, and compare  against several other models.
My take:
The idea in this paper is moderately interesting, well-founded, has plenty of precedent in the literature (while still being reasonably novel),  but the results present only a minimal improvement (a 5% relative improvement in FID over the baseline model from Rosca et al on CIFAR, especially when including SN [which is not a contribution of this paper]) and come at a substantial compute cost, requiring up to 30 extra samples per batch in order to attain this minimal increase. While I think the idea is interesting, the change in results over Rosca et. al does not seem to justify its increased computational expense (which is also not characterized in sufficient thoroughness). I am pretty borderline on this paper ( about a 5/10) but under the 1-3-6-8 scoring constraint I tend to lean reject because while I like the idea, I do not think the results are significant enough to support its adoption; I think the relative compute and implementation cost limit this method’s potential impact. I am keen to discuss this paper with the other reviewers.
Notes:
-The results on the 2D MoG toy datasets are good but are also suspect—the authors state that they use a 32-dimensional latent space, but the original code provided for VEEGAN uses a 2-dimensional latent space. The authors should re-run the experiment for BMS-VAE-GAN  using a 2D latent space (this should be very easy and take less than an hour on a GPU to get several runs in).
-“again outperforming by a significant margin (21.8 vs 22.9 FID)” This is not a significant margin; this is less than a 5% margin and, at those FID scores, represents an imperceptible change in sample quality.
-The authors seem to suggest that applying spectral norm to the GAN of Rosca et. al. is somehow a contribution (e.g. having “ours” next to this model in the tables); I would advise against even appearing to suggest this as it is clearly not a contribution.
-Characterize the increase in compute cost. “. We use as many samples during training as would fit in GPU memory so that we make the same number of forward/backward passes as other approaches and minimize the computational overhead of sampling multiple samples” is a qualitative description; I would like to see this quantitatively described. How do the runtimes differ between your baseline and the T=10 and T=30 runs? If they don’t differ, why? Are the authors e.g. reducing the batch size by a factor of 10 or 30 to make this computationally tractable?
-The latent space discriminator D_L should be referred to in section 3; its formal introduction is deferred to later in the paper, hampering the presentation and flow. 
-CelebA is not multimodal; it is in fact, highly constrained, and primarily only has textural variation (virtually no pose variation).
-ALI and BiGAN are listed under Hybrid VAE-GANs. These models are not VAE-GAN hybrids. Additionally, this section states that BiGAN builds upon ALI. This is not true, these papers are in fact proposing the same thing and were released at nearly the exact same time.  Do not mischaracterize or incorrectly summarize papers. Please re-read both papers and refer to them correctly.
-Mode collapse (when many points in z map to an unexpectedly small region in G(z)) is a different phenomenon from mode dropping (when many points in x are not represented in G(z), i.e. no point in z maps to a cluster of x’s, as is the case if e.g. a celebA model generates frowning and neutral faces but no smiling faces). While these phenomena often co-occur (especially during complete training collapse), they are not the same thing, and this paper conflates them in several places.
Minor:
Section 3, paragraph 2: “The GAN (Gθ,DI…” There’s a close parenthesis missing here. 
Section 3.3: “The network is traiend…” 
Please thoroughly proofread your paper for typos and grammatical mistakes.