This paper studies the undesired phenomena of over-sensitivity of representations learned by deep networks to semantically-irrelevant changes in data. We identify a cause for this shortcoming in the classical Variational Auto-encoder (VAE) objective, the evidence lower bound (ELBO). We show that the ELBO fails to control the behaviour of the encoder out of the support of the empirical data distribution and this behaviour of the VAE can lead to extreme errors in the learned representation. This is a key hurdle in the effective use of representations for data-efficient learning and transfer. To address this problem, we propose to augment the data with specifications that enforce insensitivity of the representation with respect to families of transformations. To incorporate these specifications, we propose a regularization method that is based on a selection mechanism that creates a fictive data point by explicitly perturbing an observed true data point. For certain choices of parameters, our formulation naturally leads to the minimization of the entropy regularized Wasserstein distance between representations. We illustrate our approach on standard datasets and experimentally show that significant improvements in the downstream adversarial accuracy can be achieved by learning robust representations completely in an unsupervised manner, without a reference to a particular downstream task and without a costly supervised adversarial training procedure.
This is a very interesting paper, I believe, a solid contribution to Variational Autoencoders. The basic argument is that encoders in VAEs are highly susceptible to noise in input data, whereas decoders are not. This argument is supported with a full fledged section 2.2, reformulating ELBO objective of VAEs, and introducing a VAE with discrete latent variables and discrete observations, so as to easily understand why and where VAEs fail.
To make encoders robust to noise in inputs, it is proposed to generate new fictive data points in the neighborhood of original data points so as to ensure that the latent representations of a data point and its fictive version are similar in "some sense" as part of the proposed regularization term. The implementation of this idea is solid in the paper, relating it to theoretical concepts such as  "entropy regularized entropy transport problem", "Wasserstein distance", etc. The most important point is that, it is easy to extend an encoder of an existing VAE with the proposed algorithm, while letting a decoder be untouched as the latter is shown to be robust/smooth anyways (in sec 2.2). It is also discussed on how to generate fictive samples, including but not restricted to approaches like projected gradient descent based adversarial attacks.
Section 2.2 can be improved further, in terms of presentation. This is the most important section which can be of interest to the community to understand VAEs' limitations, a good contribution on its own. Though challenging, I encourage the authors to improve the exposition in this section as much as possible.
Introduction is written beautifully. Good job, done!
For instance, some explanation about variables, m_j, u_i, their distribution.
How do you relate the Eq. 1 with the standard ELBO. (some reference to derivation?)
Is it not possible to explain limitations of present VAEs without introducing the particular von Mise like parameterization (last equation of page 3). I am not suggesting that you should remove it. The connections between the two could be more explicit, though I understand that it is already mentioned in the paper, "parameterization emulates a high capacity network that can model any functional relationship between latent states and observations...". 
In this context, I found the explanation after Eq. 2 to be intuitive in regards to inefficiency of encoders. If I understand correctly, to put it in even simpler terms, the encoding neural network is overfitting mapping from input data points to the latent representations, not performing any learning for the unseen data points at all; on the other hand, decoder explores the space of latent variables well because it is modeled as a Gaussian?
Some of the new equations should be numbered for easy reference. 
On page 4, the flow is a bit abrupt. Right after Fig. 3, there are points 1 and 2 added without any note on what these two points (items in latex) are about.
I found point 1 very confusing in page 4. On the other hand, point 2 is beautifully written. Though, it could be made explicit in the latter on why encoders found in VAE are not smooth, referring to Fig 2, 3.
There are minor grammar mistakes making some of sentences incoherent or confusing, in the paper. Something to do with style of language. I think, overall, language can be improved. Though, technical flow of the paper is great, and introduction is written very well, pointing out very important bold insights about the literature on unsupervised representation learning. I would say, it is a very well written paper, which is an enjoyable read, despite some of the grammar mistakes which can be easily fixed by proof reading.
Experimental evaluation is sufficient. 
Last but not the least, one could argue that we are going to the literature of kernel function based methods, or markov random fields, to improve the neural network models. This is a general trend we are observing. It is interesting to see new models such as the proposed one, getting the best from both worlds. It may be worthwhile to point  out something along these lines in the paper so that other works like this can be accomplished which are bold, and advance representation learning, digging mathematical concepts from diverse domains. If I am mistaken, please feel free to point out. It is not going to be change the review. I am inspired from this work.
 
One practical challenge is to generate fictive data points which are not very near to existing data points. I am not sure if GANs can achieve that, either. Having such points is critical to deal with more structured noise. Any comments on this?