In Variational Auto-Encoder (VAE), the default choice of reconstruction loss function between the decoded sample and the input is the squared L 2 . We propose to replace it with the log hyperbolic cosine (log-cosh) loss, which behaves as L 2 at small values and as L 1 at large values, and differentiable everywhere. Compared with L 2 , the log-cosh loss improves the reconstruction without damaging the latent space optimization, thus automatically keeping a balance between the reconstruction and the generation. Extensive experiments on MNIST and CelebA datasets show that the log-cosh reconstruction loss significantly improves the performance of VAE and its variants in output quality, measured by sharpness and FID score. In addition, the gradient of the log-cosh is a simple tanh function, which makes the implementation of gradient descent as simple as adding one sentence in coding.
The paper proposes a new loss function which can be used in the reconstruction term of various auto-encoder architectures. The pixel-wise cost function \ell(X, X') = f(X - X'; a) is defined for pairs of two input images X and X' and has one positive real-valued hyperparameter a. For small values of t the function f(t; a) behaves like a quadratic function, while for large t it behaves like |t|. As a consequence, it is smooth, everywhere differentiable (like L2) while not penalizing outliers too hard (like L1). The authors present several experiments conducted on MNIST and Celeba datasets, demonstrating that a simple change of a conventional pixel-wise squared L2 distance with the proposed log-cosh cost function improves the FID scores of generated samples as well as the visual quality of reconstructions (including "the sharpness"). 
I would say this is clearly an empirical study (even though the authors claim they provide "theoretical justifications", they are rather hand wavy), which is not a bad thing in this case. The message of the paper is very clear and I think the authors did a good job in selling their point. The main (and, perhaps, the only) contribution is the proposal to use the log-cosh function as the reconstruction cost. And this proposal is well justified by the set of experiments. 
However, there are several major issues:
(1.1) The objective functions reported in appendix A.1 corresponding to WAE have in fact nothing to do with WAE. In WAE the regularizer penalizes the divergence between the prior distribution p(a) and *the aggregated posterior* distribution \int_x q(z|x) p(x) dx. In other words, D_MMD(q(z|x) || p(z)) in Eq. 8 should be replaced with D_MMD(\int_x q(z|x) p(x) dx || p(z)) in order to result in the WAE model. In summary, if the authors indeed used objectives reported in Eq. 8 of Appendix A, they were actually not using WAE but rather some other sort of regularized auto-encoders, which in a way are quite similar to VAEs. 
(1.2)  I am surprised to see the reported FID scores for the Celeba data set. Having worked with this data set myself in combination with VAEs and WAEs, I am impressed with the extremely low FID scores: 46 for the vanilla L2 VAE and 30 for the L2 WAE. Note that while in the appendix the authors say they follow the architectural choices provided in [1] while performing the "L2 WAE Celeba" experiment, the authors arrive at FID=30 compared to FID=55 reported in the "Wasserstein Autoencoders" paper. Also, based on my experience, achieving FID=46 on CelebA with a vanilla VAE is very impressive. Note that the authors use 10^4 of samples to evaluate the FID scores, which is exactly the same as in [1]. This size is known to be large enough to reduce the variance of FID, so the difference (55 - 30) can not be explained by the fluctuations of FID. Therefore, I ask the authors to (anonymously) share the code and/or checkpoints of the 2 particular trained models: L2 VAE and L2 WAE trained on Celeba. 
Other comments:
(2.1) Note that the reconstruction cost function in VAE should be normalized for every value of the code Z, as it corresponds to the logarithm of the likelihood (density) function -log p(X|Z). L2 and L1 costs both correspond to the well known likelihood (decoder) models (Gaussian and Laplace). However, it is hard to say what decoder model (what type of conditional distribution p(X|Z) ) would give rise to the proposed log-cosh function. In particular, the normalizing constant is not known and may depend on Z. In other words, by exchanging the L2 cost with the log-cosh loss in the VAE one looses the theoretical guarantees supporting VAE, including the fact that the objective is the lower bound on the marginal log likelihood. While this is not necessarily a problem (unless one uses the value of the objective as the bound on the marginal log likelihood, which is not the case in this paper), I would suggest mentioning it. Notice that, for instance, in WAE this problem does not appear, as the reconstruction term there does not involve any likelihood functions and thus does not need to be normalized.
(2.2) In Figure 2 I don't see why the authors did not highlight bad samples in the second row corresponding to their proposed method? I see many badly looking images there. Say, (4, 9) in VAE (MLP) and (8, 9) in VAE (Conv) and (6, 1) in WAE (MLP) and (2, 10) in WAE (Conv), where (i, j) means i-th row, j-th column, indexing starting from 1. 
(2.3) How would the Huber loss perform and how does it compare to the proposed loss?
[1] Wasserstein Autoencoders. Tolstikhin et al., ICLR, 2018.