Guarantees in deep learning are hard to achieve due to the interplay of flexible modeling schemes and complex tasks. Invertible neural networks (INNs), however, provide several mathematical guarantees by design, such as the ability to approximate non-linear diffeomorphisms. One less studied advantage of INNs is that they enable the design of bi-Lipschitz functions. This property has been used implicitly by various works to design generative models, memory-saving gradient computation, regularize classifiers, and solve inverse problems. In this work, we study Lipschitz constants of invertible architectures in order to investigate guarantees on stability of their inverse and forward mapping. Our analysis reveals that commonly-used INN building blocks can easily become noninvertible, leading to questionable "exact" log likelihood computations and training difficulties. We make use of numerical analysis tools to diagnose non-invertibility in practice. Finally, based on our theoretical analysis, we show how to guarantee numerical invertibility for one of the most common INN architectures.
The paper claims that for invertible neural networks, mathematical guarantees on invertibility is not enough, and we also require numerical invertibility. To this end, the lipschitz constants/condition numbers of Jacobians of both the forward and inverse maps of invertible NNs based on coupling layers are examined mathematically and experimentally. The paper also displays cases that expose non-invertibility in these architectures via gradient-based construction of adversarial inputs, as well as a decorrelation benchmark task, and show that spectral normalization can be a remedy for stabilizing these flows.
I think it’s a good point that we need to monitor the Lipschitz constant/bounds of both directions of these invertible functions. It’s true that the focus for stabilising NNs by bounding Lipschitz constants was always on the forward function, and for invertible functions we should also ensure that the inverse is numerically stable to compute.
The mathematical contribution of the paper is twofold - 1. deriving bounds on the lipschitz constants of the forward and inverse mapping of additive/affine coupling blocks 2. summarising known lipschitz bounds of forward and inverse mappings of other invertible layers (iResNet, neuralODE, invertible 1x1 convolutions etc). The main contribution lies in 1, and the derivation for the additive coupling block (volume preserving) is neat (although fairly straightforward), but the derivation for the affine coupling layer (NVP) is not useful nor insightful; they are local Lipschitz bounds (so require bounds on all intermediate activations, which is difficult as pointed out by the authors), and the numerical value of this bound was not used at all in relation to the numerical experiments - I imagine the bound is loose. Given that it seems difficult to find a tight global lipschitz bound, I think it would be more insightful to compute a lower bound to the lipschitz constant of the model (with fixed parameter values) by maximising the spectral norm of the Jacobian with respect to the inputs (or outputs if looking at the inverse map) - this will yield a lower bound by Lemma 3. This will be numerical, but more informative since it will give you an indication of where in the input space (or output space if looking at the inverse) there could be numerical instabilities. Also I think the bound on the local lipschitz constant of the inverse for the affine coupling block might be incorrect, because in A.1.1, the inverse map is F^{-1}(y)_I1 = y_I1, F^{-1}(y)_I2 = (y_I2 - t(y_I1))/g(s(y_I1)), so the scale and shift is s’(y_I1) := 1/g(s(y_I1)) and t’(y_I1):=- t(y_I1)/g(s(y_I1)), and hence I think this needs to be taken into account for computing the lipschitz bound of the inverse 
I have mixed feelings about the experimental section. In section 4.1, it is interesting to see that we can find inputs where trained flow models can show numerical non-invertibility, evident in the poor reconstructions. It would be a nice addition to investigate whether this is coming from the forward function or its inverse, by examining the norm of the Jacobian of F and F^{-1} at the input x_delta and output F(x_delta) respectively. 
However, the decorrelation task introduced in section 4.2 is puzzling. I don’t understand why for these invertible models, you are investigating invertibility for parameter values trained to decorrelate, as opposed to parameter values used in the usual task of density estimation with flows (or any other standard application of invertible NNs). The two reasons given in the paper are that 1) decorrelation is a simpler task and 2) it allows both stable and unstable transforms as solutions, but these are not convincing. Point 2) holds for flow-based density estimation as well, and regarding point 1), density estimation is the task we usually care about when using invertible NNs, and this is also computationally plausible/tractable, whereas even if decorrelation is a simpler task, it’s not a task that users of invertible NNs are interested in. It is good to know that these invertible NN architectures CAN admit values that are numerically non-invertible, but I would be much more interested to know whether this actually holds when they have been trained for flow-based density estimation. I’m not sure whether the experimental results on models trained for the decorrelation task are useful, because a model that is stable when trained for the decorrelation task may be unstable when trained for flow-based density estimation and vice versa. The observation that spectral normalization can help address numerical instability is useful, but from the perspective of someone who wants to use these invertible NNs for density estimation, I would like to know what is the sacrifice in expressivity/validation performance (if any) when using spectral normalization in these invertible architectures. Also, the results would be more relevant if the architectures resembled the architectures used for invertible models used in the literature (e.g. GLOW) where we not only have coupling layers but they are interleaved with PLU linear flows.
In section 5, the result that Flow-GANs can be numerically non-invertible is more relevant, and it is useful to know that spectral normalisation can help resolve this issue, but again it would be useful to quantify whether this comes at the cost of the quality of generated samples (Figure 3 shows several samples, but a more thorough quantitative & qualitative comparison would be welcome). Also regarding the point about likelihood in Section 5, where the authors state “it cannot be trusted as true likelihood due to lack of invertibility”, I think it should be emphasised that this point holds specifically for flow-GANs where for F: z -> x, you need a numerically accurate F^{-1} to compute the density, but for standard flow-based density estimation where F:x -> z, you never need to compute the inverse for computing the likelihood, hence if F has a small lipschitz constant then the likelihood will be accurate, regardless of whether the inverse is numerically stable or not.
Overall I believe the experimental section can be largely improved, and given that the motivation of the paper is nice and the paper is clearly written and nicely presented, it would be a shame to leave the experiment section as it is.
Minor typos/Qs:
p2: this problems <- this problem
p8: and with maximum likelihood (ML) - should this be removed?
p13: t(x_I2) <- t(x_I1)