For distributions p and q with different supports, the divergence D(p||q) may not exist. We define a spread divergenceD(p||q) on modified p and q and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a spread divergence to train and improve implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).
The paper introduced a way to modify densities such that their support agrees and that the Kullback-Leibler divergence can be computed without diverging. Proof of concept of using the spread KL divergence to ICA and Deep Generative Models ($\delta$-VAE) are reported based on the study of spread MLE.
Comments:
In Sec 1, mention that f should be strictly convex at 1. Also mention 
Jensen-Shannon divergence, a KL symmetrization, which is always finite 
and used in GAN analysis.
In Sec 2, you can also choose to dilute the densities with a mixture: 
(1-\epsilon)p+\epsilon noise.
Explain why spread is better than that? Does spreading introduce 
spurious modes?, does it change distribution sufficiency? 
(Fisher-Neymann thm)
In Formula 4, there is an error: missing denominator of \sigma^2. See 
Appendix D too.
In footnote 4, page 8, missing a 1/2 factor in from of TV (that is upper 
bounded by 1 and not 2)
KL is relative entropy= cross-entropy minus entropy. What about spread KL?
In general, what statistical properties are kept by using the spread? 
(or its convolution subcase?)
Is spreading a trick that introduces a hyperparameter that can then be 
optimized for retaining discriminatory power, or is there
some deeper statistical theory to motivate it. I think spread MLE should 
be further explored and detailed to other scenarii.
Spreading can be done with convolution and in general by Eq.3:
Then what is the theoretical interpretation of doing non-convolutional 
spreading?
A drawback is that optimization on the spread noise hyperparameter is 
necessary (Fig 3b is indeed much better than Fig 3a).
Is there any first principles that can guide this optimization rather 
than black-box optimization?
Overall, it is a nice work but further statistical guiding principles 
or/and new ML applications of spread divergences/MLE will strengthen the 
work.
The connection, if any, with Jensen-Shannon divergence shall be stated 
and explored.
Minor comments:
In the abstract, state KL divergence instead of divergence because 
Jensen-Shannon divergence exists always.
Typos:
p. 6 boumd->bound
Bibliography : Cramir->Cramer, and various upper cases missing (eg. 
wasserstein ->Wasserstein)