Workshop track -ICLR 2018 STABLE DISTRIBUTION ALIGNMENT USING THE DUAL OF THE ADVERSARIAL DISTANCE
Methods that align distributions by minimizing an adversarial distance between them have recently achieved impressive results. However, these approaches are difficult to optimize with gradient descent and they often do not converge well without careful hyperparameter tuning and proper initialization. We investigate whether turning the adversarial min-max problem into an optimization problem by replacing the maximization part with its dual improves the quality of the resulting alignment and explore its connections to Maximum Mean Discrepancy. Our empirical results suggest that using the dual formulation for the restricted family of linear discriminators results in a more stable convergence to a desirable solution when compared with the performance of a primal min-max GAN-like objective and an MMD objective under the same restrictions. We test our hypothesis on the problem of aligning two synthetic point clouds on a plane and on a real-image domain adaptation problem on digits. In both cases, the dual formulation yields an iterative procedure that gives more stable and monotonic improvement over time.
This paper proposes to re-formulate the GAN saddle point objective (for a logistic regression discriminator) as a minimization problem by dualizing the maximum likelihood objective for regularized logistic regression (where the dual function can be obtained in closed form when the discriminator is linear). They motivate their approach by repeating the previously made claim that the naive gradient approach is non-convergent for generic saddle point problems (Figure 1); while a gradient approach often works well for a minimization formulation.
The dual problem of regularized logistic regression is an entropy-regularized concave quadratic objective problem where the Hessian is y_i y_j <x_i, x_j>, thus highlighting the pairwise similarities between the points x_i & x_j; here the labels represent whether the point x comes from the samples A from the target distribution or B from the proposal distribution. This paper then compare this objective with the MMD distance between the samples A & B. It points out that the adversarial logistic distance can be viewed as an iteratively reweighted empirical estimator of the MMD distance, an interesting analogy (but also showing the limited power of the adversarial logistic distance for getting good generating distributions, given e.g. that the MMD has been observed in the past to perform poorly for face generation [Dziugaite et al. UAI 2015]). From this analogy, one could expect the method to improves over MMD, but not necessarily significantly in comparison to an approach which would use more powerful discriminators.
This paper then investigates the behavior of this adversarial logistic distance in the context of aligning distributions for domain adaptation, with experiments on a visual adaptation task. They observe better performance for their approach in comparison to ADDA, improved WGAN and MMD, when restricting the discriminators to be a linear classifier.
== Evaluation 
I found this paper quite clear to read and enjoyed reading it. The observations are interesting, despite being on the toyish side. I am not an expert on GANs for domain adaptation, and thus I can not judge of the quality of the experimental comparison for this application, but it seemed reasonable, apart for the restriction to the linear discriminators (which is required by the framework of this paper).
One concern about the paper (but this is an unfortunate common feature of most GAN papers) is that it ignores the vast knowledge on saddle point optimization coming from the optimization community. The instability of a gradient method on non-strongly convex-concave saddle point problems (like the bilinear form of Figure 1) is a well-known property, and many alternative *stable* gradient based algorithms have been proposed to solve saddle point problems which do not require transforming them to a minimization problem as suggested in this paper. Moreover, the transformation to the minimization form crucially required the closed form computation of the dual function (with w* just defined above equation (2)), and this is limited to linear discriminators,  thus ruling out the use of the proposed approach to more powerful discriminators like deep neural nets. Thus the significance appears a bit limited to me.
== Other comments
1) Note that d(A, B'_theta) is *equal* to min_alpha max_w  (...)  above equation (2) (it is not just an upper bound). This is a standard result coming from the fact that the Fenchel dual problem to regularized maximum likelihood is the maximum entropy problem with a quadratic objective as (2).  See e.g. Section 2.2 of [Collins et al. JMLR 2008] (this is for the more general multiclass logistic regression problem, but (2) is just the binary special case of equation (4) in the [Collins ... ] reference). And note that the "w(u)" defined in this reference is the lambda*w*(alpha) optimal relationship defined in this paper (but without the 1/lambda factor because of just slightly different writing; the point though is that strong duality holds there and thus one really has equality).
[Collins et al. JMLR 2008] Michael Collins, Amir Globerson, Terry Koo , Xavier Carreras, Peter L. Bartlett, Exponentiated Gradient Algorithms for Conditional Random Fields and Max-Margin Markov Networks, , JMLR 2008.
 [Dziugaite et al. UAI 2015] Gintare Karolina Dziugaite, Daniel M. Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In UAI, 2015