Structured Generative Adversarial Networks
We study the problem of conditional generative modeling based on designated semantics or structures. Existing models that build conditional generators either require massive labeled instances as supervision or are unable to accurately control the semantics of generated samples. We propose structured generative adversarial networks (SGANs) for semi-supervised conditional generative modeling. SGAN assumes the data x is generated conditioned on two independent latent variables: y that encodes the designated semantics, and z that contains other factors of variation. To ensure disentangled semantics in y and z, SGAN builds two collaborative games in the hidden space to minimize the reconstruction error of y and z, respectively. Training SGAN also involves solving two adversarial games that have their equilibrium concentrating at the true joint data distributions p(x, z) and p(x, y), avoiding distributing the probability mass diffusely over data space that MLE-based methods may suffer. We assess SGAN by evaluating its trained networks, and its performance on downstream tasks. We show that SGAN delivers a highly controllable generator, and disentangled representations; it also establishes start-of-the-art results across multiple datasets when applied for semi-supervised image classification (1.27%, 5.73%, 17.26% error rates on MNIST, SVHN and CIFAR-10 using 50, 1000 and 4000 labels, respectively). Benefiting from the separate modeling of y and z, SGAN can generate images with high visual quality and strictly following the designated semantic, and can be extended to a wide spectrum of applications, such as style transfer.
Summary:
This paper proposes a novel GAN structure for semi-supervised learning, a setting in which there exist a small dataset with class labels along with a larger unlabeled dataset. The main idea of this paper is to disentangle the labels (y) from the hidden states (z) using two GAN problems that represent p(x,y) and p(x,z). The generator is shared between both GAN problems, but each problem is trained simultaneously using ALI[4]. There are two adversarial games defined for training the joints p(x, y) and p(x, z). Two "collaborative games" are also defined in order to better disentangle y from z and enforce structure on y.
Pros:
1) Overall, the paper is written well. All the steps are motivated intuitively. The presented ideas look rational.
2) The presented results on the semi-supervised learning task (Table 2) are very impressive. The proposed method outperforms previous GAN and VAE works. The qualitative results in terms of generative samples look very interesting.  
Cons:
1) I found the mutual predictability measure very ad-hoc. There exist other measures such as mutual information that have been used in past (e.g. infoGAN) for measuring disentanglability of DGMs.
2) The collaborative games presented in Eq.4 and Eq.5 is equivalent to the mutual information term introduced in infoGAN[2] (with \lambda=1). 
In this case, the paper can be considered as the combination of infoGAN and ALI applied to semi-supervised learning. This slightly limits the technical novelty of the paper.
Questions:
1) I am wondering why the adversarial game L_{xy} is only formed on the labeled set X_l. A similar game could be created on the unlabeled set. The only difference would be that instead of labels, one would use y=C(x) for the first term in Eq.3. However in this paper, instead, a collaborative game R_y is introduced.
2) Theorem 3.3 does not state whether minimizing R_z or R_y w.r.t G will change the equilibrium of the adversarial game or not. In other words, with minimization over G in both Eq.4 and Eq.5, can we still guarantee that G will match the data distribution. 
Minor comments:
1) line 110: Goodfellow et al is missing the reference.
2) line 197: Is l-2 distance resulted from assuming a normal prior on z?
3) line 213: p_g(z|y)  --> p_g(x|y)
4) With all the different components (p, p_g, p_c, etc) it is a bit challenging to follow the structure of adversarial and collaborative games. A figure would help readers to follow the logic of the paper easier.