Learning representations of data is an important issue in machine learning. Though GAN has led to significant improvements in the data representations, it still has several problems such as unstable training, hidden manifold of data, and huge computational overhead. GAN tends to produce the data simply without any information about the manifold of the data, which hinders from controlling desired features to generate. Moreover, most of GAN's have a large size of manifold, resulting in poor scalability. In this paper, we propose a novel GAN to control the latent semantic representation, called LSC-GAN, which allows us to produce desired data to generate and learns a representation of the data efficiently. Unlike the conventional GAN models with hidden distribution of latent space, we define the distributions explicitly in advance that are trained to generate the data based on the corresponding features by inputting the latent variables that follow the distribution. As the larger scale of latent space caused by deploying various distributions in one latent space makes training unstable while maintaining the dimension of latent space, we need to separate the process of defining the distributions explicitly and operation of generation. We prove that a VAE is proper for the former and modify a loss function of VAE to map the data into the pre-defined latent space so as to locate the reconstructed data as close to the input data according to its characteristics. Moreover, we add the KL divergence to the loss function of LSC-GAN to include this process. The decoder of VAE, which generates the data with the corresponding features from the pre-defined latent space, is used as the generator of the LSC-GAN. Several experiments on the CelebA dataset are conducted to verify the usefulness of the proposed method to generate desired data stably and efficiently, achieving a high compression ratio that can hold about 24 pixels of information in each dimension of latent space. Besides, our model learns the reverse of features such as not laughing (rather frowning) only with data of ordinary and smiling facial expression.
* Pros
- addresses an interesting problem
- gives a nice approach to the problem
- attempts to give some theoretical justification for the approach
* Cons
- I generally understand the approach, but details were not clear to me (specifics given below)
- Sections 3.2.1 and 3.2.2 (the theoretical section), I found particuarly hard to follow.
- The visual results are not particularly compelling, tbhough I suppose the panel liked them better than the competitor methods (Table 1).  For example "inverse pale skin" and "pale skin" in figure 5 do not convince me that the model understands skin. The skin and background seem to be changing colors together.  Might be worth including examples of the competitor approaches to show that they are even worse.
* Comments and Questions
- Throughout, you seem to assume binary-valued features, without ever explicitly stating this.  Would be helpful to state explicitly.
- Would be useful to specify the codomain of the discriminator D(x) -- from the objective function, seems to be a value in (0,1).
- In Section 3.2, you say: "Besides, we add the encoder for LSC-VAE into LSC-GAN to make sure that the generated data actually have the desired features.  The encoder projects back to latent space so as to be trained to minimize the difference between latent space where data is generated and the space where the compressed data is projected."  This seems like a fundamental change to training, much more than just initializing a GAN with a VAE-GAN. I think this should be elaborated on.  For example, what happens if you don't include the term with the encoder in (5).  Moreover, what goes wrong if you just try to train everything jointly using (5) without the VAE-GAN initialization step?
- I have a very difficult time understanding 3.2.1, both the text and the equations.  e.g. what are the z_i's in (8)?  In 3.2.1, you say "we pre-train G with the decoder of LSC-VAE" -- this "pretraining" is what you refer to as initialization previously, I think?  Which seems also what section 3.1 is about?
- I think some clarification would be nice in Section 3.2.2.  The conclusion of the section is that the "proposed learning process is valid and efficient."       What do you mean by "valid" and "efficient"?  Perhaps you can explain that a bit more in words at the beginning of the section.  It's not entirely clear where your theory section connects to the objective function in (5). I don't really follow your argument for LSC_VAE being a good initializer in this section. In what sense is theorem 2 demonstrating "efficiency"?  
- In Theorem 1, you write p_data \approx p_G.  I guess p_data is some unknown data generating distribution, rather than the empirical distribution of a training set?   I've also never seen \approx 0 used in formal mathematical statements and proofs especially when we're talking about \approx 0 at infinitely many points.  Can this be elaborated on?  
- In the end of section 3 intro paragraph you say "The decoder of LSC-VAE is used in the generator (G) of LSC-GAN in the second phase." By "used" do you mean used as the initialization of the generator G, when we switch to the LSC_GAN training?  Seems like it, but could be made more clear.
- In Section 3, second paragraph, you say "In the first phase, LSC-VAE is trained to project data into a specific location of latent space according to its features".  It's not clear whether or not this "projection step" (which I guess is also called encoding step or the inference step depending on the context?) uses the explicit feature values in this step, or can only use the input (e.g. the image).  I really have the same question for the decoder/generator: does it explicitly use the feature values, or does it depend only on the latent variables?  My guess is that in both cases the feature values are not depended on directly, but I think this could be made more clear, one way or the other.
- how did results vary when you deviate from using 20 latent dimension per feature?
- You say "As shown in Fig. 4, the change between images is natural so that we can say that the latent space of LSC-GAN is a manifold." --- maybe a linear manifold?
- Footnote 2 on page 4: This is confusing.  You are [basically arbitrarily] defining the conditional distribution on the latent space for any feature setting.  How can any particular distribution be "correct".
- In the equations in (5), you're taking expectations over z_i, but don't you need to have an expectation over i (the feature assigments) as well?  Do you use the same feature distributions as you have in the training data?  Should be clarified.
- Also, in (5), the expectation over z_i applies to the first term, as well as the z_i in G(z_i) in the second KL divergence term, right?  
- In VAE the encoder typically produces the parameters of the Q distribution on the latent space.  What distribution does Q have and how are you parameterizing it?  Indendent Gaussians on each coordinate, each with its own mean and standard deviation?  or what? If you are allowing the encoder to take the feature values as input (which I don't think you are, but am not entirely sure of), does the encoder have to learn the means for each feature setting, or are you explicitly building those feature-based offsets into the encoder?