summaries: The paper uses Generative Adversarial Networks (GAN) for unsupervised and semi-supervised clustering. Neural network based generators are used for sampling using a mixture model. The parameters of the generators are optimised during training against a discriminator that tries to distinguish between generated distributions. Experimental results on MNIST and SVHN datasets are given to motivate their models. The paper presents a generative model that can be used for unsupervised and semi-supervised data clustering. unlike most of previous method  the latent  variable is composed of both continuous and discrete variables. Unlike previous methods like ALI the conditional probability  p(y|x) of the labels given the object is represented by a neural network and not simply drown from the data.  The  authors show a  clustering error rate on the MNIST data that is better than previously proposed methods. 
abstractText: The Adversarially Learned Mixture Model (AMM) is a generative model for unsupervised or semi-supervised data clustering. The AMM is the first adversarially optimized method to model the conditional dependence between inferred continuous and categorical latent variables. Experiments on the MNIST and SVHN datasets show that the AMM allows for semantic separation of complex data when little or no labeled data is available. The AMM achieves unsupervised clustering error rates of 3.32% and 20.4% on the MNIST and SVHN datasets, respectively. A semi-supervised extension of the AMM achieves a classification error rate of 5.60% on the SVHN dataset.
references.authors: Xi Chen
references.title: InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems
references.authors: Zihang Dai Zhilin Yang
references.title: Good semi-supervised learning that requires a bad GAN
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Zhijie Deng Hao Zhang
references.title: Structured generative adversarial networks
references.venue: In Advances in Neural Information Processing Systems
references.authors: Nat Dilokthanakul Pedro A.M. Mediano
references.title: Deep unsupervised clustering with Gaussian mixture variational autoencoders
references.venue: arXiv preprint arXiv:1611.02648,
references.authors: Jeff Donahue Philipp Krähenbühl Trevor Darrell
references.title: Adversarial feature learning
references.venue: In International Conference on Learning Representations,
references.authors: Vincent Dumoulin Ishmael Belghazi
references.title: Adversarially learned inference
references.venue: In International Conference on Learning Representations,
references.authors: Zhe Gan Liqun Chen
references.title: Triangle generative adversarial networks
references.venue: In Advances in Neural Information Processing Systems
references.authors: Ian Goodfellow Jean Pouget-Abadie
references.title: Generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems
references.authors: Ishaan Gulrajani Faruk Ahmed
references.title: Improved training of Wasserstein GANs
references.venue: arXiv preprint arXiv:1704.00028,
references.authors: Weihua Hu Takeru Miyato
references.title: Learning discrete representations via information maximizing self-augmented training
references.venue: arXiv preprint arXiv:1702.08720,
references.authors: Eric Jang Shixiang Gu Ben Poole
references.title: Categorical reparameterization with Gumbel-softmax
references.venue: arXiv preprint arXiv:1611.01144,
references.authors: Zhuxi Jiang Yin Zheng
references.title: Variational deep embedding: An unsupervised and generative approach to clustering
references.venue: In Proceedings of the 26th International Joint Conference on Artificial Intelligence,
references.authors: Alex Kendall Yarin Gal
references.title: What uncertainties do we need in Bayesian deep learning for computer vision
references.venue: In Advances in Neural Information Processing Systems
references.authors: Ozsel Kilinc Ismail Uysal
references.title: Learning latent representations in neural networks for clustering through pseudo supervision and graph-based activity regularization
references.venue: arXiv preprint arXiv:1802.03063,
references.authors: Diederik Kingma Max Welling
references.title: Auto-encoding variational Bayes
references.venue: In International Conference on Learning Representations,
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Diederik P Kingma Shakir Mohamed
references.title: Semi-supervised learning with deep generative models
references.venue: In Advances in Neural Information Processing Systems
references.authors: Chongxuan Li Kun Xu Jun Zhu Bo Zhang
references.title: Triple generative adversarial nets
references.venue: arXiv preprint arXiv:1703.02291,
references.authors: Lars Maaløe Marco Fraccaro Ole Winther
references.title: Semi-supervised generation with cluster-aware generative models
references.venue: arXiv preprint arXiv:1704.00637,
references.authors: Chris J. Maddison Andriy Mnih Yee Whye Teh
references.title: The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
references.venue: In International Conference on Learning Representations,
references.authors: Alireza Makhzani Jonathon Shlens
references.title: Adversarial autoencoders
references.venue: In International Conference on Learning Representations,
references.authors: Takeru Miyato Shin-ichi Maeda
references.title: Virtual adversarial training: a regularization method for supervised and semi-supervised learning
references.venue: arXiv preprint arXiv:1704.03976,
references.authors: Yuval Netzer Tao Wang
references.title: Reading digits in natural images with unsupervised feature learning
references.venue: In NIPS Workshop on Deep Learning and Unsupervised Feature Learning,
references.authors: Kevin Roth Aurelien Lucchi
references.title: Stabilizing training of generative adversarial networks through regularization
references.venue: In Advances in Neural Information Processing Systems
references.authors: Tim Salimans Ian Goodfellow
references.title: Improved techniques for training GANs
references.venue: In Advances in Neural Information Processing Systems
references.authors: Jost Tobias Springenberg
references.title: Unsupervised and semi-supervised learning with categorical generative adversarial networks
references.venue: arXiv preprint arXiv:1511.06390,
sections.heading: 1 INTRODUCTION
sections.text: Semi-supervised or unsupervised representation learning enables the utilization of all available data when tackling problems where there are little or no labeled examples. This is a common scenario in many applications of machine learning, such as medical image analysis, where it is reinforced by the expense of obtaining expert labeled examples. Moreover, machine-learned representations are more likely to be used for subsequent tasks if they are interpretable and meaningful. Deep generative modelling is a suitable approach to this problem, as derived models have been shown capable of learning from both labeled and unlabeled examples, embedding data according to desired latent variable distributions, and producing realistic data examples generated from samples of those latent variables.
  The Generative Adversarial Network (GAN) has recently emerged as a powerful framework for modeling complex data distributions without having to approximate intractable likelihoods. In the formulation by Goodfellow et al. (2014), a GAN consists of two networks: a generator G that is trained to yield unique samples from the data distribution, and a discriminator D that is trained to distinguish between generated and true data samples.
  Dumoulin et al. (2016) and Donahue et al. (2017) have proposed the ALI and BiGAN models that add an inference process, i.e., the ability to map data samples to points in the latent space, to the GAN framework. A second generator for inference, or encoder, is added to the original GAN generator and the discriminator is adapted for the two-dimensional space of data inputs and latent representations. A variant of the resulting model is also introduced by Dumoulin et al. (2016) for conditional data generation, but still assumes that the class of the data is always observed, as inference of categorical variables is not included.
  Adversarial approaches for the inference of both continuous and categorical variables are actively researched. Chen et al. (2016) introduce a hybrid adversarial method that is capable of modelling both continuous and categorical latent variables for unsupervised clustering and feature disentanglement. Another hybrid adversarial method is introduced by Makhzani et al. (2016) where adversarial objectives on continuous and categorical latent variables are optimized for unlabeled examples and categorical cross entropy on categorical variables is optimized for labeled examples. Li et al. (2017) and Deng et al. (2017) point toward fully adversarial semi-supervised classification using inferred categorical variables by introducing a “three player” adversarial game, but stop short by adding auxiliary “collaborative” objectives. In each of these methods, it is assumed that categorical and continuous latent variables are independently distributed. This independence assumption results in discontinuities in the latent space between categories, which removes the notion of inter-categorical proximity.
  Another notable family of generative models, Variational Autoencoders (VAEs), maximize the posterior distribution of latent representations given the data instead of using an adversarial approach. As VAEs integrate inference, semi-supervised classification can be performed by conditioning the
  continuous latent variable of the VAE on the class label (Kingma et al., 2014; Dilokthanakul et al., 2016; Maaløe et al., 2017). However, the quality of VAE results depend on the expressiveness of the inference distribution and every time the assumptions about the inference or data distributions are changed a new objective function needs to be derived. In this way, variational optimization is not as versatile as adversarial training.
  We present the Adversarially Learned Mixture Model (AMM). The AMM is, to our knowledge, the first generative model inferring both continuous and categorical latent variables to perform either unsupervised or semi-supervised clustering of data using a single adversarial objective. This is enabled, in part, by explicitly modelling the dependence between continuous and categorical latent variables, which eliminates discontinuities between categories in the latent space. Semi-supervised clustering and classification is enabled by a simplified formulation of the “three player game”, presented by Li et al. (2017). In this paper we show that the AMM achieves an unsupervised clustering error rate of 3.32% and 20.4% on the MNIST (LeCun & Cortes, 2010) and SVHN datasets (Netzer et al., 2011) respectively, and that a semi-supervised extension, SAMM, achieves a classification error rate of 5.60% on the SVHN dataset . To support the reproducibility of the methodology and experiments presented, a public version of the code will be made available.
sections.heading: 2 METHOD
sections.heading: 2.1 PRELIMINARIES
sections.text: The ALI and BiGAN models are trained by matching two joint distributions of images x∈RD and their latent code z ∈RL. The two distributions to be matched are the inference distribution q(x,z) and the synthesis distribution p(x,z), where,
  q(x,z) = q(x)q(z |x), (1) p(x,z) = p(z)p(x |z). (2)
  Samples of q(x) are drawn from the training data and samples of p(z) are drawn from a prior distribution, usuallyN (0,1). Samples from q(z |x) and p(x |z) are drawn from neural networks that are optimized during training. Dumoulin et al. (2016) show that sampling from q(z |x)=N (µ(x),σ2(x)I) is possible by employing the reparametrization trick (Kingma & Welling, 2013), i.e. computing
  z=µ(x)+σ(x) , ∼N (0,I), (3)
  where is element wise vector multiplication. A conditional variant of ALI has also been explored by Dumoulin et al. (2016) where an observed class-conditional categorical variabley has been introduced. The joint factorization of each distribution to be matched are:
  q(x,y,z) = q(x,y)q(z |y,x), (4) p(x,y,z) = p(y)p(z)p(x |y,z). (5)
  Samples of q(x,y) are drawn from the data. Samples of p(z) are drawn from a continuous prior on z, and samples of p(y) are drawn from a categorical prior on y, both of which are marginally independent. Samples from q(z | y,x) and p(x | y,z) are drawn from neural networks that are optimized during training.
  In the following sections we present graphical models for q(x,y,z) and p(x,y,z) that build off of conditional ALI. Where conditional ALI requires the full observation of categorical variables, the models we present will account for both unobserved and partially observed categorical variables. We finally show how they can be optimized using a single adversarial objective.
sections.heading: 2.2 ADVERSARIALLY LEARNED MIXTURE MODEL
sections.text: The AMM is an adversarial generative model for deep unsupervised clustering of data. Figure 1 presents an overview of the model.
  Like conditional ALI, a categorical variable is introduced to model the labels. However, the unsupervised setting now requires a different factorization of the inference distribution in order to
  enable inference of the categorical variable y, namely:
  q1(x,y,z)=q(x)q(y |x)q(z |x,y), (6) or q2(x,y,z)=q(x)q(z |x)q(y |x,z). (7) Samples of q(x) are drawn from the training data, and samples from q(y |x), q(z |x,y) or q(z |x), q(y |x,z) are generated by neural networks. The reparametrization trick is not directly applicable to discrete variables and multiple methodologies have been introduced to approximate categorical samples (Jang et al., 2016; Maddison et al., 2017). We follow Kendall & Gal (2017) and sample from q(y |x) by computing
  hy(x) = µy(x)+σy(x) , ∼N (0,I), (8) y(x) = softmax(hy(x)). (9)
  Then, we can sample from q(z |x,y) by computing z(x,hy(x))=µz(x,hy(x))+σz(x,hy(x)) , ∼N (0,I). (10)
  A similar sampling strategy can be used to sample from q(y |x,z) in (7). The factorization of the synthesis distribution p(x,y,z) also differs from conditional ALI:
  p(x,y,z)=p(y)p(z |y)p(x |y,z). (11) The product p(y)p(z | y) can be conveniently given by a mixture model. Samples from p(y) are drawn from a multinomial prior, and samples from p(z | y) are drawn from a continuous prior, for example,N (µy=k,1). Samples from p(z |y) can alternatively be generated by a neural network by again employing the reparameterization trick. Namely,
  z(y)=µ(y)+σ(y) , ∼N (0,I). (12) This approach effectively learns the parameters ofN (µy=k,σy=k).
sections.heading: 2.2.1 ADVERSARIAL VALUE FUNCTION
sections.text: We follow Dumoulin et al. (2016) and define the value function that describes the unsupervised game between the discriminatorD and the generatorG as:
  min G max D V (D,G)=Eq(x)[log(D(x,Gy(x),Gz(x,Gy(x))))]
  +Ep(y,z)[log(1−D(Gx(y,Gz(y)),y,Gz(y)))]
  = ∫∫∫ q(x)q(y |x)q(z |x,y)log(D(x,y,z))dxdydz
  + ∫∫∫ p(y)p(z |y)p(x |y,z)log(1−D(x,y,z))dxdydz.
  (13)
  There are four generators in total: two for the encoderGy(x) andGz(x,Gy(x)), which map the data samples to the latent space; and two for the decoderGz(y) andGx(y,Gz(y)), which map samples from the prior to the input space. Gz(y) can either be a learned function, or be specified by a known prior (see Algorithm 1 for a detailed description of the optimization procedure).
  Algorithm 1 AMM training procedure using distributions (6) and (11). θGy(x),θGz(x,Gy(x)),θGz(y),θGx(y,Gz(y)),θD . Initialize AMM parameters while not done do x(1),...,x(M)∼q(x) . Sample from data and priors y(1),...,y(M)∼p(y) z(j)∼p(z |y=y(j)), j=1,...,M x̃(j)∼p(x |y=y(j),z=z(j)), j=1,...,M . Sample from conditionals ỹ(i)∼q(y |x=x(i)), i=1,...,M z̃(i)∼q(z |x=x(i),y= ỹ(i)), i=1,...,M ρ (i) q ←D(x(i),ỹ(i),z̃(i)), i=1,...,M .Compute discriminator predictions ρ (j) p ←D(x̃(j),y(j),z(j)), j=1,...,M LD←− 1M ∑M i=1log(ρ (i) q )− 1M ∑M j=1 log(1−ρ (j) p ) .Compute discriminator losses
  LGx(y,Gz(y))=LGz(y)←− 1M ∑M i=1log(ρ (i) p ) .Compute x generator losses
  LGy(x)=LGz(x,Gy(x))←− 1M ∑M i=1log(1−ρ (i) q ) .Compute y and z generator loss θD←θD−∇θDLD .Update discriminator parameters θGx(y,Gz(y))←θGx(y,Gz(y))−∇θGx(y,Gz(y))LGx(y,Gz(y)) .Update generator parameters θGz(y)←θGz(y)−∇θGz(y)LGz(y) θGy(x)←θGy(x)−∇θGy(x)LGy(x) θGz(x,Gy(x))←θGz(x,Gy(x))−∇θGz(x,Gy(x))LGz(x,Gy(x))
sections.heading: 2.3 SEMI-SUPERVISED ADVERSARIALLY LEARNED MIXTURE MODEL
sections.text: The Semi-Supervised Adversarially Learned Mixture Model (SAMM) is an adversarial generative model for supervised or semi-supervised clustering and classification of data. The objective for training SAMM involves two adversarial games to match pairs of joint distributions. The supervised game matches inference distribution (4) to synthesis distribution (11) and is described by the following value function: min G max D V (D,G)=Eq(x,y)[log(D(x,y,Gz(x,y)))]+Ep(y,z)[log(1−D(Gx(y,Gz(y)),y,Gz(y)))]
  = ∫∫∫ q(x,y)q(z |x,y)log(D(x,y,z))dxdydz
  + ∫∫∫ p(y)p(z |y)p(x |y,z)log(1−D(x,y,z))dxdydz.
  (14)
  The unsupervised game matches either of the inference distributions, (6) or (7) to the synthesis distribution (11). In the case using distribution (6), the unsupervised game is described by (13). The generator for semi-supervised learning has three components: encoders Gz(x,Gy(x)) and Gz(x,y) map the labeled and unlabeled data samples, respectively, to the latent space, and a decoder Gx(y,Gz(y)) maps samples of y and z to the input space, where Gz(z) can either be a learned function or be specified by a prior. The encoder for labeled data again consists of two generators (Figure 1). A detailed description of the training algorithm is given in algorithm 2 of the appendix. In practice, optimization of each of the generators and the discriminator can be done simultaneously for both the unsupervised and semi-supervised updates.
sections.heading: 3 RELATED WORKS
sections.text: Unsupervised clustering using hybrid adversarial approaches are proposed by both Makhzani et al. (2016) (AAE) and Chen et al. (2016) (InfoGAN). For AAE, the synthesis generator is optimized
  by minimizing the per-example L2 loss between between input data {xi} and their reconstructions {ẋi=Gxi(Gy(xi),Gz(xi))}, while the inference generator is optimized using both the L2 objective and an adversarial objective. For InfoGAN, the inference generator is optimized by maximizing the per-example Mutual Information (MI) between samples of categorical latent variables {yi∼p(y)} and continuous latent variables {zi∼p(z)} and their “reconstructions” {{ẏi,żi}=Gy,z(Gx(yi,zi)}, while the synthesis generator is optimized using both the MI objective and an adversarial objective.
  Recent approaches using self-supervision with data augmentation have also been proposed for unsupervised discrete representation learning by Kilinc & Uysal (2018) (LALNets) and Hu et al. (2017) (IMSAT). In IMSAT, a network is trained to maximize the Mutual Information between input data and a discrete representation, similar to InfoGAN. In addition to this objective, the network is regularized by encouraging the representation of original, unperturbed data {xi} to be close to that of transformed data {T (xi)}. In the LALNets framework, a network is trained to distinguish unperturbed data and augmented data, and then k-means clustering is performed on the latent space representation of the unperturbed data to obtain cluster labels.
  On the other end of the generative spectrum, Dilokthanakul et al. (2016) and Jiang et al. (2017) offer non-adversarial, VAE-based approaches for unsupervised clustering. Like in the AMM, the combination of priors for the latent variables y and z is modeled as a Gaussian mixture model, where y corresponds to the mixture components.
  Multiple adversarial methodologies have been proposed for supervised or semi-supervised learning (Springenberg, 2015; Salimans et al., 2016; Miyato et al., 2017; Dai et al., 2017), but they suffer from the same limitation as the original GAN: they do not provide inference. Gan et al. (2017), Li et al. (2017) and Deng et al. (2017) introduce a third player to the adversarial game. Although this extra player allows to infer categorical variables, these approaches are not fully adversarial as auxiliary “collaborative” terms are added to the objective function. Moreover, categorical and continuous latent variables are modeled independently.
  The adversarial and hybrid-adversarial approaches thus far discussed all model y and z as being conditionally independent from each other. This may be an ideal prior structure for inference, for example, in learning disentangled representations ofx sampled from a limited domain (Chen et al., 2016). However, the independence assumption cannot account for the notion of proximity between categories because z is identically distributed for each category in y. Therefore, the distance between categories is equal and indeterminate. AMM and SAMM are presented as adversarial approaches to model conditional dependencies between y and z, but they do not preclude the independence assumption. The proposed methods can model y and z as conditionally independent with inference distribution
  q(x,y,z)=q(x)q(y |x)q(z |x), (15) and synthesis distribution
  p(x,y,z)=p(y)p(z)p(x |y,z); (16)
  however, analysis of this graphical model is left for future work.
sections.heading: 4 EVALUATION
sections.text: AMM and SAMM are evaluated using two image datasets: MNIST (LeCun & Cortes, 2010) and SVHN (Netzer et al., 2011). The provided training and testing splits are used for MNIST experiments with 5000 randomly selected examples left out of the training set for validation. The same training, testing, and validation splits as Dumoulin et al. (2016) are used for SVHN. Preprocessing is limited to scaling image intensities on the range [0,1]. Detailed architectures for each experiment are shown in Section B of the appendix. We optimize all networks using Adam (Kingma & Ba, 2014) withα=0.0002 and β1=0.5. All kernel weights are initialized using a Gaussian distribution with standard deviation 0.02, all biases are initialized to 0.0. Following the criteria in Jiang et al. (2017), the performance is evaluated as follows:
  ACC=max n∈N
  ∑M i=11{yi=n(ỹz̃,i)}
  M , (17)
  whereM is the number of samples, yi is the true label, and ỹz̃,i is the predicted cluster label, andN is the set of all mappings between cluster labels and true labels.
sections.heading: 4.1 GRADIENT PENALTY
sections.text: The gradient penalty introduced by Gulrajani et al. (2017) is added to the discriminator loss to help stabilize training of AMM and SAMM models. This penalty keeps the gradients of the discriminator with respect to the inputs x, y, and z on the same order of magnitude. The penalty applied to the discriminator loss is
  L∇x̂,ŷ,ẑ =λ E (x̂,ŷ,ẑ)∼Px̂,ŷ,ẑ
  [ (||∇x̂,ŷ,ẑD(x̂,ŷ,ẑ)||2−1)2 ] , (18)
  where points (x̂,ŷ,ẑ) are drawn at random on straight lines between real or prior samples (x,ỹ,z̃) and synthesized or inferred samples (x̃,y,z). The gradient penalty for Jensen-Shannon GAN introduced by Roth et al. (2017) has also been explored, but did not produce better results. The regularization term is set to λ=10.0, and λ=0.01 for MNIST and SVHN experiments, respectively.
sections.heading: 4.2 MNIST
sections.text: In this section, the AMM is evaluated on the task of unsupervised clustering of hand-drawn digits using the MNIST dataset. To model p(y)p(z |y), a 64 dimensional mixture of Gaussians is used with 10, 20, and 30-components across 3 experiments. A multinomial prior is used for p(y) with uniform probability for each class. The mean and variance of each component distribution are learned using the reparameterization trick via (12). Table 1 reports the test-set clustering error-rate mean and standard deviation over 10 trials, with the 10-component AMM achieving a 3.32± 0.39 percent error rate. Figure 2 visualizes results from 1 of the 10 trials.
sections.heading: 4.3 SVHN
sections.heading: 4.3.1 UNSUPERVISED CLUSTERING
sections.text: In this section, unsupervised clustering is revisited. The SVHN dataset is used to investigate how confounding attributes, such as color and contrast, affects the semantic separation of digits. To model p(y)p(z |y) a 64 dimensional mixture of Gaussians is used with 10, 20, and 30-components across 3 experiments. A multinomial prior is used for p(y)with uniform probability for each class, and the mean and variance of each component distribution are learned using the reparameterization trick via (12).
  Figure 3a shows random samples drawn from each component distribution generated byGx. Figure 3b is a t-SNE projection of test samples drawn from Gz onto a 2D manifold. We show in 3c that AMM learns a smooth latent manifold as we interpolate between examples from SVHN. The cluster matrix for the SVHN test set is shown in figure 3d, demonstrating an overall classification error rate of 20.4%±0.80. From these results, we can make a few observations. First, the generated images are realistic looking. This is the case for every cluster other than cluster 5, which appears to be a noise cluster. Secondly, each cluster only contains a single digit. Finally, we observe that for each digit there are multiple clusters, one containing a dark background with light numbers (ie. cluster 28), and another containing a light background with dark numbers (ie. cluster 12). This is important, as it shows that the AMM has learned semantically meaningful clusters.
sections.heading: 4.3.2 SEMI-SUPERVISED CLUSTERING AND CLASSIFICATION
sections.text: It is evident from the last experiment that the confounders introduced by the SVHN dataset made unsupervised semantic clustering more difficult. In this section we show how SAMM can be used to guide clustering along predefined categories using only a small amount of labeled data. To this end, we limit the samples drawn from q(x,y) to a random selection of 1000 examples from the training set. To model p(y)p(z | y) we use a 64 dimensional mixture of 10 spherical Gaussians, where the the mean and variance of each component distribution are learned using the reparameterization trick via (12). There is considerable class imbalance in the SVHN dataset, so a multinomial prior is used for p(y) with each class probability set to the frequency observed in the labeled subset of the training data.
  Table 2 reports the test-set error-rate mean and variance over 10 trials. SAMM achieves 5.60±0.45 percent error rate, which is an improvement over the ALI baseline. Figure 4 shows visualizations of results from 1 of the 10 trials. Finally, given that we have defined p(y)p(z |y) we can use Bayes’ theorem to derive p(y |z) and get a classifier given an image embedding z̃:
  ỹz̃=argmax k
  [p(z= z̃ |y=k)p(y=k)] (19)
  Figures 4e and 4f compare the confusion matrices for predictions given by ỹz̃ and those given by ỹ fromGy. The similarity between each is further evidence that the inference network has learned to embed data according to the desired distribution.
sections.heading: 5 CONCLUSION
sections.text: The AMM is presented as a generative model for unsupervised or semi-supervised data clustering with several contributions. The AMM is the first fully adversarially optimized method to model the conditional dependence between categorical and continuous latent variables, providing impressive unsupervised clustering and competitive semi-supervised classification results on benchmark datasets. In contrast with other semi-supervised approaches, we have shown that the use of additional losses or discriminators (Gan et al. (2017); Li et al. (2017); Deng et al. (2017)) are redundant additions to frameworks, as the AMM yields similar or better results to these methods. In contrast with Dai et al. (2017), our strong semi-supervised performance and qualitatively good image generation demonstrate that semi-supervised performance and image generation are not necessarily opposing goals. As a fully adversarial framework, the AMM provides a simple, yet powerful formulation as a foundation for
  future work. For example, the distribution of mixture components can be something be other than a Gaussian without the need to derive an new evidence lower bound. Furthermore, by learning conditional dependencies betweeny andz, the AMM preserves the notion of proximity between classes in the latent space, which could be useful for applications in metric space learning and few shot learning. On top of this, learning the mean and variance of mixture components gives access to subsequent analysis of the likelihood function, which could provide an interpretable insight of the learned distributions. With that, the mixture coefficients are the only terms to be integrated into the framework. As a whole, the AMMs demonstrably strong performance validates its use in future work and extension into other domains.
sections.heading: APPENDIX A SAMM ALGORITHM
sections.text: Algorithm 2 outlines the SAMM training procedure.
  Algorithm 2 SAMM training procedure using distributions (4), (6), and (11). θGy(x),θGz(x,Gy(x)),θGz(y),θGx(y,Gz(y)),θD . Initialize SAMM parameters while not done do x (1) u ,...,x (M) u ∼q(x) . Sample from unlabeled data and priors
  y (1) u ,...,y (M) u ∼p(y) z (j) u ∼p(z |y=y(j)u ), j=1,...,M x̃ (j) u ∼p(x |y=y(j)u ,z=z(j)u ), j=1,...,M . Sample from conditionals ỹ (i) u ∼q(y |x=x(i)u ), i=1,...,M z̃ (i) u ∼q(z |x=x(i)u ,y= ỹ(i)u ), i=1,...,M( x (1) ` ,...,x (M) ` ) , ( ỹ (1) ` ,...,ỹ (M) ` ) ∼q(x,y) . Sample from labeled data and priors
  y (1) ` ,...,y (M) ` ∼p(y) z (j) ` ∼p(z |y=y (j) ` ), j=1,...,M x̃ (j) ` ∼p(x |y=y (j) ` ,z=z (j) ` ), j=1,...,M . Sample from conditionals z̃ (i) ` ∼q(z |x=x (i) ` ,y= ỹ (i) ` ), i=1,...,M ρ (i) qu←D(x(i)u ,ỹ(i)u ,z̃(i)u ), i=1,...,M .Compute predictions for unlabeled data ρ (j) pu ←D(x̃(j)u ,y(j)u ,z(j)u ), j=1,...,M ρ (i) q` ←D(x(i)` ,ỹ (i) ` ,z̃ (i) ` ), i=1,...,M .Compute predictions for labeled data ρ (j) p` ←D(x̃(j)` ,y (j) ` ,z (j) ` ), j=1,...,M
  LDu←− 12M ∑M i=1log(ρ (i) qu )− 12M ∑M j=1 log(1−ρ (j) pu ) .Compute discriminator losses
  LD`←− 12M ∑M i=1log(ρ (i) q` )− 12M ∑M j=1 log(1−ρ (j) p` )
  LGyu (x)=LGzu (x,Gy(x))←− 12M ∑M i=1log(1−ρ (i) qu ) .Compute inference losses LGz` (x,Gy(x))←− 1 2M ∑M i=1log(1−ρ (i) q` )
  LGxu (y,Gz(y))=LGzu (y)←− 12M ∑M i=1log(ρ (i) pu ) .Compute x generator losses LGx` (y,Gz(y))=LGz` (y)←− 1 2M ∑M i=1log(ρ (i) p` ) θD←θD−∇θD (LDu+LD`) .Update discriminator parameters θGz(x,Gy(x))←θGz(x,Gy(x))−∇θGz(x,Gy(x)) ( LGzu (x,Gy(x))+LGz` (x,Gy(x)) ) . Update z inference parameters θGy(x)←θGy(x)−∇θGy(x)LGyu (x) .Update y inference parameters θGx(y,Gz(y))←θGx(y,Gz(y))−∇θGx(y,Gz(y)) ( LGxu (y,Gz(y))+LGx` (y,Gz(y)) ) . Update x synthesis parameters θGz(y)←θGz(y)−∇θGz(y) ( LGzu (y)+LGz` (y) ) .Update z synthesis parameters
sections.heading: APPENDIX B EXPERIMENT INFORMATION
sections.text: B.1 MODEL ARCHITECTURES
  Tables 3, 4, 5 and 6 detail the model architectures for the MNIST experiments. Tables 7, 8, 9 and 10 detail the model architectures for the SVHN experiments. Model inputs and outputs are highlighted in boldface. “Leak” denotes Leaky ReLU activations. “ExpN” denotes the following activation function:
  ExpN(x)=exp(0.5 x) , ∼N (0,I). (20)