Max-Margin Deep Generative Models
Deep generative models (DGMs) are effective on learning multilayered representations of complex data and performing inference of input data by exploring the generative ability. However, little work has been done on examining or empowering the discriminative ability of DGMs on making accurate predictions. This paper presents max-margin deep generative models (mmDGMs), which explore the strongly discriminative principle of max-margin learning to improve the discriminative power of DGMs, while retaining the generative capability. We develop an efficient doubly stochastic subgradient algorithm for the piecewise linear objective. Empirical results on MNIST and SVHN datasets demonstrate that (1) maxmargin learning can significantly improve the prediction performance of DGMs and meanwhile retain the generative ability; and (2) mmDGMs are competitive to the state-of-the-art fully discriminative networks by employing deep convolutional neural networks (CNNs) as both recognition and generative models.
This paper introduces a max-margin criterion to the learning of a deep generative model, essentially enhancing the discriminative abilities of the latent variables z.Unsurprisingly, this improves the classification abilities of the model, compared to the use of a simple linear classifier trained on the latent variables of the original generative model.On the MNIST and SVHN sets, the final model is reasonably competitive with purely discriminative models, though not state-of-the-art.However, it retains the advantage of being a generative model.The paper is well written.The technical sections are particularly clear, although I found the motivation section was weak and unclear in places.Although the idea of incorporating the max-margin criterion is not highly innovative, this paper adds a large amount of useful work regarding how the optimisation should be achieved.I am not an expert on the literature in this area, so am unable to make a strong judgement on the level of originality.I found that the work is not clearly motivated in the early sections.I was initially rather sceptical about whether there is any gain to using a discriminative criterion with a generative model - if you want generative behaviour, why not use a generative model; if you want discriminative behaviour, why not use a discriminative model?However, I am prepared to accept that there are advantages.In particular, section 4.4 is very interesting and in a sense, motivates the rest of the paper for me.However, there is a missed opportunity to compare the classification performance against other systems in the missing data scenarios - I would find good results in this area to be a compelling reason for using the proposed model.Simply comparing error rates against a series of discriminative models in Tables 1 and 3 aren't so interesting.The mathematics seems correct to me, but I have not checked every equation in detail.Guassian -> GaussianThere are some minor inconsistencies in the references section, eg. [4] vs. [33], and [13].