Revealing latent structure in data is an active field of research, having introduced exciting technologies such as variational autoencoders and adversarial networks, and is essential to push machine learning towards unsupervised knowledge discovery. However, a major challenge is the lack of suitable benchmarks for an objective and quantitative evaluation of learned representations. To address this issue we introduce Morpho-MNIST, a framework that aims to answer: "to what extent has my model learned to represent specific factors of variation in the data?" We extend the popular MNIST dataset by adding a morphometric analysis enabling quantitative comparison of trained models, identification of the roles of latent variables, and characterisation of sample diversity. We further propose a set of quantifiable perturbations to assess the performance of unsupervised and supervised methods on challenging tasks such as outlier detection and domain adaptation.
This paper discusses the problem of evaluating and diagnosing the representations learnt using a generative model. This is a very important and necessary problem.
However, this paper lacks in terms of experimental evaluation and has some technical flaws.
1. Morphological properties deals with only the "shape" properties of the image object. However, when the entire image is subject to the generative model, it learns multiple properties from the image apart from shape too - such as texture and color. Additionally, there are lot of low level pixel relations that the model learns to fit the distribution of the given images. However, here the authors have assumed that the latent space of the generative models are influenced only by the morphological properties of the image - which is wrong. Latent space features could be affected by the color or texture of the image as well.
2. Extracting morphological properties of the image is straight-foward for MNIST kind of objects. However, it becomes really difficult for other datasets such as CIFAR or some real world images. Studying the properties of a generative model on such datasets is very challenging and the authors have not added a discussion around that. 
3. Now assuming that my GAN model has learnt good representation in Morpho-MNIST dataset, is it guaranteed to learn good representations in other datasets as well? There is no guarantee on generalizability or extensibility of the work.