Deep Convolutional Inverse Graphics Network
This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that aims to learn an interpretable representation of images, disentangled with respect to three-dimensional scene structure and viewing transformations such as depth rotations and lighting variations. The DC-IGN model is composed of multiple layers of convolution and de-convolution operators and is trained using the Stochastic Gradient Variational Bayes (SGVB) algorithm [10]. We propose a training procedure to encourage neurons in the graphics code layer to represent a specific transformation (e.g. pose or light). Given a single input image, our model can generate new images of the same object with variations in pose and lighting. We present qualitative and quantitative tests of the model's efficacy at learning a 3D rendering engine for varied object classes including faces and chairs.
Summary: the paper proposes a CNN for learning explicit image representations as an inverse graphics problem.The image representation has interpretable explicit representations, in particular pose angles and lighting angles, along with implicit representations (texture, appearance).This is done in an autoencoder framework with reconstruction error.To make a particular latent dimension focus on one aspect (e.g. azimuth), in each mini-batch, training data is presented with only variations in one aspect, while activations in other nodes are surpressed to a fixed value (the mean value).Experiments on two datasets showing reconstructions of a 3D object at varying poses and illumination directions.Quality: The methodology is fine, but some interesting cases are not tested in the experiments. - the pose variables are trained independently.How well can they be used in conjunction?For example, changing elevation and azimuth at the same time? - L344-348 - I would have liked to see an actual quantitative evaluation about how well the actual pose could be estimated.The GT pose and pose variables are in different spaces, correlation between the GT and inferred could be calculated.Or a simple transformation function from one space to the other could be learned. - Sec 4.1.1 - For the comparison with a standard network, the latent variable with the largest variance is used as the surrogate for azimuth (L359).It would be interest to also use the linear subspace with the largest variance (i.e., PCA on the latent vector space), since it is possible the azimuth is encoded in multiple latent variables. - no comparisons to other inverse-graphics models [10,17,16,12].Originality: appears to be original.The training method for specializing a node for a particular transformation appears novel.Clarity: - in (1) and (2), why is the mean and variance coupled? i.e., \Sigma = diag(exp(\mu)) - Sec 4.1, it's unclear whether the "static test image" are the same faces as those in the training set. - L376: The error 2.7722e-4 is misleading.The images in Fig 7 have a lot of white space, which are easy to predict and artifically decreases the error rate.The error rate should be for only the pixels in the bounding box of the object.Significance: The paper is interesting, but I'm left to wonder how significant is the result.What is the end-goal of such a network?If it is for viewing objects at different viewpoints, it seems limited since the quality of the images is poor and low resolution (looking at Fig 7).If it is for learning the mapping from image to graphics code (pose parameters), then this is not evaluated in the experiments so it's unclear its advantage over directly predicting the pose parameters with a CNN.