Iterative Neural Autoregressive Distribution Estimator (NADE-k)
Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in k steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.
The paper describes a new variant of (order agnostic, deep) NADE called NADE-k that performs k successive reconstructions/inference steps when imputing the unobserved components of the input vector. To train NADE-k, one must sample a subset of the input variables to impute while observing the remaining input variables. The paper includes experiments showing improved log probs over other versions of NADE on binarized MNIST digit generation and Caltech-101 silhouette generation. There are also experiments exploring different values of k and the variance of the imputations with respect to the choice of variable order.The NADE-k algorithm presented in the paper seems sensible and is for the most part explained clearly. The experiments included are reasonable ones. The NADE-k algorithm is an incremental improvement of deep, order-agnostic NADE and a relatively straightforward one. However, it is more expensive at both training time and test time (since test k is best set to equal training k). Any algorithm based on unrolling inference steps and backpropagating through them can have a k step version of itself created. The question is really how interesting a multi-step variant is and whether it is worth the computational expense. If CD-1 had been introduced without reference to the infinite steps of maximum likelihood and separately from CD-k, would a paper on extending CD-1 to CD-k have been very novel?One possible experiment not included in the paper that would have been interesting to see is whether slowly increasing k during training works well. In RBM training, to get very good generative performance quickly, it is helpful to switch from CD-1 to CD-k by stepping up k during training. At the start of training, the model is so inaccurate that CD-1 can make rapid progress, but as training continues, more CD steps can help. Would this idea benefit NADE-k?Somewhat more practical applications of high quality density models would have been welcome to motivate the paper more strongly, perhaps in compression or denoising. Learning a density over binarized MNIST pixels is not a very compelling task. Since the NADE family of algorithms provide tractable, normalized, high-quality density models, an application where those features are essential would have been more interesting.The paper was mostly clear. There are a few sentence-level issues, such as line 199 "allow to compute" and footnote 2 "for helping better converge."I had to slow down a lot while reading when I came upon equation 10, it would be clearer if there was a bit more english text introducing it. In general, the paper is well organized.