DISTRIBUTED TRANSFER LEARNING FOR DEEP CONVOLUTIONAL NEURAL NETWORKS BY BASIC PROBABILITY ASSIGNMENT
Transfer learning is a popular practice in deep neural networks, but fine-tuning of a large number of parameters is a hard challenge due to the complex wiring of neurons between splitting layers and imbalance class distributions of original and transferred domains. Recent advances in evidence theory show that in an imbalance multiclass learning problem, optimizing of proper objective functions based on contingency tables prevents biases towards high-prior classes. Transfer learning usually deals with highly non-convex objectives and local minima in deep neural architectures. We propose a novel distributed transfer learning to tackle both optimization complexity and class-imbalance problem jointly. Our solution imposes separated greedy regularization to each individual convolutional filter to make single-filter neural networks such that the minority classes perform as the majority ones. Then, basic probability assignment from evidence theory boosts these distributed networks to improve the recognition performance on the target domains. Our experiments on several standard datasets confirm the consistent improvement as a result of our distributed transfer learning strategy.
Transfer learning is a popular practice in deep neural networks, but fine-tuning of a large number of parameters is a hard challenge due to the complex wiring of neurons between splitting layers and imbalance class distributions of original and transferred domains. Recent advances in evidence theory show that in an imbalance multiclass learning problem, optimizing of proper objective functions based on contingency tables prevents biases towards high-prior classes. Transfer learning usually deals with highly non-convex objectives and local minima in deep neural architectures. We propose a novel distributed transfer learning to tackle both optimization complexity and class-imbalance problem jointly. Our solution imposes separated greedy regularization to each individual convolutional filter to make single-filter neural networks such that the minority classes perform as the majority ones. Then, basic probability assignment from evidence theory boosts these distributed networks to improve the recognition performance on the target domains. Our experiments on several standard datasets confirm the consistent improvement as a result of our distributed transfer learning strategy.
1 INTRODUCTION
In supervised learning, many classification algorithms assume the same distribution for training and testing data. Consequently, change of distribution requires rebuilding of the statistical models which is not always practical because of the hardship of recollecting of training data or heavy learning process. One of the solutions is transfer learning that transfers the classification knowledge into a new domain Pan & Yang (2010). This aims at learning of highly-generalized models with different probability distributions across domains to learn novel domains without labeled data Wang & Schneider (2014) Zhang et al. (2013). Here, the main challenge is to reduce the shifts in data distribution between domains by algorithms that minimize the discriminant of the domains. It is worth mentioning that this could not get rid of domain-specific variations Long et al. (2016).
Transfer learning for deep neural networks has been proved highly beneficial to boost their overall performance. Deep learning practices usually require huge amount of labeled data to learn powerful models. The transfer learning enables adaptation to a different source with small training samples. On the other hand, deep neural networks practically learn intermediate features. They could provide better transfer among domains because some of them generalize well among various domains of knowledge Glorot et al. (2011). These transferable features generally underlies several probability distributions Oquab et al. (2014) which reduce the cross-domain discrepancy Yosinski et al. (2014).
The common observation among several deep architectures is that features learned in bottom layers are not that specific, but transiting towards top layers makes them tailored to a dataset or task. A recent study Yosinski et al. (2014) of the generality or specificity of deep layers for the sake of transfer learning reveals two difficulties which may affect the transfer of deep features. First, top layers get quite specialized to their original tasks and second, some optimization difficulties rise due to the splitting of the network between co-adapted layers. In spite of these negative effects, it
Algorithm 1 Basic Probability Assignment (BPA) Input: train/validation set X Output: basic probability assignment BPA(φ)
1: compute R(φ) and P(φ) (Eqs.1- 2) 2: calculate recall and precision assignments(Eq.3) 3: apply Dempster rule for accumulation (Eq.4)
is shown that transferred features not only perform better than random ones but also provide better initialization. This gives a boost to the generalization of deep neural networks as well.
In this paper, we propose a framework for distributed transfer learning in deep convolutional networks. This tries to alleviate the burden of splitting networks in the middle of fragile co-adapted layers. The intuition is that above difficulty relates to the complexity of deep architectures and also, class-imbalance in the transferred domain.
On the matter of network complexity, we argue that the splitting of layers leads to a hard optimization problem because of high complexity in the interconnections between neurons of co-adapted layers. It seems that transfer learning is not able to thoroughly reconstruct the original powerful wiring for the transferred domain. This is due to the size of network and large number of interconnections across neurons. To address this issue, we fine-tune the convolutional filters separately and hence, reduce the complexity of the non-convex optimization.
On the other hand, it seems that the class-imbalance problem rises form different distribution of data in original and transferred domains. This issue can be handled by cost-sensitive imbalanced classifications methods. By class-imbalance in transferred domain, we mean variable coverage of common classes in this domain and the ones from the original domain. It is probable that both original and transferred datasets have uniform distributions of data among their classes, but some classes in one domain may be fully or partly covered by the other domain. This results in imbalance class distribution in the transfer learning.
The determination of a probabilistic distribution from the confusion matrix is highly effective to produce a probability assignment which contributes to class-imbalance problems. This basic probability assignment can be either constructed from recognition, substitution and rejection rates Xu et al. (1992) or both precision and recall rates of each class Deng et al. (2016). The key point is harvesting of maximum possible prior knowledge provided by the confusion matrix to overcome the imbalance classification challenge.
Since the power of deep convolutional models come from mutual optimization of all parameters, we join the above distributed fine-tuned filters by a boosting scheme based on basic probability assignment. Our experiments confirm the functionality of our distributed strategy for deep transfer learning. The rest of paper is organized as follows. We present the formulation of our method in Section 2, report our experiments in Section 3 and conclude in Section 4.
2 FORMULATION
In general, a confusion matrix represents the class-based predictions against actual labels in form of a square matrix. Inspired by Dempster−Shafer theory, construction of basic probability assignment (BPA) Sentz & Ferson (2002) gives a vector which is independent of the number of class samples and sums up to one for each individual label. This basic probability assignment provides the ability to reflect the difference contributions of a classifier to each individual classes or combine the outcomes of multiple week classifiers.
2.1 BASIC PROBABILITY ASSIGNMENT
A raw two-dimensional confusion matrix indexed by predicted classes and actual labels provides some common measures of classification performance. They are accuracy (the proportion of the total number of predictions that were correct), precision (a measure of the accuracy provided that
a specific class has been predicted), recall (a measure of the ability of a prediction model to select instances of a certain class from a dataset) and F-score (the harmonic mean of precision and recall) Sammut & Webb (2011).
Suppose a set of train/validation samples X = {X1, . . . , X|X |} from C = {C1, . . . , C|C|} different classes are assigned to a label set L = {L1, . . . , L|L|} by a classifier (φ) such that |C| = |L|. If each element (nij) of the confusion matrix C(φ) is considered as the number of samples belonging to class Ci which assigned to label Lj , then we can define recall (rij) and precision (pij) ratios as follows Deng et al. (2016)
rij = nij∑|C| j=1 nij pij = nij∑|L| i=1 nij
(1)
It can be seen that the recall ratio is summed over the actual labels (rows) whilst the precision ratio is accumulated by the predicted classes (columns) of the confusion matrix C(φ). Now, we are able to define recall and precision matrices as
R(φ) = {rij} P(φ) = {pij}
for i ∈ [1 . . . |L|], j ∈ [1 . . . |C|] (2)
The basic probability assignments of these matrices contain recall and precision probability elements for each individual class Ci such that
mri = rii∑|C| j=1 rji mpi = pii∑|L| j=1 pij
(3)
These elements are synthesized to form the final probability assignments representing the recognition ability of classifier φ to each of the classes of set C
mi = mri ⊕mpi = mri ×mpi 1− ∑|C| i=1mri ×mpi (4)
Here, operator ⊕ is an orthogonal sum which is applied by Dempster rule of combination Sentz & Ferson (2002). The overall contribution of the classifier φ cab be presented as a probability assignment vector
BPA(φ) = {mi} for i ∈ [1 . . . |C|] (5)
It is worth mentioning that BPA(φ) should be computed by the train/validation set because we assume that the test set does not include actual labels. Besides, combination of different classes under vertical or horizontal categories is a common practice in visual classification. The benefit lies in the fact that bottom layers of deep convolutional architectures make better contribution to detect first and second order features that are usually of specific directions (vertical vs horizontal) rather than detailed distinguished patterns of the objects. This leads to a powerful hierarchical feature learning in the case that |C| |L|. In contrast, some classes can be divided to various sub-categories although they all get the same initial labels and hence this holds |C| |L| to take the advantage of top layers. In the above formulation, we do not merge or divide the original setup of the datasets under study (|C| = |L|) although it seems that our BPA-based approach is also able to boost the trained classifiers for each of the merge/divide scenarios.
2.2 DISTRIBUTED TRANSFER LEARNING
A general practice in transfer learning includes training of an original deep neural network on a dataset and then, fine-tuning of learned features for another dataset on a new target network. Bengio et al. (2012). The generality of selected features for both original and target domains is critical to the success of the transfer learning. For implementation, we train the original network and copy its bottom layers to form the target network. The top layers of the target network are initialized randomly and trained on the target dataset. We are able to employ backpropagation from top to bottom layers and fine-tune their parameters for the target task or freeze the copied originals and only update top target layers. This can be decided by size of the target dataset and number of parameters in the original layers. Fine-tuning of large networks for small dataset leads to overfitting but for small network or large dataset, performance will be improved Sermanet et al. (2013).
Based on our formulation for basic probability assignment (BPA) on Section 2.1, we are able to follow the above transfer learning procedure by learning of a classifier φ (SVM or Softmax) and computing BPA(φ) using Algorithm 1. Here, the learning means fine-tuning of target domain using the rained weights and biases of the original network. To implement this, we train the original fullyconnected layers by the features calculated by presenting target’s train set to convolutional layers of the same original network. We deploy this procedure for each of the available convolutional filters separately and compute the BPA of each individual single-filter network for train/validation sets. Then, we combine unary potentials of all the fine-tuned classifiers by employing BPA weights to come up with a unit set of class probabilities. Figure 1 provides an overview of conventional and distributed transfer learning processes.
Suppose that Ci is the predicted class for a test sample T provided by classifier φ. To revise the classification outcome by the BPA calculation, we multiply the test sample’s unary potentials U(T ) = {u1, . . . , u|C|} (probabilities of belonging to each class) by an assignment vector M(φ) = {1 − m1, . . . , 1 − m|C|} (contributions of the classifier φ to each class) and pick the maximum index as the revised predicted label
C(T ) = I ( arg max {u1 × (1−m1), . . . , u|C| × (1−m|C|)} ) (6)
This implies that if classifier φ performs well on class Ci (high mi), it is highly probable that C(T ) leans towards Ci. At the same time, other minority classes like Cj (low mj) have a chance to win if their unary potentials would be high enough (uj > ui). In contrast, if φ does poor classification on class Ci (low mi), the possibility of updating C(T ) to another class (Cj) with even worse unary potential (uj < ui) would be higher. Therefore, BPA shows quite successful in handling imbalance data distribution among classes.
Algorithm 2 Distributed Transfer Learning Input: train/validation set X , test sample T , set of week classifiers F Output: predicted class CF (T )
for i = 1 to |C| do for j = 1 to |F| do
1: compute mij ∈ BPA(F) (Alg.1) 2: calculate unary potential uij ∈ UF (T)
end for end for
3: predict boosted output CF (T ) (Eq.8) 4: employ error backpropagation for fine-tuning
As described in Section 1, employing probability assignment addresses the class-imbalance problem but does not reduce the complexity of optimization because of the fact that both forward learning and error backpropagation are applied to all the model parameters. To break this non-convex optimization, we introduce our distributed transfer learning strategy. For implementation, we replace the mutual learning of all the parameters with learning of each individual convolutional filter in a separate classifier fed by the bottom original layer. It means that we train a set of week single-filter classifiers F = {φ1, . . . , φ|F|} which |F| equals the number of convolutional filters in the deep neural architecture.we follow the recipe of single classifier in Equation 5 but extend it to redefine
BPA(F) = {mij} for i ∈ [1 . . . |C|], j ∈ [1 . . . |F|] (7)
such that mij is the probability assignment of class Ci to week single-filter classifier φj . To come up with class of the test sample T , we update the Equation 6 as follows
CF (T ) = I ( arg max { u1j × (1−m1j)∑F
j=1 u1j × (1−m1j) , . . . , uij × (1−m|C|j)∑F j=1 u|C|j × (1−m|C|j)
} )
(8)
Here, uij is the unary potential of class Ci determined by the week single-filter classifier φj . Building on the above formulations, we are able to distribute the transfer learning among convolutional filters and join them later to implement a better fine-tuning for the target deep convolutional network according to the Algorithm 2.
3 EXPERIMENTS
We conduct our experiments on MNIST, CIFAR and Street View House Numbers (SVHN) datasets. The MNIST dataset LeCun et al. (1998) contains 60, 000 training examples and 10, 000 test samples normalized to 20 × 20, centered by center of mass in 28 × 28 and sheared by horizontally shifting such that the principal axis is vertical. The foreground pixels were set to one and the background to zero. The CIFAR dataset Krizhevsky & Hinton (2009) includes two subsets. CIFAR-10 consists of 10 classes of objects with 6, 000 images per class. The classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. It was divided to 5, 000 randomly selected images per class as training set and the rest, as testing samples. The second subset is called CIFAR-100 having 600 images in each of 100 classes. These classes also come in 20 super-classes of five class each. The SVHN dataset Netzer et al. (2011) was extracted from a large number of Google Street View images by automated algorithms and the Amazon Mechanical Turk (AMT) framework. It consists of over 600, 000 labeled characters in full numbers and MNIST-like cropped digits in 32×32. Three subsets are available containing 73, 257 digits for training, 26, 032 for testing and 531, 131 extra samples.
We consider two different scenarios to evaluate the performance of our distributed transfer learning algorithm. In the first experiment, we try to observe the performance of fine-tuning for pairs
of datasets with close data distributions or number of classes. We select MNIST & SVHN and CIFAR-10 & CIFAR-100 as original-target domains and report the transfer learning results in form of train-test errors. In the second experiment, we apply transfer learning for pairs of datasets with far data/class setups which are MNIST & CIFAR-10 and SVHN & CIFAR-100. In this experiment, we arrange the datasets to examine the effect of dissimilar distributions rather than overfitting.
Before moving forward to discuss the experiments, we report the baseline train-test errors for the datasets in Table 1. These results are produced by the deep learning library provided by the Oxford Visual Geometry Group Vedaldi & Fulkerson (2008).
3.1 EXPERIMENT 1
Table 2 shows the performance of conventional and distributed transfer learnings for the first scenario. The first values before dash correspond to the training errors (left) and the second ones present the testing errors (right).
In this experiment, we target two pairs of datasets (original-target domains) which contain similar data and perform number/object recognition tasks. We report the results for both conventional and our distributed transfer learning methods. By conventional Bengio et al. (2012), we mean training the original dataset and fine-tuning of the target one. With distributed, we aim at training the original dataset but employing the basic probability assignment for the transfer learning.
It can be seen that the results for the conventional transfer learning follows our argument on size of network and number of model parameters Sermanet et al. (2013). Compared to Table 1, MNIST does a poor job on transferring of SVHN due to the overfitting of SVHN over MNIST network. In contrast, SVHN perform quite well on transferring MNIST.
On the other hand, transferring of SVHN from MNIST does not overfit when our distributed transfer learning is employed. In both settings of original-target domains, our distributed strategy outperforms the conventional transfer learning approach.
The experiment on CIFAR pair exposes more interesting results due to the fact that both datasets have the same number of samples but completely different distributions among the classes. In practice, CIFAR-100 includes all the classes of CIFAR-10 but CIFAR-10 does not have any clue of the several classes of CIFAR-100. The conventional experiments show that CIFAR-10 transfers well on CIFAR-100 but it cannot perform transferring although the target network does not overfit.
All in all, the performance of our distributed transfer learning (bold values) is better than the conventional scheme and also, outperforms the baseline deep learning practices.
3.2 EXPERIMENT 2
In Table 3, we reports the results for both conventional and distributed transfer learnings on the second scenario. Here, we pair datasets such that the similarity of their data distributions and number of classes get minimized and they are originally trained for different tasks. It is obvious that our distributed transfer learning outperforms all the conventional results.
For the first setup, CIFAR-10 does a better transfer learning than MNSIT although the number of classes are the same. It seems that CIFAR-10 provides better generalization due to higher diversity among its classes. Here, our distributed algorithm performs better than the conventional process and,
targeting of MNIST on CIFAR-10 network gives close performance to the deep learning outcomes. The second setup leads to the overfitting of SVHN over CIFAR-100 network due to huge number of samples. The other outcome is the poor performance of transferring CIFAR-100 over SVHN network as a result of huge conceptual gap between original-target domains.
Our observations show that fine-tuning on training set and calculating BPA on validation, result in better generalization of the transferred model on testing set. On the other hand, computing of BPA on training plus validation sets gives higher performance in case of hugely different number of classes in original-target datasets. Since we employ BPA to address the class-imbalance problem, we reckon that it better captures the distribution of data by adjoining both train/validation sets especially when we intend to transfer few classes of original dataset to the larger number of classes in the target.
4 CONCLUSION
We introduce a novel transfer learning for deep convolutional networks that tackles the optimization complexity of a highly non-convex objective by breaking it to several distributed fine-tuning operations. This also resolves the imbalance class coverage between original-target domains by using basic probability assignment across several week single-filter classifiers. By the above boosting, the overall performance shows considerable improvement over conventional transfer learning scheme. We conduct several experiments on publicly available datasets and report the performance as traintest errors. The results confirm the advantage of our distributed strategy for the transfer learning.

Update: I thank the author for his comments! At this point, the paper is still not suitable for publication, so I'm leaving the rating untouched.
This paper proposes a transfer learning method addressing optimization complexity and class imbalance.
My main concerns are the following:
1. The paper is quite hard to read due to typos, unusual phrasing and loose use of terminology like “distributed”, “transfer learning” (meaning “fine-tuning”), “softmax” (meaning “fully-connected”), “deep learning” (meaning “base neural network”),  etc. I’m still not sure I got all the details of the actual algorithm right.
2. The captions to the figures and tables are not very informative – one has to jump back and forth through the paper to understand what the numbers/images mean.
3. From what I understand, the authors use “conventional transfer learning” to refer to fine-tuning of the fully-connected layers only (I’m judging by Figure 1). In this case, it’s essential to compare the proposed method with regimes when some of the convolutional layers are also updated. This comparison is not present in the paper.
Comments on the pre-review questions:
1. Question 1: If the paper only considers the case |C|==|L|, it would be better to reduce the notation clutter.
2. Question 2: It is still not clear what the authors mean by distributed transfer learning. Figure 1 is supposed to highlight the difference from the conventional approach (fine-tuning of the fully-connected layers; by the way, I don’t think, Softmax is a conventional term for fully-connected layers). From the diagram, it follows that the base CNN has the same number of convolutional filters at every layer and, in order to obtain a distributed ensemble, we need to connect (for some reason) filters with the same indices. This does not make a lot of sense to me but I’m probably misinterpreting the figure. Could the authors revise the diagram to make it clearer?
Overall, I think the paper needs significant refinement in order improve the clarity of presentation and thus cannot be accepted as it is now.


A recognized obstacle to training undirected graphical models with latent variables such as Boltzmann machines is that the maximum likelihood training procedure requires sampling from Monte-Carlo Markov chains which may not mix well, in the inner loop of training, for each example. We first propose the idea that it is sufficient to locally carve the energy function everywhere so that its gradient points in the “right” direction (i.e., towards generating the data). Following on previous work on contrastive divergence, denoising autoencoders, generative stochastic networks and unsupervised learning using non-equilibrium dynamics, we propose a variational bound on the marginal log-likelihood of the data which corresponds to a new learning procedure that first walks away from data points by following the model transition operator and then trains that operator to walk backwards for each of these steps, back towards the training example. The tightness of the variational bound relies on gradually increasing temperature as we walk away from the data, at each step providing a gradient on the parameters to maximize the probability that the transition operator returns to its previous state. Interestingly, this algorithm admits a variant where there is no explicit energy function, i.e., the parameters are used to directly define the transition operator. This also eliminates the explicit need for symmetric weights which previous Boltzmann machine or Hopfield net models require, and which makes these models less biologically plausible.
1 INTRODUCTION
Although earlier research focused on generating data through Monte Carlo Markov chains (MCMCs), e.g. with various Boltzmann machines (Salakhutdinov & Hinton, 2009), most of the recent effort in designing deep generative models is based on single-step generation, e.g., with variational auto-encoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014). However, generating a sample by going through a series of stochastic transformations that gradually improve the generated sample (or its latent representation) to make it more plausible could hold some advantages. A generative process can be seen as a mapping from simple noise variates (e.g., uniform, Gaussian) to samples from a very complicated distribution (maybe concentrated near a low-dimensional manifold) approximating the one which we are trying to learn from. If the data distribution is complex (e.g., the corresponding manifold is highly convoluted and non-linear), the generative process may involve a highly non-linear transformation which could be difficult to learn and optimize. Such highly non-linear transformations are probably best represented (and learned) by composing a large number of slightly non-linear transformations, either with a fixed-depth deep network, or with a variable depth recurrent computation, which is what the repeated application of a transition operator corresponds to.
1.1 MOTIVATIONS
The main motivation for the paper are the following.
• The main difference between feedforward generation and recurrent generation is twofold:(1) in the recurrent case, the same parameters are used for each step of the transition
∗anirudhgoyal9119@gmail.com †rosemary.nan.ke@gmail.com ‡lambalex@iro.umontreal.ca §CIFAR Senior Fellow
operator, and (2) by providing an interpretation of each of these steps as the application of a transition operator, we can design training procedures which do not require backpropagating through all the steps of the unfolded computation (from the raw noise samples to the generated output). This is a potential that clearly deserves to be explored further and motivates the learning framework introduced here.
• Another motivation for the Variational Walkback is the idea that we only need to carve the energy function in the right direction at each point in the space of the random variables of interest, which may sideskip the need to actually sample from the stationary distribution of a Markov chain in order to obtain the gradients of the training objective. The intuition is that if the model’s transition operator wants to move away from the data and into an area without data, this is a clue that the energy gradient is pointing in the wrong direction at that place. Consider a chain of samples following the model’s transition operator (or variants of it at different temperatures), starting at a data point. If the chain moves us away from data points, then we can use the previous state in the chain as a target for the operator when that operator is applied to the next next state, i.e., we want to teach the operator to walk back towards the data. This intuition was already exploited by Bengio et al. (2013c) but without a firm mathematical grounding. In Variational Walkback this is rigorously justified by a variational bound.
• Yet another motivation for the particular approach presented here is that it innovates in the rarely explored direction of parametrizing directly the generative model via a transition operator, rather than via an explicit probability function or energy function. This idea has already been discussed in the context Generative Stochastic Networks (GSNs) (Bengio et al., 2013b), a generalization of denoising auto-encoders (DAEs) (Vincent et al., 2008) which interprets the auto-encoder as estimating the gradient of an energy function (Alain & Bengio, 2014) or as a transition operator (Bengio et al., 2013c). An advantage of being able to parametrize directly the generator is seen with GANs and DAEs: we directly parametrize and learn the function which will be used to perform the task of interest (e.g. generating answers to some questions). Instead, the traditional approach is to parametrize a probability function or energy function (e.g., with a Boltzmann machine) and then then use another procedure (the MCMC method of your choice) to sample from it and do inference. Another important reason for exploring algorithms for directly learning a transition operator is that they put less constraint on the form of the transition operator, compared with a transition operator derived from an energy function. More specifically, neural net implementations of transition operators derived from an MCMC typically require the presence of symmetric weights (due to the symmetry of the second derivative of the energy with respect to a pair of units in the neural network), as discussed by Bengio et al. (2015). When we consider a biologically plausible implementation of these learning algorithms, the weight symmetry constraint (Wij = Wji) is not reasonable as a hard constraint. Instead, if the transition operator (rather than the energy function) is the object being parametrized and learned, then there is no such hard constraint.
1.2 GENERAL THEORY
We introduce a novel variational bound which is an alternative to and improves upon the traditional reconstruction error as a training objective for DAEs and GSNs. Similar variational bounds have been used for VAEs as well as for the non-equilibrium thermodynamics generative models (SohlDickstein et al., 2015). A distribution P over a chain of samples is defined, which corresponds to iteratively applying transition operators with shared parameters, starting from a pure noise initial state. We would like this process to produce training examples. An inverting flow Q is defined starting from a training example (the “walk-away” trajectory), and following the transition operator of the model, i.e., estimating the posterior distribution of the generative chain produced by P , given that it were landing at a training example. If the model does not match the data distribution, that chain Q will tend to walk away from the training samples, and we want to inhibit that by training P to “walk back”. Instead of using a completely different parametrization for the variational approximation of the posterior (theQ distribution), like in VAEs and non-equilibrium dynamics, we propose to exploit the decomposition of P as a series of stochastic transformations in order to parametrize Q with the same parameters as P , with the step-wise estimated posterior matching the correct one (from P ) for all but the last step of the walk-away trajectory. To make the approximation in the
last step of the chain of walk-away steps better (and thus the variational bound tighter) we introduce the idea of gradually increasing temperature at each step of the walk-away Q chain of transitions (or gradually reducing temperature, at each step of the corresponding walkback trajectory under P ). This also has the advantage that the training procedure will more easily converge to and eliminate spurious modes (those modes of the model where there is no nearby training data). This is because the walk-away Q chain will be making large steps towards the dominant and most attractive modes when the temperature becomes large enough. Unless those modes are near data points, the walkback algorithm will thus “seek and destroy” these modes, these spurious modes.
We present a series of experimental results on several datasets illustrating the soundness of the proposed approach on the MNIST, CIFAR-10 and CelebA datasets.
2 MIXING-FREE TRAINING FRAMEWORK BASED ON THE WALKBACK IDEA
2.1 MAXIMUM LIKELIHOOD TRAINING OF UNDIRECTED GRAPHICAL MODELS
Let v denote the vector of visible units and h denote the vector of hidden random variables, with the full state of the model being s = (v,h). Let pθ denote the model distribution, with joint energy function Eθ and parameter vector θ:
pθ(s) := e−Eθ(s)
Zθ , (1)
where Zθ is the partition function
Zθ :=
∫ e−Eθ(s)ds. (2)
Let pD be the training distribution, from which a sample D is typically drawn to obtain the training set. The maximum likelihood parameter gradient is
Ev∼pD [ −∂ log pθ(v)
∂θ
] = Ev∼pD,h∼pθ(h|v) [ ∂Eθ(v,h)
∂θ
] − Es∼pθ(s) [ ∂Eθ(s)
∂θ
] (3)
which is zero when training has converged, with expected energy gradients in the positive phase (under pD(v)pθ(h|v)) matching those under the negative phase (under pθ(s)). Note that in the (common) case of a log-linear model, the energy gradient (with respect to parameters) corresponds to the sufficient statistics of the model. Training thus consists in matching the shape of two distributions, as captured by the sufficient statistics: the positive phase distribution (influenced by the data, via the visible) and the negative phase distribution (where the model is free-running and generating configurations by itself).
2.2 MIXING-FREE TRAINING FRAMEWORK FOR UNDIRECTED GRAPHICAL MODELS
The basic idea of the proposed mixing-free training framework for undirected graphical models is the following. Instead of trying to match the whole positive phase and negative phase distributions (each of which require a difficult sampling operation, generally with an MCMC that may take very long time to mix between well separated modes), we propose to only match the shape of the energy function locally, around well-chosen points st. Another way to think about this is that instead of trying to directly maximize the likelihood of pθ which requires expensive inference (ideally an MCMC) in the inner loop of training (for each example v ∼ pD), we would like to learn a transition operator pT (st+1|st) such that following it at temperature T = 1 would gradually move the state st towards the data generating distribution.
For this purpose, we propose to use a walkback strategy similar to the one introduced by Bengio et al. (2013c), illustrated in Algorithm 1. The idea is to start from a configuration of s which is compatible with the observed data x, let the state evolve according to our transition operator, and then punish it for these moves, making it more likely to make backwards transitions on this trajectory. If learning was completed, the only moves that would remain are those between highly probable configurations under the data generating distribution. The other ones would be “punished”,
like a child walking away from its designated task and forced to walk back (towards the data)1. Following the model’s inclination in order to generate this random trajectory is more efficient than simply adding noise (like in the denoising auto-encoder (Vincent et al., 2008) or the non-equilibrium dynamics (Sohl-Dickstein et al., 2015) algorithms) because it makes the learning procedure focus its computation on state configurations corresponding to spurious modes to be eliminated. To make sure these spurious modes are approached efficiently, the proposed algorithm also includes the idea of gradually increasing temperature (i.e., the amount of noise) along this walk-away trajectory. At high temperature, the transition operator mixes very easily and quickly reaches the areas corresponding to large spurious modes.
Interestingly, all this comes out naturally of the variational bound presented below, rather than as something imposed in addition to the training objective.
Algorithm 1 VariationalWalkback(θ) Train a generative model associated with a transition operator pT (s|s′) at temperature T (temperature 1 for sampling from the actual model). This transition operator injects noise of variance Tσ2 at each step, where σ2 is the noise level at temperature 1. Require: Transition operator pT (s|s′) from which one can both sample and compute the gradient
of log pT (s|s′) with respect to parameters θ, given s and s′. Require: Precomputed σ2data, the overall variance (or squared diameter) of the data.
repeat Tmax ← σ 2 data
σ2
K ← log2 Tmax Sample x ∼ data (or equivalently sample a minibatch to parallelize computation and process each element of the minibatch independently) Let s0 = (x) and initial temperature T = 1, initialize L = 0 for t = 1 to K do
Sample st ∼ pT (s|st−1) Increment L ← L+ log pT (st−1|st) Update parameters with log likelihood gradient ∂ log pT (st−1|st)∂θ Increase temperature with T ← 2T
end for Increment L ← L+ log p∗(sK)
until convergence (monitoring L on a validation set and doing early stopping)
3 VARIATIONAL LOWER BOUND ON THE LOG-LIKELIHOOD
Let us first consider a way in which our model could approximately generate samples according to our model and the associated transition operator pT (s|s′). That process would start by sampling a state sK inside a volume that contains all the data, e.g., with a broad Gaussian p∗(sK) whose variances are set according to the training data. Then we would sample sK−1 from pTmax(s|s′ = sK), where Tmax is a high enough temperature so that the noise dominates the signal and is strong enough to move the state across the whole domain of the data on the visible portion of the state. If σ2data is the maximum variance of the data (corresponding to the visible dimensions of the state) and σ2 is the amount noise injected by the transition operator on the visible units at temperature 1, then we could pick
Tmax = σ2data σ2
(4)
to achieve that goal. From that point on we are going to continue sampling the “previous” state st according to pT (s|s′ = st+1) while gradually cooling the temperature, e.g. by dividing it by 2 after each step. In that case we would need
K = log2 Tmax (5)
1This analogy with a child was first used in talks by Geoff Hinton when discussing constrastive divergence (personal communication)
steps to reach a temperature of 1. Finally, we would look at the visible portion of s0 to obtain the sampled x. In practice, we would expect that a slower annealing schedule would yield samples more in agreement with the stationary distribution of p1(s|s′), but we explored this aggressive annealing schedule in order to obtain faster training.
The marginal probability of v = x at the end of the above K-step process is thus
p(x) = ∫ sK1 pT0(s0 = x|s1) (∏K t=2 pTt(st−1|st) ) p∗(sK)ds K 1 (6)where Tt is an annealing schedule with T0 = 1 and TK = Tmax and p∗ is the “starting distribution”, such as the Gaussian of variance σ2data. We can rewrite this as follows by taking the log and multiplying and dividing by an arbitrary distribution q(s1, . . . , sK) decomposed into conditionals qTt(st|st−1):
log p(x) = log ∫ sK1 qT0(x)qT1(s1|s0(x, )) ( K∏ t=2 qTt(st|st−1) ) pT0(s0 = x|s1) (∏K t=2 pTt(st−1|st) ) p∗(sK)
qT0(x)qT1(s1|s0 = x) (∏K t=2 qTt(st|st−1) ) dsK1 (7)
where we understand that s0 = x. Now we can apply Jensen’s inequality as usual to obtain the variational bound
log p(x) ≥ L
= ∫ sK1
qT0(x)qT1(s1|s0 = x) ( K∏ t=2 qTt(st|st−1) )
log pT0(s0 = x|s1)
(∏K t=2 pTt(st−1|st) ) p∗(sK)
qT0xqT1(s1|s0 = x) (∏K t=2 qTt(st|st−1) ) dsK1 . (8)
This bound is valid for any q but will be tight when q(sK , sK−1, . . . , s1|s0) = p(sK , sK−1, . . . , s1|s0), and otherwise can be used to obtain a variational training objective. Note that both q and p can be decomposed as a product of one-step conditionals. Here, we can make most of the qTt transition probabilities match their corresponding pTt transition probabilities exactly, i.e., for 1 ≤ t < K we use qTt(s|s′) = pTt(s|s′). (9) The only approximations will be on both ends of the sequence:
• Sampling exactly from the model’s p(v = x) is typically not feasible exactly (it involves the usual posterior inference, e.g., as used in VAEs) but as explained below we will exploit properties of the algorithm to approximate this efficiently. We call the chosen approximation q1(v).
• At the last step, the optimal qTK (sK |sK−1) is not simply the model’s transition operator at temperature TK , because this conditional also involves the marginal “starting distribution” p∗(sK). However, because we have picked TK large enough to make samples from qTmax(sK |sK−1) dominated by noise of the same variance as that of p∗, we expect the approximation to be good too.
3.1 ESTIMATING THE LOG-LIKELIHOOD USING IMPORTANCE SAMPLING
In practice we cannot compute L exactly (nor its gradient), but we can easily obtain an unbiased estimator of L (or of its gradient) by sampling sK1 from the q distributions, i.e., approximate the L integral by a single Monte-Carlo sample. This is what is done by the training procedure outlined in Algorithm 1, which thus performs stochastic gradient ascent on the variational boundL, and this will
tend to also push up the log-likelihood log p(x) of training examples x. Note that such variational bounds have been used successfully in many learning algorithms in the past (Kingma & Welling, 2013; Lamb et al., 2016).
We derive an estimate of the negative log-likelihood by the following procedure. For each training example x, we sample a large number of diffusion paths. We then use the following formulation to estimate the negative log-likelihood.
log p(x) = logEx∼pD,qT0 (x)qT1 (s1|s0(x,))( ∏K t=2 qTt (st|st−1))pT0(s0 = x|s1) (∏K t=2 pTt(st−1|st) ) p∗(sK)
qT0(x)qT1(s1|s0 = x) (∏K t=2 qTt(st|st−1) )  (10)
4 TRANSITION OPERATORS FOR VARIATIONAL WALKBACK
Up to now we have not specified what the form of the transition operators should be. Two main variants are possible here. Either we directly parametrize the transition operator, like with denoising auto-encoders or generative stochastic networks, or we obtain our transition operator implicitly from some energy function, for example by applying some form of Gibbs sampling or Langevin MCMC to derive a transition operator associated with the energy function.
An advantage of the direct parametrization is that it eliminates the constraint to have symmetric weights, which is interesting from the point of view of biological plausibility of such algorithms. An advantage of the energy-based parametrization is that at the end of the day we get an energy function which could be used to compute the unnormalized joint probability of visible and latent variables. However, note that in both cases we can easily get an estimator of the log-likelihood by simply using our lower bound L, possibly improved by doing more expensive inference for pTK (sK |sK−1).
4.1 PARAMETRIC TRANSITION OPERATOR
In our experiments we considered Bernoulli and isotropic Gaussian transition operators for binary and real-valued data respectively.
When we sample from the transition operator we do not attempt to pass gradients through the sampling operation. Accordingly, backpropagation is performed locally on each step of the walk-back, and there is no flow of gradient between multiple walk-back steps.
Additionally, we use a “conservative” transition operator that averages its input image together with the sample from the learned distribution (or takes a weighted average with a fixed α weighting) for the transition operator. Just after parameter initialization, the distribution learned by the transition operator’s output is essentially random, so it is very difficult for the network to learn to reconstruct the value at the previous step.
Bernoulli Transition Operator
ρ = sigmoid( (1− α) ∗ xt−1 + α ∗ Fρ(xt−1)
Tt ) (11)
Gaussian Transition Operator
µ = (1− α) ∗ xt−1 + α ∗ Fµ(xt−1) (12)
σ = sigmoid(Tt log(1 + e Fσ(xt−1))) (13)
Fρ, Fµ, Fσ are functions (in our case neural networks) which take the previous x value from the walkback chain and return estimates of the value of µ and σ respectively. T is the temperature which is dependent on the walkback step t. xt−1 is the previous value in the walkback chain.
5 RELATED WORK
Contrastive Divergence
This algorithm is clearly related to the contrastive divergence algorithm with k = T steps (CDk). The CD-k algorithm approximates the log-likelihood gradient by trying to match the sufficient statistics with the data clamped to the sufficient statistics after k steps of the transition operator. The parameter update is the difference of these sufficient statistics, which also corresponds to pushing down the energy of the data-clamped configuration while pushing up the energy of the random variables after k steps of the transition operator.
Two important differences are that, because the temperature is increasing in the variational walkback procedure,
1. the energy gradients ∂E(s)∂s do not cancel each other telescopically along the chain from s0 to sT ,
2. as t increases we move more and more randomly rather than following the energy of the model, allowing to hunt more effectively the areas near spurious modes.
A third difference is that the learning procedure is expressed in terms of the transition operator rather than directly in terms of the energy function. This allows one to thus train a transition operator directly, rather than indirectly via an energy function.
Generative Stochastic Networks
The Generative Stochastic Networks (GSN) algorithm proposed by Bengio et al. (2013b) learns a transition operator by iteratively injecting noise and minimizing the reconstruction error after a number of transition operator steps starting at a data point, and back-propagating through all these steps. One thing in common is the idea of using the walkback intuition instead of isotropic noise in order to converge more efficiently. A major difference is that the algorithm proposed for GSNs involves the minimization of overall reconstruction error (from the input data point x to the sampled reconstruction many steps later). This will tend to blur the learned distribution. Instead, the variational walk-back algorithm minimizes reconstruction error one step at a time along the walk-away trajectory.
In addition, the variational walkback GSNs require back-propagating through all the iterated steps, like the DRAW algorithm (Gregor et al., 2015). Instead the variational walk-back algorithm only requires back-propagating through a single step at a time of the transition operator. This should make it easier to train because we avoid having to optimize a highly non-linear transformation obtained by the composition of many transition operator steps.
Non-Equilibrium Thermodynamics
There are two main differences between the Variational Walkback algorithm and the NonEquilibrium Thermodynamics:
1. Instead of isotropic noise to move away from the data manifold, we propose to use the model’s own transition operator, with the idea that it will “seek and destroy” the spurious modes much more efficiently than random moves.
2. Instead of injecting a fixed amount of noise per time step, we increase the noise as it moves away from the data manifold, and anneal the noise when we are close to the data manifold. This way, we can quickly reach the noise prior without loosing the details of the data. Our model takes significantly fewer steps to walk away and back to the manifold, as compared to the 1000 steps used for Non-Equilibrium Thermodynamics.
Annealed Importance Sampling (AIS)
Annealed Importance Sampling is a sampling procedure. Like variational walkback, it uses an annealing schedule corresponding to a range of temperature from infinity to 1. It is used to estimate a partition function. Unlike Annealed Importance Sampling, variational walkback is meant to provide a good variational lower bound for training a transition operator.
Reverse Annealed Importance Sampling Estimator (RAISE)
RAISE is a reverse AIS, as it starts from a data point and then increases the temperature. In this way it is similar to the Q-chain in variational walkback. The advantage of RAISE over AIS is that it yields an estimator of the log-likelihood that tends to be pessimistic rather than optimistic, which makes it better as an evaluation criteria.
Like AIS, RAISE estimates the log-likelihood using a form of importance sampling, based on a product (over the chain) of the ratios of consecutive probabilities (not conditional probabilities from the model). Variational walkback does not work with estimates of the model’s unconditional probability, and instead works directly with a conditional probability defined by the transition operator. It is for this reason that variational walkback does not need to have an explicit energy function).
6 EXPERIMENTS
We evaluated the variational walkback on three datasets: MNIST, CIFAR (Krizhevsky & Hinton, 2009), and CelebA (Liu et al., 2015). The MNIST and CIFAR datasets were used as is, but the aligned and cropped version of the CelebA dataset was scaled from 218 x 178 pixels to 78 x 64 pixels and center-cropped at 64 x 64 pixels (Liu et al., 2015). For all of our experiments we used the Adam optimizer (Kingma & Ba, 2014) and the Theano framework (Al-Rfou et al., 2016). The training procedure and architecture are detailed in appendix A.
We reported samples on CIFAR, MNIST, CelebA and inpainting results on MNIST. Our inpainting results on MNIST are competitive with generative stochastic networks and show somewhat higher consistency between the given part of the image and the generated portion (Bengio et al., 2013c). However, we note that our samples on CIFAR and CelebA show the same “blurring effect” that has been observed with autoencoder-based generative models trained to minimize reconstruction loss (Lamb et al., 2016).
7 CONCLUSION AND FUTURE WORK
We have introduced a new form of walk-back and a new algorithm for learning transition operators or undirected graphical models. Our algorithm learns a transition operator by allowing the model to walk-away from the data towards the noise prior and then teaching it to actually to have its transitions trained to go backwards each of these walk-away steps, i.e., towards the data manifold. Variational walk-back increases the temperature along the chain as it is moving further away from the data manifold, and inversely, anneals the temperature at generation time, as it gets closer to the estimated manifold. This allows the training procedure to quickly find and remove dominant spurious modes. Learning a transition operator also allows our model to learn only a conditional distribution at each step. This is much easier to learn, since it only needs to capture a few modes per step. The model also only locally carves the energy function, which means that it does not have to learn the entire joint probability distribution, but rather steps towards the right direction, making sure that everywhere it puts probability mass as well as around the data, the energy gradient is pointing towards the data.
Our experimental results have confirmed that the model can walk towards the data manifold in a few steps, even when the modes are sharp.
Future work should extend this algorithm and experiments in order to incorporate latent variables. The state would now include both the visible ~x and some latent ~h. Essentially the same procedure can be run, except for the need to initialize the chain with a state ~s = (~x,~h) where ~h would ideally be an estimate of the posterior distribution of ~h given the observed data point ~x. Another interesting direction to expand this work is to replace the log-likelihood objective at each step by a GANlike objective, thus avoiding the need to inject noise independently on each of the pixels, during one application of the transition operator, and allowing the latent variable sampling to inject all the required high-level decisions associated with the transition. Based on the earlier results from Bengio et al. (2013a), sampling in the latent space rather than in the pixel space should allow for better generative models and even better mixing between modes.Bengio et al. (2013b)
ACKNOWLEDGMENTS
The authors would like to thank Benjamin Scellier and Aaron Courville for their helpful feedback and discussions, as well as NSERC, CIFAR, Google, Samsung, Nuance, IBM and Canada Research Chairs for funding, and Compute Canada for computing resources.
A ARCHITECTURE DETAILS
The architecture that was used for the CelebA and CIFAR dataset was similar to the architecture used by Lamb et al. (2016), with a convolutional encoder followed by two fully connected hidden layers, followed by a decoder with strided convolutions (Radford et al., 2015). Batch norm was applied in all layers except for the last layer. For all layers except for the last we used the tanh activation function. Surprisingly, we were unable to obtain good results using the RELU or Leaky RELU activation .
On the binarized MNIST dataset we used a transition operator with Bernoulli outputs. A feedforward neural network was used to estimate the parameters (per-pixel probabilities) for the Bernoulli outputs. This neural network consisted of a single hidden layer with 4096 hidden units and the tanh activation function.
B WALKBACK PROCEDURE DETAILS
The variational walkback algorithm has three unique hyperparameters. One is the number of walkback steps performed during training. Another is the number of walkback steps performed when sampling from the model. Still another is the temperature schedule used during training, reconstruction, or sampling.
The most conservative hyperparameter setting would involve using a large number of walkback steps during training and slowly increasing the temperature. However, this could make training slow, and if too few steps are used, the end of the walkback chain will not match the noise prior, leading to low quality samples.
A dynamic approach to setting the number of walkback steps and temperature schedule may be possible, but in this work we set these hyperparameters empirically. We found that during training using a temperature schedule of T = T0 √ 2t produced good results, where T0 = 1.0 is the initial temperature and t is the step index. During sampling, we found good results using the reverse schedule: T =
√ 2N√ 2t , where t is the step index and N is the total number of sampling steps.
For MNIST, we achieved our results using 8 training steps of walkback. For CIFAR, we used 15 training steps and 20 sampling steps. For CelebA, we used 30 training steps and 35 sampling steps. In general, we found that we could achieve higher quality results by using more steps during sampling then we used during training. We found that more difficult datasets, like CIFAR and CelebA, required longer walkback chains. Finally, our model is able to achieve results competitive with Non-Equilibrium Thermodynamics (Sohl-Dickstein et al., 2015), despite that method requiring chains with far more steps (1000 steps for MNIST).
C ALTERNATIVE FORMULATION OF VARIATIONAL BOUND
The marginal probability of v = x at the end of the above K-step process is thus
p(x) = ∫ sK1 ( K∏ t=1 pTt(st−1|st) ) p∗(sK)ds K 1 (14)
where Tt is an annealing schedule with T0 = 1 and TK = Tmax and p∗ is the “starting distribution”, such as the Gaussian of variance σ2data. We can rewrite this as follows by taking the log and multiplying and dividing by an arbitrary distribution q(s1, . . . , sK) decomposed into conditionals qTt(st|st−1):
q(s0, s1, ..., sk) = ( K∏ t=1 qTt(st|st−1) ) q(sK) (15)
giving us:
log p(x) = log ∫ sK1 qT0(x) ( K∏ t=1 qTt(st|st−1) ) (∏K t=1 pTt(st−1|st) ) p∗(sK) qT0(x) (∏K t=1 qTt(st|st−1) ) dsK1 (16)
where we understand that s0 = x. Now we can apply Jensen’s inequality as usual to obtain the variational bound
log p(x) ≥ L = ∫ sK1 qT0(x) ( K∏ t=1 qTt(st|st−1) ) log (∏K t=1 pTt(st−1|st) ) p∗(sK) qT0(x) (∏K t=1 qTt(st|st−1) ) dsK1 . (17)
D TIGHTNESS OF THE VARIATIONAL BOUND
We present an argument that running the walkback chain for a sufficient number of steps will cause the variational bound to become tight.
Consider a sequence st, ..., s1 generated in that order by our model p through a sequence of applications of the transition operator T, i.e., p(s1, ..., st) = p(st)T (st−1|st)...T (s1|s2), i.e. p(sn−1|sn) = T (sn−1|sn), but note that p(sn|sn−1) 6= p(sn−1|sn). Let pi(s) denote the stationary distribution associated with T. Note that T and pi and related by the detailed balance equation, i.e., T (s|s′)pi(s′) = T (s′|s)pi(s). We want to approximate the posterior
p(st, st−1, ..., s2|s1) = ∏t n=2 p(sn|sn−1)
now by Bayes rule = ∏t n=2 p(sn−1|sn) p(sn) p(sn−1) by telescopic cancellation and definition of T
= p(st)p(s1) ∏t n=2 T (sn−1|sn) now by detailed balance
= p(st)p(s1) ∏t n=2 T (sn|sn−1) pi(sn−1) pi(sn) by telescopic cancellation = p(st)pi(st) pi(s1) p(s1) ∏t n=2 T (sn|sn−1) again by definition of T = p(st)pi(st) pi(s1) p(s1) ∏t n=2 p(sn|sn−1)
So our approximation error in the posterior is the factor p(st)pi(st) pi(s1) p(s1) .
If t is large enough, then s1 (being at the end of the generative sequence) has pretty much converged, i.e., p(s1) ≈ pi(s1). If we throw in temperature annealing along the way (now the notation would have to be changed to put an index n on both p and T), with the initial temperature being very high, then we can hope that the initial Gaussian p(st) is very similar to the stationary distribution at high temperature pi(st).
These arguments suggest that as we make t larger and the final (initial) temperature larger as well, the approximation becomes better.

I very much like the underlying idea for this paper. I wasn't convinced by the execution in its current state. My primary concern is the one I expressed in my pre-review question below, which I don't think the authors addressed. Specifically, I think the choice of q(s | s') = p(s | s') will make the forward and reverse trajectories almost pathologically mismatched to each other, and will thus make the variational bound extremely loose and high variance.
The claim about the tightness of the bound in Appendix D relies on the assumption that the transition distribution obeys detailed balance. The learned transition distribution in the paper does not obey detailed balance, and therefore the tightness claim in Appendix D does not hold. (In Section 2.1 you briefly discuss the idea of learning an energy function, rather than directly learning a transition distribution. I think this would be excellent, and in that case you could choose an MCMC transition operator that does obey detailed balance for that energy function.) I did not go through Appendix D beyond this step.
The experimental results were not visually impressive. I suspect this is primarily driven by the mismatch between generative and inference trajectories. See my concern above and in the pre-review question below.
Also, see note below for sec. 5. I suspect some terms are being dropped from the training gradient.
The paper is optimizing a variational bound on log likelihood. You should really, really, really report and compare log likelihoods against competing methods!
Detailed comments below. Some of these were written based on a previous version of the paper.
sec 1.2 - first paragraph is very difficult to follow
"these modes these spurious modes" -> "these spurious modes"
sec 2.1 - "s = (v,h)" -> "s = {v,h}"
sec 2.2 - "with an MCMC" -> "with an MCMC chain"
"(ideally an MCMC)" -> "(e.g. via MCMC)" MCMC is not ideal ... it's just often the best we can do.
sec 3, last bullet - could make the temperature infinite for the last step, in which case the last step will sample directly from the prior, and the posterior and the prior will be exactly the same.
sec. 4 -- Using an energy function would be great!! Especially, because many MCMC transition operators obey detailed balance, you would be far less prone to suffer from the forward/backward transition mismatch that is my primary concern about this technique.
eq. 12,13 -- What is alpha? How does it depend on the temperature. It's never specified.
sec. 5, last paragraph in GSN section -- Note that q also depends on theta, so by not backpropagating through the full q chain you are dropping terms from the gradient.
sec. 5, non-equilibrium thermodynamics -- Note that the noneq. paper also increases the noise variance as the distance from the data increases.
Fig. 1 -- right/left mislabeled
Fig. 2 -- label panes
Fig. 3 -- After how many walkback steps?