Gradient Estimation Using Stochastic Computation Graphs
In a variety of problems originating in supervised, unsupervised, and reinforcement learning, the loss function is defined by an expectation over a collection of random variables, which might be part of a probabilistic model or the external world. Estimating the gradient of this loss function, using samples, lies at the core of gradient-based learning algorithms for these problems. We introduce the formalism of stochastic computation graphs-directed acyclic graphs that include both deterministic functions and conditional probability distributions-and describe how to easily and automatically derive an unbiased estimator of the loss function's gradient. The resulting algorithm for computing the gradient estimator is a simple modification of the standard backpropagation algorithm. The generic scheme we propose unifies estimators derived in variety of prior work, along with variance-reduction techniques therein. It could assist researchers in developing intricate models involving a combination of stochastic and deterministic operations, enabling, for example, attention, memory, and control actions.
The manuscript proposes a formalism for computing stochastic estimates of gradients for loss functions.The formalism, referred to as stochastic computation graphs, is very general, applying to models with deterministic and stochastic components, and allowing the computation of gradient estimates for a broad range of models.Methods for deep networks, variational inference, and reinforcement learning are identified as special cases of the proposed framework.The proposed stochastic computation graphs are essentially Bayesian networks which may also contain deterministic nodes, and which are interpreted as encoding distributions over loss functions that are to be minimized.The gradient estimation algorithm extends backpropagation to the partially stochastic case covered by these models, by simply composing score function estimators and pathwise derivative estimators.The main result of the paper is a theorem stating that the gradient estimation algorithm recovers an unbiased estimate of the gradient for these models, under broad conditions.The formalism and algorithm are both simple and represent fairly small adaptions of existing ideas.However, the ideas are likely to have substantial practical impact by facilitating automatic optimization of a wide range of custom models and architectures.The approach is both elegant and general.The biggest weakness of the paper is a complete lack of empirical analysis.This cannot be excused by considering it a "theory paper," since the proposed algorithm is practically motivated, and derives much of its potential impact from its potential use.It would GREATLY strengthen the paper to include experimental results demonstrating that the method is competitive with state of the art special-purpose methods on a variety of real world problems.It would also be insightful to explore and graph its performance on randomly generated problems, varying difficulty parameters such as the depth of the networks, percent of stochastic nodes, variance of stochastic nodes, number of parameters, etc.The only mitigating factor regarding the lack of empirical results is that special cases of the method are identified in the literature, thereby indirectly demonstrating that it can have practical utility.Together with the potential impact of the work, this leads me to consider it for publication, despite the above reservations.