On Blackbox Backpropagation and Jacobian Sensing
From a small number of calls to a given "blackbox" on random input perturbations, we show how to efficiently recover its unknown Jacobian, or estimate the left action of its Jacobian on a given vector. Our methods are based on a novel combination of compressed sensing and graph coloring techniques, and provably exploit structural prior knowledge about the Jacobian such as sparsity and symmetry while being noise robust. We demonstrate efficient backpropagation through noisy blackbox layers in a deep neural net, improved data-efficiency in the task of linearizing the dynamics of a rigid body system, and the generic ability to handle a rich class of input-output dependency structures in Jacobian estimation problems.
The paper focuses on automatic differentiation of multiple-variable vector-valued functioned in the context of training model parameters from input data. A standard approach in a number of popular environments rely on a propagation of the errors in the evaluation of the model parameters so far using backpropagation of the estimation error as computed of the (local) gradient of the loss function. In this paper, the emphasis is on settings where some of the operators of the model are exogenous blackboxes for which the gradient cannot be computed explicitly and one resorts to finite differencing of the function of interest. Such approach can be prohibitively expensive if the Jacobian does not have some special structure that can be exploited. 
 
The strategy pursued in this paper consists in exploiting the relationship between graph colouring and Jacobian estimation. In a nutshell, the mapping encodes the outputs that jointly influence the model. The consequence of this is a known result stating that the chromatic number of the associated graph provides a bound on the number of calls to a function to compute its Jacobian, in the noiseless setting. The paper goes one step beyond this established fact and proposes an algorithms “rainbow” that defines the perturbation directions where the blackbox needs to be evaluated in order to compute the associated Jacobian . The crux of the algorithm is to encode the properties of the Jacobian (e.g. sparsity, symmetry) using a graph colouring procedure, arising from the aforementioned graph mapping, into a simple linear programme.
 
The authors theoretically evalute the performance of this algorithm in section 3.4.  Theorem 3.3. upperbounds the number of blackbox needed to evaluate its Jacobian in terms of the graph encoding input output dependencies as well as its intrinsic degrees of freedom among other things. In Section 4, the proposed algorithm is evaluated in following settings: two synthetic input-output dependency structures (random and power-law), a CNN, and a planning and control setting.
 
The paper is clearly written and properly spells out the relevant technical details of the proposed algorithm as well as its performance. 
The main weakness of the paper resides in section 3.3, focusing on the LP approach to define the perturbation direction needed to evaluate the Jacobian. There are also two minor points that would improve the legibility of the paper. First, the paragraph after theorem 3.3 (page 7 l.257-l .260) should provide further intuition about the bounds on the calls to the function needed by the algorithm to estimate its Jacobian. Then, the experiment section 4 would benefit from providing a rationale behind the choice of the scenarios explored and what is illustrated about the algorithm in each of them.