Many machine learning systems are implemented as pipelines. A pipeline is essentially a chain/network of information processing units. As information flows in and out and gradients vice versa, ideally, a pipeline can be trained end-to-end via backpropagation provided with the right supervision and loss function. However, this is often impossible in practice, because either the loss function itself may be non-differentiable, or there may exist some non-differentiable units. One popular way to superficially resolve this issue is to separate a pipeline into a set of differentiable sub-pipelines and train them with isolated loss functions. Yet, from a decision-theoretical point of view, this is equivalent to making myopic decisions using ad hoc heuristics along the pipeline while ignoring the real utility, which prevents the pipeline from behaving optimally. In this paper, we show that by converting a pipeline into a stochastic counterpart, it can then be trained endto-end in the presence of non-differentiable parts. Thus, the resulting pipeline is optimal under certain conditions with respect to any criterion attached to it. In experiments, we apply the proposed approach -reinforced pipeline optimization -to Faster R-CNN, a state-of-the-art object detection pipeline, and obtain empirically near-optimal object detectors consistent with its base design in terms of mean average precision.
Pros:
+ Improving joint training of non-differentiable pipelines is a meaningful and relevant problem
+ Using the stochastic computation graph structure to smooth a pipeline in a structured way is a plausible idea
Cons:
+ The main result of the paper concerning sufficient conditions for optimality of the method seems dubious
+ It is not obvious why this method would outperform simple baselines, and baselines for joint training were tried
+ The notation seems unnecessarily bloated and overly formal
+ The exposition spends too much time on prior work, too little on the contribution, and the description of the contribution is confusing
The submission describes a method for smoothing a non-differentiable machine learning pipeline (such as the Faster-RCNN detector), so that gradient-based methods may be applied to jointly train all the parameters of the pipeline.  In particular, the proposal involves recasting the pipeline as a stochastic computation graph (SCG), adding stochastic nodes to this graph, and then using REINFORCE-style policy gradients to perform parameter learning on the SCG.  It is claimed that under certain conditions, the optimal parameters of the resulting SCG are also optimal for the original pipeline.  The method is applied to optimizing the parameters of Faster-RCNN.
I think making non-differentiable pipelines differentiable is an intuitively appealing concept.  A lot of important, practical machine learning systems fall into this category, so devising a nice way to do global parameter optimization for such systems could potentially have significant impact.  In general, we can’t hope to make much meaningful progress on the problem of optimizing general nonlinear, differentiable functions, but it is plausible that a method that targets key non-differentiable components for smoothing—such as this paper—could outperform a generic black-box optimizer.  So, I think the basic idea here is plausible and addresses an important problem.
Unfortunately, I think this work loses sight of that high-level goal: to me, the key question is whether the proposed approach outperforms any other simple method for global parameter optimization in the presence of nonlinearities and nondifferentiability.  The paper fails to answer this question because no baselines for global parameter optimization were tried.  We can just treat the pipeline as a black box mapping parameters to training set performance, and so any black-box optimization method can be applied to this problem.  It is not clear that the proposed method would outperform an arbitrary black box optimization method such as simulated annealing, Nelder-Mead, cross-entropy method, etc.
I think there are also much simpler methods in a similar vein to the proposed method that might also perform just as well as the proposal.  One key conceptual issue here is that reducing the problem to a reinforcement learning problem, as the submission does, is not much of a reduction at all.  First, if the goal is to do global parameter optimization, then we don’t really have to smooth the pipeline itself: we can just smooth the black box mapping parameters to performance, and then optimize that with SGD.  There are many ways to do this--if we want to use policy gradient, we can just express the problem as something in this form:
min_\phi E_{\theta ~ q_\phi} C(\theta)
where C is the black-box mapping parameters \theta to a performance index (such as mean AP), q_\phi is a distribution over parameters (e.g., Gaussian), and \phi are the distribution parameters (e.g., mean, covariance of the Gaussian).  We can then optimize this using REINFORCE policy gradients.
If we want to really smooth the pipeline itself, then it is also easy to do this by devising a suitable MDP and then applying REINFORCE with the usual MDP structure.  We simply identify the state s_t at time t with the output of the t’th pipeline stage, introduce a new ‘action’ variable a_t representing a ’stochastified output’, and trivial dynamics (P(s_{t+1} | s_t, a_t) = \delta(s_{t+1} - a_t)).  If the policy is a Gaussian (P(a_t | s_t) = N(a_t; s_t, \Sigma)), then this is similar to relaxing the constraint that one stage’s output is equal to the input of the next stage, and somehow quadratically penalizing their difference.  In fact, there is a neural network training method based explicitly on this penalization view [A], and it would make yet another great baseline to try.
In fact, the proposed method is essentially similar to what I have just described, but it is unfortunately described in an overcomplicated way that obscures the true nature of the method.  I think the whole SCG framework is overkill here.  Too much of the paper is spent just rehashing the SCG framework, and the very heavy notation again just obscures the essential character of the method.
If there were, as the paper claims, some interesting condition under which the method produces solutions that are optimal under the original pipeline, that would be remarkable and interesting.  However, I have serious doubts about this part of the paper.  The key problem is the statement that “It follows that c(k_c, DEPS_c - k_c) = c(…) + z_c”.  The paper seems to be claiming that if E z = 0, then c(k + z) = c(k) + z, which can’t possibly be true in general.  
The heavy and opaque notation makes it very difficult to understand this section.  Perhaps it would help to consider a very simple example.  Suppose we want to minimize E_{x ~ q} c(y(x)) (where x ~ q means x is distributed as q).  We can introduce only one new stochastic node (k = y + z), between y and c.  Clearly c(y + z) is not generally equal to c(y) + z, even if E z = 0.
In summary, I think the submission needs a lot of work on multiple axes before it can make a significant impact.  The most important issues are a complete lack of relevant baselines and the dubious claims about sufficient conditions for optimality.  The idea could have merit, but it needs to be carefully compared and motivated with respect to existing work (such as [A]) as well as the simple baselines I have mentioned.  The presentation also needs to be revised to find the simplest expression of the method and to focus on the interesting parts.
[A] Taylor, Gavin, et al. "Training neural networks without gradients: A scalable admm approach." International Conference on Machine Learning. 2016.