Task-based End-to-end Model Learning in Stochastic Optimization
With the increasing popularity of machine learning techniques, it has become common to see prediction algorithms operating within some larger process. However, the criteria by which we train these algorithms often differ from the ultimate criteria on which we evaluate them. This paper proposes an end-to-end approach for learning probabilistic machine learning models in a manner that directly captures the ultimate task-based objective for which they will be used, within the context of stochastic programming. We present three experimental evaluations of the proposed approach: a classical inventory stock problem, a real-world electrical grid scheduling task, and a real-world energy storage arbitrage task. We show that the proposed approach can outperform both traditional modeling and purely black-box policy optimization approaches in these applications.
--Brief summary of the paper:
The paper proposes a learning method for solving two-stage stochastic programming problems which involve minimizing f(x,y,z) w.r.t. z. The main idea of the paper is to learn a predictive model p(y|x;theta) such that the task's objective function f is directly optimized. In contrast, traditional approaches learn p(y|x;theta) to minimize a prediction error without considering f. The main technical challenge in the paper is to solve a sub-optimization problem involving argmin w.r.t. z, and the proposed method can do so in an efficient manner by assuming that the optimization problem is convex in z. The method is experimentally evaluated on two problems and it is shown to outperform traditional methods.
--Major comments:
The idea of adopting end-to-end learning to solve two-stage stochastic programming is interesting. However, I have a major concern for the proposed method which is the lack of convergence guarantees. Since the optimization problem is assumed to be convex in z, the obtained solution z*(x;theta) is supposed to be the "true" optimal if data is drawn from the true distribution p(x,y). However, a solution obtained using the predictive model p(y|x;theta) is unlikely to be true optimal unless p(y|x;theta) is the true conditional distribution p(y|x). (This issue is commonly known as model bias in the context of model-based reinforcement learning which usually involves non-convex objectives.) Since the proposed method does not theoretically guarantee that p(y|x;theta) converges to p(y|x) even when the model hypothesis is correct, it seems likely that even for a convex optimization problem the method may only find a sub-optimal solution. For this reason, I think having convergence guarantees or error bounds either for the predictive model or for the obtained solution itself are very important to theoretically justify the method and would be a significant contribution to the paper.
--Questions:	
1) It is not clear why Algorithm 1 requires mini-batches training since Line 7 of the algorithm only checks the constraint for a single sample.
2) In the first experiment, why does the performance of the end-to-end policy optimization method depend on the model hypothesis when it does not rely on a predictive model?
--Minor suggestions:
1) In line 154 the paper argue that the model-free approach requires a rich policy class and is data inefficient. However, the model-based approach also requires a rich model class as well. Moreover, the model-based approach can suffer from model bias while the model-free approach cannot.
2) The applicability of the proposed method is quite limited. As mentioned in the paper, solving a sub-optimization problem with argmin is not trivial and convexity assumption can help in this regard. However, practical decision making problems may involve non-convex or unknown objective functions. A variant of the proposed method that is applicable to these tasks would make the method more appealing.
3) The last term of Eq.(4) should have an expectation over the density of x.
--Comments after author's response:
I feel more positive about the paper after reading the author’s response. Now I think that the proposed method is an important contribution to the field and I will increase my score. However, I am still not convince that the proposed method will be useful outside domains with convex objectives without empirical evidences.