Scalable Planning with Tensorflow for Hybrid Nonlinear Domains
Given recent deep learning results that demonstrate the ability to effectively optimize high-dimensional non-convex functions with gradient descent optimization on GPUs, we ask in this paper whether symbolic gradient optimization tools such as Tensorflow can be effective for planning in hybrid (mixed discrete and continuous) nonlinear domains with high dimensional state and action spaces? To this end, we demonstrate that hybrid planning with Tensorflow and RMSProp gradient descent is competitive with mixed integer linear program (MILP) based optimization on piecewise linear planning domains (where we can compute optimal solutions) and substantially outperforms state-of-the-art interior point methods for nonlinear planning domains. Furthermore, we remark that Tensorflow is highly scalable, converging to a strong plan on a large-scale concurrent domain with a total of 576,000 continuous action parameters distributed over a horizon of 96 time steps and 100 parallel instances in only 4 minutes. We provide a number of insights that clarify such strong performance including observations that despite long horizons, RMSProp avoids both the vanishing and exploding gradient problems. Together these results suggest a new frontier for highly scalable planning in nonlinear hybrid domains by leveraging GPUs and the power of recent advances in gradient descent with highly optimized toolkits like Tensorflow.
This paper describes a gradient ascent approach to action selection for planning domains with both continuous and discrete actions. The paper shows that backpropagation can be used to optimize the actions while avoiding long horizon problems such as exploding or vanishing gradients. Experimental results are given for 2D planar navigation, reservoir control and HVAC control and the proposed approach is shown to outperform existing hybrid optimization techniques.
Planning in hybrid domains is an important problem and existing methods do not scale to real world domains. This paper proposes a scalable solution to these problems and demonstrates its usefulness on several domains including two real world domains. As such, I believe it makes an important contribution to addressing real world planning problems. The paper is well written except for some minor comments (see below).
Evidence that the proposed solution works comes in the form of several experiments that demonstrate scalability and solution quality.
I have two concerns about the experiments that I would like to see the author's comments on:
1. The paper lacks any statistical significance results, error bars, or other statistics of variation.
2. Are the experimental baselines the state-of-the-art of hybrid domain planning problems? The introduction mentions other methods but justification is not given for not making an empirical comparison.
This paper needs a distinct related work section. The current treatment of related work is limited and is just in the introduction.
My last major question is whether or not Tensorflow is a critical part of the proposed method? It seems that what is important is the use of backpropagation and Tensorflow is just an implementation detail. I think the impact of the work might be more if the method was presented more generally. Or is there something I'm missing that makes Tensorflow essential?
Overall, the work presents a useful method for planning in hybrid domain problems. Its main limitations are the handling of related literature and statistical significance. Clarifying these issues and addressing the minor comments given below would help improve the impact of the work.
I have read the author's response and other reviews.
Minor comments:
Line 35: I.e., -> i.e.,
Line 35: Should cite Watkins for Q-learning
Line 106: Do you mean optimizing V directly?
Line 117: What do you mean by transfer function? Did you mean activation function?
Line 134: This statement seems too strong? Why are there always constraints on the actions? Is it more appropriate to say, "planning problems frequently have constraints on the actions that can be taken"?
Line 164 - 167: The top line (d_t = ||s_t - s||) is too far over.
Line 170: It is not clear what the dynamics are for linear and bilinear. It would be useful to write them explicitly as done for the nonlinear dynamics.
Figure 3: This figure would be more clear if the y-axis was in the correct direction. It would also be useful to have a statement such as "lower is better" so the reader can quickly compare TF to the other baselines.
Figure 3: Since the MILP solution is optimal it would be more useful to see relative error values for TF and Heuristic. Absolute reward isn't as informative since its sensitive to reward scaling.
All figures: It would be nice to see error bars on all applicable plots
Line 235: "to for" -> for
Figure 6: space before (b)
All figures: check for black and white printing, especially figure 6
Line 262: RMSProb -> RMSProp