We propose a meta path planning algorithm named Neural Exploration-Exploitation Trees (NEXT) for learning from prior experience for solving new path planning problems in high dimensional continuous state and action spaces. Compared to more classical sampling-based methods like RRT, our approach achieves much better sample efficiency in high-dimensions and can benefit from prior experience of planning in similar environments. More specifically, NEXT exploits a novel neural architecture which can learn promising search directions from problem structures. The learned prior is then integrated into a UCB-type algorithm to achieve an online balance between exploration and exploitation when solving a new problem. We conduct thorough experiments to show that NEXT accomplishes new planning problems with more compact search trees and significantly outperforms state-of-the-art methods on several benchmarks.
Summary:
Motion-planning in high dimensional spaces is challenging due to the curse of dimensionality. Sampling-based motion planners like PRM, PRM*, RRT, RRT*, BIT* etc have been the go-to solution family. But often these algorithms solve every planning problem tabula rasa. This work combines learning with sampling-based planning such that the parent-sampling and expansion steps instead of being done by common heuristics are learnt in an online manner. Also the resulting exploration-exploitation problem is naturally dealt via using a UCB-style contextual bandit algorithm. Since the number of parents are always varying the common trick of 'describe the action choices with respect to the environment' is adopted so that varying number of actions (states to be sampled from) can be naturally incorporated. 
The other significant aspect of this paper is that there is a self-improving component (Algorithm 3) where a dataset is built up every time step, of environments where either an expansion with RRT or the learnt expansion policy is attempted with the policy being invoked more as time goes on and it trains more. If the process succeeds in finding a path to the goal then this example is added to a dataset and the dataset used to update the policy and associated value function to guide it towards the feasible paths found in the tree. 
Comments:
- Algorithm 3: "Reconstruct optimal path". These paths are not really optimal for the problem. They are optimal in the tree T that is built so far for example U. But for the problem they are feasible and if RRT* were to be run asymptotically then perhaps near-optimal. The accompanying text should be updated accordingly so that there isn't confusion.
- Here is my main concern with Algorithm 3: For equation 6  where the policy and value functions are updated, the policy is inevitably going to suffer from covariate shift. This is because the algorithm is essentially doing behavior cloning (BC) with respect to the feasible paths found on the planning examples. Since we are inherently in a sequential setting (non-iid) where the states visited by the policy are a direct result of its own decisions the error bound will be quadratic in the horizon (path-length) for equation 6. This phenomenon has been well-understood in imitation learning literature and algorithms like DAgger, AggreVate or online versions like AggreVateD, LOLS already address these problems in a principled manner. Equation 6 should ideally be replaced with an inner DAgger/AggreVateD like loop (with an RRT* dynamic oracle) for stable learning of policy and value function. I am happy to be convinced that covariate shift and resulting quadratic mistake-bound problems are not present here.
- Application of imitation learning to both self-improvement style path planning and leveraging experience in planning has been done before: See "Learning to Search via Retrospective Imitation
Jialin Song, Ravi Lanka, Albert Zhao, Aadyot Bhatnagar, Yisong Yue, Masahiro Ono, 2018" (this is unpublished it seems so it is unfair of me to mention this perhaps but I wanted to give an example of how to use dynamic oracles for stable imitation in planning.) and "Data-driven Planning via Imitation Learning
Sanjiban Choudhury, Mohak Bhardwaj, Sankalp Arora, Ashish Kapoor†, Gireeja Ranade, Sebastian Scherer and Debadeepta Dey", IJRR 2018. At least the last paper should be cited and discussed in related work.
- Also would be curious how the authors would situate methods which are non-learning based but leverage experience in planning (example E-Graphs: Bootstrapping Planning with Experience Graphs, Phillips et al, RSS 2012) via graphs discovered in other problems directly. Perhaps a discussion in related work is warranted?
Update: After rebuttal updating to Accept.