Decision-time planning policies with implicit dynamics models have been shown to work in discrete action spaces with Q learning. However, decision-time planning with implicit dynamics models in continuous action space has proven to be a difficult problem. Recent work in Reinforcement Learning has allowed for implicit model based approaches to be extended to Policy Gradient methods. In this work we propose Policy Tree Network (PTN). Policy Tree Network lies at the intersection of Model-Based Reinforcement Learning and Model-Free Reinforcement Learning. Policy Tree Network is a novel approach which, for the first time, demonstrates how to leverage an implicit model to perform decision-time planning with Policy Gradient methods in continuous action spaces. This work is empirically justified on 8 standard MuJoCo environments so that it can easily be compared with similar work done in this area. Additionally, we offer a lower bound on the worst case change in the mean of the policy when tree planning is used and theoretically justify our design choices.
The paper proposes a modification to Policy Prediction Networks (PPN) in which the learned transition-, reward- and value function models are used at test-time in a planning procedure. 
A second contribution is the "pi-Q-backup" which uses the geometric mean of both the policy and the value function as maximisation target of the planning step.
 Overall, I find the idea interesting and the experimental evaluations promising. However, I am voting for "weak reject" for the reasons outlined below. If some (or all) of them are address, I'd be happy to raise my score. 
- I found the paper hard to understand. In particular, the algorithm PPN on which this work is build is not explained at all, requiring the reader to read the original PPN paper. Including a description of PPN, including it's main features, would greatly help the paper. Second, I am still not sure I correctly undestand when each component is used. As I currently understand it, the main usage of the model during training time is to compute the Advantages in equation (2)? Or are those computed based on rollouts? If so, where is the model actually being used during training?
- Figure 1 is taken directly from the PPN paper without any reference or citation (as far as I can tell).
- For the comparison in Figure 5, it would be great if PPN could also be tested with the newly introduced parameter \beta_i. At the moment, it is hard to tell whether the performance gains are due to \beta or due to the proposed planning scheme.
- I'm confused about Theorem 1: Wouldn't we want an upper bound on the difference of means?  Also, what does 'worst-case' mean? Is that for the 'worst' Q-function we could choose? 
Edit:
Thank you for your comments and updated manuscript.
I think the writing has improved significantly, but could still be further improved and clarified. In particular, the question of how the model and other components at various points in time could be made more obvious. I found the authors' response to R3 here helpful as well. 
At least for me some of the confusion arises not due to the complexity of the proposed approach, but just because combining real and 'simulated' transitions can be used and mixed in so many different ways that it's important to be clear about it. Also, at least personally, I found the explanation "Learning is done with a model-based approach that follows the behaviour policy’s rollout trajectory. However, the test policy follows a model-based approach" still not very helpful. 
Overall, I think the presentation is on a good way but needs some more work.
With that being said and now having a better understanding of the algorithm I think this is very interesting work. However, I share R1's concerns about the computation of the \pi-Q backup, in particular that it seems arbitrary and doesn't handle negative values. I'm also not convinced that adding |min(Q)| is a good solutions as a) we don't always have access to that value and b) If I'm not wrong, than \pi_F is not invariant under a shift of Q. 
I'm wondering why the authors decided to take the geometric mean instead of following the more typically used approach of using exp(Q/Temperature)*pi to combine Q-function and a policy distribution (see e.g. the "Control as Inference" literature, the "Maximum a Posteriori Policy Optimisation" or "Soft Actor Critic" algorithms, in particular the "Soft Value functions". I think this should at the very least be an ablation study, and could be even performing better and at the very least be robust to negative values.
Overall, I think this is very interesting work and could become a very strong paper, but I will remain to recommend a "weak reject" because I think it needs some more work to get there.