We propose a reinforcement learning framework for discrete environments in which an agent makes both strategic and tactical decisions. The former manifests itself through the use of value function, while the latter is powered by a tree search planner. These tools complement each other. The planning module performs a local what-if analysis, which allows to avoid tactical pitfalls and boost backups of the value function. The value function, being global in nature, compensates for inherent locality of the planner. In order to further solidify this synergy, we introduce an exploration mechanism with two distinctive components: uncertainty modelling and risk measurement.
To model the uncertainty we use value function ensembles, and to reflect risk we use propose several functionals that summarize the implied by the ensemble. We show that our method performs well on hard exploration environments: Deep-sea, toy Montezuma's Revenge, and Sokoban. In all the cases, we obtain speed-up in learning and boost in performance.
The authors propose to combine planning methods like MCTS with an ensemble of value functions to a) estimate the value of leaf nodes of the search tree and b) use the ensemble estimate of uncertainty to guide exploration during MCTS search. 
The MCTS rollouts are also used as optimization targets for the value function.
I believe this is a clear reject. On the one hand, the paper needs signficiantly more work on the writing and clarity. On the other hand I have several worries on the method and evaluation side. 
Regarding the presentation of the paper:
Overall, the paper seems quite rushed. This is not a strong reason for rejection but should be improved in a future version. For example, punctuation and sentence structure is often wrong, the paper has only slighlty over 7 pages, a citation is undefined on p.7 and images and whitespace is formatted wrongly on occasion (e.g. top of page 6).
More importantly, on the content side, the experimental section is sufficiently clear and well written, however, the method description needs more detail and background information. The paper relies on several prior works which are referred to but not described (E.g. MCTS , the sampling mechanism by Osband et al. which they are using but not describing, the 'mask' from Osband et al which they are using but not describing).
Furthermore, the algorithm itself is not described in sufficient detail:
- How does the 'soft-penalization' work?
- How exactly does the mechanism "similar in fashing to" Thomson sampling work?
- Are you learning a model or do you have access to the true transition function?
Regarding the method:
I can't say anything definitive about the method as I'm not entirely clear how exactly it works. However, I have several worries that might need addressing:
- It seems to me that the method relies on access to the _true_ transition and reward function and not on a learned model. This is a big difference to much of the prior work they compare against. This also makes the comparison against any pure model free method like PPO much less meaningful.
- Similarly, manually avoiding dead-ends and loops is a very strong assumption 
- Also, being able to distinguish and use a fixed ratio of "solved" and "unsolved" episodes is a strong assumption. 
- The one main contribution seems to be a new way of how \phi_a(x) is defined. Their particular choice needs a clearer motivation. Furthermore, if there is more contribution and differences to prior work, highlighting them more would help the reader understand the contribution. 
- As the work makes several strong assumptions regarding the environment and access to the model, significantly more work (e.g. ablation studies) is needed to clearly show which assumption and feature of the algorithm is important for performance (and ideally also why). For example (but that's just a first idea): To understand the impact of their choice of \phi vs. their planning architecture, it would be be interesting to maybe train PPO using an exploration bonus based on \phi. This would allow disentangling the contribution of: Access to the true model, "discrete-environment-tricks" like penalizing dead-ends, and exploration incentivication of \phi. 
Edit:
Thank you for your response and the updated manuscript, which reads considerably better.
I also agree with your point regarding the strength of assumption regarding "solved" and "unsolved" episodes.
Consequently, I will raise my score to a "weak reject" to express that I think this is promising work.
I do believe that ablation studies would add a lot to the paper as they would allow one to see which of the (many) added components help how much, for example between the selection function $\phi$ and the various penalizations used.