Model-based reinforcement learning (MBRL) has shown its advantages in sampleefficiency over model-free reinforcement learning (MFRL). Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. In this paper, we propose a simple and elegant model-based reinforcement learning algorithm called soft stochastic value gradient method (S2VG). S2VG combines the merits of the maximum-entropy reinforcement learning and MBRL, and exploits both real and imaginary data. In particular, we embed the model in the policy training and learn Q and V functions from the real (or imaginary) data set. Such embedding enables us to compute an analytic policy gradient through the back-propagation rather than the likelihood-ratio estimation, which can reduce the variance of the gradient estimation. We name our algorithm Soft Stochastic Value Gradient method to indicate its connection with the well-known stochastic value gradient method in .
This paper presents a slightly new model-based reinforcement learning algorithm, claiming that the new method is elegant and combines the merits of earlier methods. In particular, it claims that the new method is more sample efficient and computationally efficient than previous methods.The novelty of the new method has to do with how the model participates in “the policy improvement step”. The paper does not achieve a high level of quality in its presentation or empirical results, as i discuss further below, and as a result I do not consider it to make a scientific contribution, and I recommend rejection.
The presentation of the method is not clear; it uses an informal notation and leaves out many steps; in many cases the statements are not formally correct. I believe that many of these weaknesses are not introduced in this work but are present also in the works that it builds on. It is possible that someone intimately familiar with the previous works would be able to fully understand this method, but perhaps not, and certainly it does not stand alone. 
The new method is presented in Section 3. Equation (3) defines an objective for the policy parameter. It uses things called r-hat, \rho-hat, p, and f. It is not clear whether these things are functions, random variables, or distributions. What is the domain of f? Is r(s,a) a random variable? Is r(s1,a1) different from r(s2,a2) if s1=s2 and a1=a2? Do they have the same distribution? This is just scratching the surface. Many, many things are not well defined, and some critical things, like V, appear to be defined multiple times.
The new method is ultimately presented in Algorithm 1, but it is just a sketch, with essential things left out. What is an “iteration”? Where are the “epochs” discussed in the empirical results? Many steps ask us to calculate gradients with respect to objective functions given by equations in the text. Almost all of these are unclear. To literally calculate the gradient of the expected values would involve a full sweep over the replay buffer, which appears never to be emptied, and thus would grow without bound. It this really what is meant. 
There are also many problems with the empirical results. To begin with, there is no discussion of how the many hyper-parameters were set. And the statistics are not present.  The results present means and standard deviations. The paper does not appear to say how many runs were done, but from the wide variation in the standard deviations, it would appear that there were not many. The very basics of a valid experiment are thus missing. We really should not conclude anything from such results, but if they are published, then many readers would conclude things. Our field should not go any further this way, in my opinion.