Data-Efficient Reinforcement Learning in Continuous State-Action Gaussian-POMDPs
We present a data-efficient reinforcement learning method for continuous stateaction systems under significant observation noise. Data-efficient solutions under small noise exist, such as PILCO which learns the cartpole swing-up task in 30s. PILCO evaluates policies by planning state-trajectories using a dynamics model. However, PILCO applies policies to the observed state, therefore planning in observation space. We extend PILCO with filtering to instead plan in belief space, consistent with partially observable Markov decisions process (POMDP) planning. This enables data-efficient learning under significant observation noise, outperforming more naive methods such as post-hoc application of a filter to policies optimised by the original (unfiltered) PILCO algorithm. We test our method on the cartpole swing-up task, which involves nonlinear dynamics and requires nonlinear control.
SUMMARY
In this paper, the authors propose to learn and plan with a model that represents uncertainty in a specific case of partial observability - that of noisy observations. Their method the analogue of a known model based method - PILCO - for this noisy observation case. It handles the noise by assuming a joint Gaussian distribution over the latent states and observations at each time step when making predictions (at execution time, observations are known and a filtered latent state is used). On the kind of dynamical system that these method are commonly evaluated on - the cart-pole swing-up in this case - the proposed methods seems to be better than relevant baselines at dealing with noise. 
TECHNICAL QUALITY
The proposed algorithm is supported by a derivation of the update equation and experimental validation on the cart-pole swing-up task. The derivation seems technically solid. I just wondered why in (5) the covariance between M_t|t-1 and Z_t is 0, since in Fig. 2 it seems that Z_t is a noisy version of B_t|t-1 of which M_t|t-1 is the mean. 
Although the experiments only consider a single system, the results seem quite convincing. The empirical loss per episode is significantly lower, and predictions of the cost per time step are much better than for the compared methods. I just wondered what is meant by the "empirical loss distribution", and whether the 100 independent runs that the error bars are based on are 100 evaluations of the same learned controller; or 100 independently trained controllers. In Figs. 7 and 8 I would be interested to see the n=0 case (or, if that causes numerical problems, n=0.01 or similar). 
In terms of related work, the authors mainly restrict themselves to discussing Bayesian model learning and Bayesian RL techniques. It might be interesting to also discuss other styles of policy search methods for the partially observable setting. 
It might be insightful to describe what about the current method limits us to the noisy observation case. If we observed positions but not velocities, coulnd't we filter that, too?
NOVELTY
I haven't seen earlier extensions of PILCO-type methods to the partially observable case. So I would consider the method quite novel.
SIGNIFICANCE AND RELEVANCE
Noisy dynamical systems occur in many (potential) application domain of reinforcement learning (robotics, smart grid, autonomous driving), so, this is a relevant research topic for the RL community. On the investigated system, the improvement relative to the state of the art seems quite significant, however, the method is only evaluated on a single system. 
CLARITY
The paper is mostly well-written. However, in some part the clarity could be improved. For example, the paper does not mention explicitly they assume the reward function to be known. The policy improvement step (8 in the algorithm) is not explained at all. For readers already familiar with PILCO this is quite obvious but it would be good to give a short introduction from the others. 
The technical section in 4.1 - 4.2 is very dense. I understand space is rather limited but it could be good to explain this part in more detail in the appendix. For example, the paper states V=E[V'] is assumed fixed, but doesn't state how this fixed value is computed or set.
Figures 1 and 2 are presented as analogues, yet the latent state X is not shown in the right image. It would be easier to understand the relationship if the underlying latent process was shown in both images. 
There are quite many grammar issues. I'll point some out under minor comments. 
MINOR COMMENTS
Grammar - please rephrase sentences in lines 7, 52, 86-87, 134,209,250, 283, 299-300
Contractions and abbreviations - please write out (doesn't, etc.) 
Some sentence start with note but don't seem grammatical. I suggest writing "Note that, ..." or similar.
Line 55: Greater -> Larger
Line 65: Seems like the author name shouldn't appear here
Line 99: Less -> fewer
line 112: "Overfitting leads to bias". Overcapacity generally causes a high variance, not necessarily bias. Or is the bias here a result of optimizing the policy on the erroneous model? (similar in line 256)
line 118: I guess k(x_i, x_j) should have a superscript a.
line 118: are the phi_a additional hyperparameters? How are they set?
line 147, 148 "detailed section" -> detailed in Section"
line 162: it's not clear to me what you mean by "hierarchically random"
line 174: it seems you switched x and z in p(x)=N(z, Sigma). 
line 177: Gaussian conditioning is not a product.
line 181: "the distribution f(b)". This seems a bit odd. Maybe assign f(b) to a random variable?
line 189: not sure what you mean by "in principle".
- you might want to introduce notation \mathbb{C} and \mathbb{V}
line 211: "a mixture of basis functions" -> do you mean it is a linear combination of basis functions?
line 245: "physically" -> "to physically"
line 273  "collect" -> to collect
line 274: trails -> trials
line 283: this seems a strong claim, you might want to rephrase (fPILCO doesn't predict how well other methods will do as is implied here )
line 290 - you might want to choose another symbol for noise factor than n (often number of data points).
297 : data - > data points ?
315: Give greater performance than otherwise -> improved performance with respect to the baselines ?
References: I would consistently use either full names or initials. Some references like 11 are quite messy (mentioning the year 3 times in this case). The words Gaussian and POMDPs in paper titles should be capitalized correctly.