Reinforcement Learning under Model Mismatch
We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs of [1,15,11] to the model-free Reinforcement Learning setting, where we do not have access to the model parameters, but can only sample states from it. We define robust versions of Q-learning, SARSA, and TD-learning and prove convergence to an approximately optimal robust policy and approximate value function respectively. We scale up the robust algorithms to large MDPs via function approximation and prove convergence under two different settings. We prove convergence of robust approximate policy iteration and robust approximate value iteration for linear architectures (under mild assumptions). We also define a robust loss function, the mean squared robust projected Bellman error and give stochastic gradient descent algorithms that are guaranteed to converge to a local minimum.
Overview:
The authors propose an extension of Robust RL methods to model-free applications. Common algorithms (SARSA, Q-learning, TD(lambda)) are extended to their robust, model-free version.
The central point of the paper is replacing the true and unknown transition probabilities  P_i^a with a known confidence region U_i^a centered on an unknown probability p_i^a. Transitions according to p_i^a are sampled via a simulator. The robustness of the algorithms derive then from an additional optimization step in U_i^a. The authors clarify that, since the addition of term U to p must not violate the probability simplex. However, since p_i^a is unknown, the additional optimization is actually performed on a term U_hat that ignores the above restriction. This mismatch between U and U_hat, represented by a term beta, determines whether the algorithms converge towards to an epsilon-optimal policy. The value of epsilon is also function of beta. 
The authors propose also an extension of RL with function approximation architectures, both linear and nonlinear. In the linear case, the authors show that a specific operator T is a contraction mapping under an opportune weighted Euclidean norm. By adopting the steady state distribution of the exploration policy as weights, and by adopting an assumption from previous literature concerning a constant alfa < 1, the authors prove that, under conditions on alfa and beta, the robust operator T_hat is a contraction, and that the solution converges close to the true solution v_pi, where the closeness depends again on alfa and beta. 
Furthermore, the authors extend the results of Ref.6 to the robust case for smooth nonlinear approximation architectures. An additional assumption here is to assume a constant confidence bound U_i^a = U for all state-action pairs for the sake of runtime. The authors extend on the work of Ref.6 by introducing the robust extension of the mean squared projected bellman equation, the MSRPBE, and of the GTD2 and TDC update algorithms, which converge to a local optima given assumptions on step lengths and on confidence region U_hat. 
In conclusion, the authors apply both nominal and robust Q-learning, SARSA and TD-learning to several OpenAI gym experiments. In these experiments, a small probability of transitioning to a random state is added to the original models during training. The results show that the best robust Q-learning outperforms nominal Q-learning. However, it is shown that the performance of the robust algorithms is not monotone with p: after increasing for small values of p, the advantage of the robust implementation decreases as U and U_hat increase in discrepancy.
Evaluation:
- Quality: the paper appears to be mathematically sound and assumptions are clearly stated. However, the experimental results are briefly commented. Figures are partial: of the three methods compared, Q-learning, SARSA and TD-learning, only results from Q-learning are briefly presented. The algorithms of section 4 (with function approximation) are not accompanied by experimental results, but due to the theoretical analysis preceding them this is of relative importance. 
- Clarity: the paper is easy to read, with the exception of a few symbolic mismatches that are easy to identify and rectify. The structure of the paper is acceptable. However, the overall clarity of the paper would benefit from extending section 4.2 and 5, possibly by reducing some portions of section 3. 
-  Originality: the model-free extensions proposed are an original contribution with respect to existing literature and to NIPS in particular.
- Significance: the motivation of the paper is clear, as is the contribution to the existing theoretical research. Less clear is the kind of application this work is intended for. In particular, the paper assumes the existence of a simulator with unknown transition probabilities, but known confidence bounds to the actual model being simulated, to obtain samples from. While this assumption is not by itself wrong, it is not clearly mentioned in the abstraction or in the introduction and takes the risk of appearing as artificial. 
Other comments:
- Line 194: the definition of beta given in this line is the one Lemma 4.2 rather than the one of Lemma 3.2;
- Line 207: here and in the following, nu is used instead of theta to indicate the discount factor. Please make sure the symbology is consistent;
- Line 268: the sentence "the state are sampled with from an" might contain a typo;
- Eq. 7-8: there appears to be an additional minus signs in both equations.