Recent studies have shown that reinforcement learning (RL) models can be vulnerable in various scenarios, where noises from different sources could appear. For instance, the observed reward channel is often subject to noise in practice (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible. Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively.
This paper investigates reinforcement learning with a perturbed reward signal. In particular, the paper proposes a particular model for adding noise to the reward function via a confusion matrix, which offers a nuanced notion of reward-noise that is not too complicated so-as to make learning impossible. I take this learning setting to be both novel and interesting for opening up areas for future work. The central contributions of the work are to 1) leverage a simple estimator to prove the convergence of Q-Learning under the reward-perturbed setting along with the sample-complexity of a variant of (Phased) Q-Learning which they call "Phrased" Q-Learning, and 2) An algorithmic scheme for learning in the reward-perturbed setting (Algorithm 1), and 3) An expansive set of experiments that explore the impact of various reward models on learning across different environment-algorithm combinations. The sample complexity term extends Phased Q-Learning to incorporate aspects of the reward confusion matrix, and to my knowledge is novel. Further, even though Theorem 1 is unsurprising (as the paper suggests), I take the collection of Theorem 1, 2, and 3 to be collectively novel.
Indeed, the paper focuses on an interesting and relatively unexplored direction for RL. Apart from the work cited by the paper (and perhaps work like Krueger et al. (2016), in which agents must pay some cost to observe true rewards), there is little work on learning settings of this kind. This paper represents a first step in gaining clarity on how to formalize and study this problem. I did, however, find the analysis and the experiments to be relatively disjointed -- the main sample complexity result presented by the paper (Theorem 2) was given for Phased Q-Learning, yet no experiments actually evaluate the performance of Phased Q-Learning. I think the paper could benefit from experiments focused on simple domains that showcase how traditional algorithms do in cases where it is easier to understand (and visualize) the impact of the reward perturbations (simple chain MDPs, grid worlds, etc.) -- and specifically experiments including Phased Q-Learning. 
Pros:
	- General, interesting new learning setting to study.
	- Initial convergence and sample complexity results for this new setting.
	- Depth and breadth of experimentation (in terms of diversity of algorithms and environments), includes lots of detail about the experimental setup.
Cons:
	- Clarity of writing: lots of typos and bits of math that could be more clear (see detailed comments below) [Fixed]
	- The plots in Section 4 are all extremely jagged. More trials seem to be required. Moreover, I do think simpler domains might help offer insights into the reward perturbed setting. [Fixed]
	- The reward perturbation model is relatively simple.
Some high level questions/comments:
	- Why was Phrased Q-Learning not experimented with?
	- Why use majority voting as the rule? When this was introduced it sounded like any rule might be used. Have you tried/thought about others?
	- Your citation to Kakade's thesis needs fixing; it should read:
		"Kakade, Sham Machandranath. On the sample complexity of reinforcement learning. Ph.D Thesis. University of London, 2003."
		(right now it is cited as "(Gatsby 2003)" throughout the paper)
	- You might consider picking a new name for Phrased Q-Learning -- right now the name is too similar to Phased Q-Learning from [Kearns and Singh NIPS 1999].
        - As mentioned in the "cons" section, the confusion matrix is still a somewhat simple model of reward noise. I was left wondering: what might be the next most complicated form of adding reward noise? How might the proposed algorithm(s) respond to this slightly more complex model? That is, it's unclear how general the results are, or if they are honed too tightly to the specific proposed reward noise model. I was hoping the authors could respond to this point.
	
Section 0) Abstract:
	- Not immediately clear what is meant by "vulnerability" or "noisy settings". Might be better to pick a more clear initial sentence (same can be said of the "sources of noise..."")
Section 1) Introduction:
	- "adversaries in real-world" --> "adversaries in the real-world"
	- You might consider citing Loftin et al. (2014) regarding the bulleted point about "Application-Specific Noise".
	- "unbiased reward estimator aided reward robust reinforcement learning framework" --> this was a bit hard to parse. Consider making more concise, like: "unbiased reward estimator for use in reinforcement learning with perturbed rewards".
	- "Our solution framework builds on existing reinforcement learning algorithms, including the recently developed DRL ones" --> cite these up front So, cite: Q-Learning, CEM, SARSA, DQN, Dueling DQN, DDPG, NAF, and PPO, and spell out the acronym for each the first time you introduce them.
	- "layer of explorations" --> "layer of exploration"
Section 2) Problem Formulation
	- "as each shot of our" --> what is 'shot' in this context?
	- "In what follow," --> "In what follows,"
	- "where 0 < \gamma \leq 1" --> Usually, $\gamma \in [0,1)$, or $[0,1]$. Why can't $\gamma = 0$?
	- The transition notation changes between $\mathbb{P}_a(s_{t+1} | s_t)$ and $\mathbb{P}(s_{t+1} | s_t, a_t)$. I'd suggest picking one and sticking with it to improve clarity.
	- "to learn a state-action value function, for example the Q-function" --> Why is the Q-function just an example? Isn't is *the* state-action value function? That is, I'd suggest replacing "to learn a state-action value function, for example the Q-function" with "to learn a state-action value function, also called the Q-function"
	- "Q-function calculates" --> "The Q-function denotes"
	- "the reward feedbacks perfectly" --> "the reward feedback perfectly"
	- I prefer that the exposition of the perturbed reward MDP be done with C in the tuple. So: $\tilde{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{R}, C, \mathcal{P}, \gamma \rangle$. This seems the most appropriate definition, since the observed rewards will be generated by $C$.
	- The setup of the confusion matrix for reward noise over is very clean. It might be worth pointing out that $C$ need not be Markovian. There are cases where C is not just a function of $\mathcal{S}$ and $\mathcal{R}$, like the adversarial case you describe early on.
Section 3) Learning w/ Perturbed Rewards
	- Theorem 1 builds straightforwardly on Q-Learning convergence guarantee (it might be worth phrasing the result in those terms? That is: the addition of the perturbed reward does not destroy the convergence guarantees of Q-Learning.)
	- "we firstly" --> "we first"
	- "value iteration (using Q function)" --> "value iteration"
	- "Definition 2. Phased Q-Learning" --> "Definition 2. Phrased Q-Learning". I think? Unless you're talking about Phased Q from the Kearns and Singh '99 work.
	- "It uses collected m samples" --> "It uses the collected m samples"
	- Theorem 2: it would be helpful to define $T$ since it appears in the sample complexity term. Also, I would suggest specifying the domain of $\epsilon$, as you do with $\delta$.
	- "convergence to optimal policy" --> "convergence to the optimal policy"
	- "The idea of constructing MDP is similar to" --> this seems out of place. The idea of constructing which MDP? Similar to Kakade (2003) in what sense?
	- "the unbiasedness" --> "the use of unbiased estimators"
	- "number of state-action pair, which satisfies" --> "number of state-action pairs that satisfy"
	- "The above procedure continues with more observations arriving." --> "The above procedure continues indefinitely as more observation arrives." Also, which procedure? Updating $\tilde{c}_{i,j}$? If so, I would specify.
	- "is nothing different from Eqn. (2) but with replacing a known reward confusion" --> "replaces a known reward confusion"
4) Experiments:
	- Diverse experiments! That's great. Lots of algorithms, lots of environment types.
	- I expected to see Phrased Q-Learning in the experiments. Why was it not included?
	- The plots are pretty jagged, so I'm left feeling a bit skeptical about some of the results. The results would be strengthened if the experiments were repeated for more trials.
5) Conclusion:
	- "despite of the fact" --> "despite the fact"
	- "finite sample complexity of Q-Learning with estimated surrogate rewards are given" --> It's not really Q-Learning, though. It's a variant of Q-Learning. I'd suggest being explicit about that.
Appendix:
	- "It is easy to validate the unbiasedness of proposed estimator directly." --> "It is easy to verify that the proposed estimator is unbiased directly."
	- "For the simplicity of notations" --> "For simplicity"
	- "the Phrased Q-Learning could converge to near optimal policy" --> ""the algorithm Phrased Q-Learning can converge to the near optimal policy""
	- "Using union bound" --> "Using a union bound"
	- Same comment regarding $\gamma$: it's typically $0 \leq \gamma < 1$.
	- Bottom of page 16, the second equation from the bottom, far right term: $c.j$ --> $c,j$.
	- "Using CauchySchwarz Inequality" --> "Using the Cauchy-Schwarz Inequality"
References:
	Loftin, Robert, et al. "Learning something from nothing: Leveraging implicit human feedback strategies." Robot and Human Interactive Communication, 2014 RO-MAN: The 23rd IEEE International Symposium on. IEEE, 2014.
	Krueger, D., Leike, J., Evans, O., & Salvatier, J. (2016). Active reinforcement learning: Observing rewards at a cost. In Future of Interactive Learning Machines, NIPS Workshop.