Reinforcement Learning with Perturbed Rewards
Recent studies have shown the vulnerability of reinforcement learning (RL) models in noisy settings. The sources of noises differ across scenarios. For instance, in practice, the observed reward channel is often subject to noise (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible as a result. Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively.
 The paper aims at studying the setting of perturbed rewards in a deep RL setting. Studying the effect of noise in the reward function is interesting. The paper is quite well-written. However the paper studies a rather simple setting, the limitations could be discussed more clearly and there are one or two elements unclear (see below). The paper assumes first the interesting case where the generation of the perturbed reward is a function of S*R into the perturbed reward space. But then the confusion matrix does *not* take into account the state, which is justified by "to let our presentation stay focused (...)". I believe these elements should at least be clearly discussed. Indeed, in that setting, the theorems given seem to be variations of existing results and it is difficult to understand what is the message behind the theorems. In addition, it is assumed that the confusion matrix C is known or estimated from data but it*s not clear to me how this can be done in practice. In equation 4, how do you have access to the predicted true rewards? Additional comments: - The discount factor can be 0 but can not, in general, be equal to 1. So the equation in paragraph 2.1 "0 < γ ≤ 1" is wrong. - The paper mention that "an underwhelming amount of reinforcement learning studies have focused on the settings with perturbed and noisy rewards" but there are some works on the subject (e.g., https://arxiv.org/abs/1805.03359) and a discussion about the differences with the related work would be interesting.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Reinforcement Learning with Perturbed Rewards
Recent studies have shown the vulnerability of reinforcement learning (RL) models in noisy settings. The sources of noises differ across scenarios. For instance, in practice, the observed reward channel is often subject to noise (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible as a result. Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper aims at studying the setting of perturbed rewards in a deep RL setting. Studying the effect of noise in the reward function is interesting. The paper is quite well-written. However the paper studies a rather simple setting, the limitations could be discussed more clearly and there are one or two elements unclear (see below). The paper assumes first the interesting case where the generation of the perturbed reward is a function of S*R into the perturbed reward space. But then the confusion matrix does *not* take into account the state, which is justified by "to let our presentation stay focused (...)". I believe these elements should at least be clearly discussed. Indeed, in that setting, the theorems given seem to be variations of existing results and it is difficult to understand what is the message behind the theorems. In addition, it is assumed that the confusion matrix C is known or estimated from data but it*s not clear to me how this can be done in practice. In equation 4, how do you have access to the predicted true rewards? Additional comments: - The discount factor can be 0 but can not, in general, be equal to 1. So the equation in paragraph 2.1 "0 < γ ≤ 1" is wrong. - The paper mention that "an underwhelming amount of reinforcement learning studies have focused on the settings with perturbed and noisy rewards" but there are some works on the subject (e.g., https://arxiv.org/abs/1805.03359) and a discussion about the differences with the related work would be interesting.
assistant
 The paper aims at studying the setting of perturbed rewards in a deep RL setting. Studying the effect of noise in the reward function is interesting. The paper is quite well-written. However the paper studies a rather simple setting, the limitations could be discussed more clearly and there are one or two elements unclear (see below). The paper assumes first the interesting case where the generation of the perturbed reward is a function of S*R into the perturbed reward space. But then the confusion matrix does *not* take into account the state, which is justified by "to let our presentation stay focused (...)". I believe these elements should at least be clearly discussed. Indeed, in that setting, the theorems given seem to be variations of existing results and it is difficult to understand what is the message behind the theorems. In addition, it is assumed that the confusion matrix C is known or estimated from data but it*s not clear to me how this can be done in practice. In equation 4, how do you have access to the predicted true rewards? Additional comments: - The discount factor can be 0 but can not, in general, be equal to 1. So the equation in paragraph 2.1 "0 < γ ≤ 1" is wrong. - The paper mention that "an underwhelming amount of reinforcement learning studies have focused on the settings with perturbed and noisy rewards" but there are some works on the subject (e.g., https://arxiv.org/abs/1805.03359) and a discussion about the differences with the related work would be interesting.