Learning Key Steps to Attack Deep Reinforcement Learning Agents
Deep reinforcement learning agents are known to be vulnerable to adversarial attacks. In particular, recent studies have shown that attacking a few key steps is effective for decreasing the agent*s cumulative reward. However, all existing attacking methods find those key steps with human-designed heuristics, and it is not clear how more effective key steps can be identified. This paper introduces a novel reinforcement learning framework that learns more effective key steps through interacting with the agent. The proposed framework does not require any human heuristics nor knowledge, and can be flexibly coupled with any white-box or black-box adversarial attack scenarios. Experiments on benchmark Atari games across different scenarios demonstrate that the proposed framework is superior to existing methods for identifying more effective key steps.
Learning Key Steps to Attack Deep Reinforcement Learning Agents This paper proposes an extension of existing discrete action image space adversarial attack algorithms. Instead of choosing the steps by heuristic, the authors propose to choose the key steps by augmenting the reward function with a penalty to decrease the ratio of attacks. I tend to vote rejection for this paper, given that the proposed algorithms seem incremental compared to the existing algorithms, and the experiments seem not sufficient enough to support the core claim proposed in the paper. Pros: - The paper is well written, with sufficient background and related work section for the paper to be self-contained. - The proposed framework is an interesting and practical framework for attacking RL agents. Cons: - The experiment section is insufficient. More specifically: 1) Results from Figure 5 and Figure 6 seem to disagree with what the authors claim in the paper. In many (the majority) of the environments, the proposed algorithm has only trivial improvement and even worse performance under the same attack rate. 2) It is not very convincing when only one result sample is plotted in Figure 5 and Figure 6. I think it is necessary to show the performance of the proposed algorithm under different attack rate. A wide range of candidate penalty parameter lambda should be tested, so that a curve can be fitted for the proposed algorithm similar to the baselines (similar to the one shown in Table 1, but with much more test values). 3) Related to 2), it seems the Lagrange relaxation makes it hard to control the attack rate in the proposed algorithms. How sensitive it is to control the attack rate? 3) Can the authors elaborate on why the algorithms is not too sensitive to the value of penalty in section 4.5? Table 1, where the performance is almost the same for different penalty parameter, does not necessarily show that the algorithms is not too sensitive to the choice of the penalty parameter. As mentioned by the author, -21 is the minimum reward (or random reward) an agent can get from Pong. In general, given the current status of the paper, where there is a lot of room for improvement of experiment section, I will vote for a rejection.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning Key Steps to Attack Deep Reinforcement Learning Agents
Deep reinforcement learning agents are known to be vulnerable to adversarial attacks. In particular, recent studies have shown that attacking a few key steps is effective for decreasing the agent*s cumulative reward. However, all existing attacking methods find those key steps with human-designed heuristics, and it is not clear how more effective key steps can be identified. This paper introduces a novel reinforcement learning framework that learns more effective key steps through interacting with the agent. The proposed framework does not require any human heuristics nor knowledge, and can be flexibly coupled with any white-box or black-box adversarial attack scenarios. Experiments on benchmark Atari games across different scenarios demonstrate that the proposed framework is superior to existing methods for identifying more effective key steps.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Learning Key Steps to Attack Deep Reinforcement Learning Agents This paper proposes an extension of existing discrete action image space adversarial attack algorithms. Instead of choosing the steps by heuristic, the authors propose to choose the key steps by augmenting the reward function with a penalty to decrease the ratio of attacks. I tend to vote rejection for this paper, given that the proposed algorithms seem incremental compared to the existing algorithms, and the experiments seem not sufficient enough to support the core claim proposed in the paper. Pros: - The paper is well written, with sufficient background and related work section for the paper to be self-contained. - The proposed framework is an interesting and practical framework for attacking RL agents. Cons: - The experiment section is insufficient. More specifically: 1) Results from Figure 5 and Figure 6 seem to disagree with what the authors claim in the paper. In many (the majority) of the environments, the proposed algorithm has only trivial improvement and even worse performance under the same attack rate. 2) It is not very convincing when only one result sample is plotted in Figure 5 and Figure 6. I think it is necessary to show the performance of the proposed algorithm under different attack rate. A wide range of candidate penalty parameter lambda should be tested, so that a curve can be fitted for the proposed algorithm similar to the baselines (similar to the one shown in Table 1, but with much more test values). 3) Related to 2), it seems the Lagrange relaxation makes it hard to control the attack rate in the proposed algorithms. How sensitive it is to control the attack rate? 3) Can the authors elaborate on why the algorithms is not too sensitive to the value of penalty in section 4.5? Table 1, where the performance is almost the same for different penalty parameter, does not necessarily show that the algorithms is not too sensitive to the choice of the penalty parameter. As mentioned by the author, -21 is the minimum reward (or random reward) an agent can get from Pong. In general, given the current status of the paper, where there is a lot of room for improvement of experiment section, I will vote for a rejection.
assistant
Learning Key Steps to Attack Deep Reinforcement Learning Agents This paper proposes an extension of existing discrete action image space adversarial attack algorithms. Instead of choosing the steps by heuristic, the authors propose to choose the key steps by augmenting the reward function with a penalty to decrease the ratio of attacks. I tend to vote rejection for this paper, given that the proposed algorithms seem incremental compared to the existing algorithms, and the experiments seem not sufficient enough to support the core claim proposed in the paper. Pros: - The paper is well written, with sufficient background and related work section for the paper to be self-contained. - The proposed framework is an interesting and practical framework for attacking RL agents. Cons: - The experiment section is insufficient. More specifically: 1) Results from Figure 5 and Figure 6 seem to disagree with what the authors claim in the paper. In many (the majority) of the environments, the proposed algorithm has only trivial improvement and even worse performance under the same attack rate. 2) It is not very convincing when only one result sample is plotted in Figure 5 and Figure 6. I think it is necessary to show the performance of the proposed algorithm under different attack rate. A wide range of candidate penalty parameter lambda should be tested, so that a curve can be fitted for the proposed algorithm similar to the baselines (similar to the one shown in Table 1, but with much more test values). 3) Related to 2), it seems the Lagrange relaxation makes it hard to control the attack rate in the proposed algorithms. How sensitive it is to control the attack rate? 3) Can the authors elaborate on why the algorithms is not too sensitive to the value of penalty in section 4.5? Table 1, where the performance is almost the same for different penalty parameter, does not necessarily show that the algorithms is not too sensitive to the choice of the penalty parameter. As mentioned by the author, -21 is the minimum reward (or random reward) an agent can get from Pong. In general, given the current status of the paper, where there is a lot of room for improvement of experiment section, I will vote for a rejection.