Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

Digital Object Identifier 10.1109/ACCESS.2017.DOI

## On the Perturbed States for Transformed Input-robust Reinforcement Learning

Tung M. Luu , Haeyong Kang , Tri Ton, Thanh Nguyen, and Chang D. Yoo ∗ ∗

1 School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

- ∗ Equal contribution, Corresponding author: Chang D. Yoo (e-mail: cd\_yoo@kaist.ac.kr).

ABSTRACT Reinforcement Learning (RL) agents demonstrating proficiency in a training environment exhibit vulnerability to adversarial perturbations in input observations during deployment. This underscores the importance of building a robust agent before its real-world deployment. To alleviate the challenging point, prior works focus on developing robust training-based procedures, encompassing efforts to fortify the deep neural network component's robustness or subject the agent to adversarial training against potent attacks. In this work, we propose a novel method referred to as Transformed Input-robust RL (TIRL) , which explores another avenue to mitigate the impact of adversaries by employing input transformationbased defenses. Specifically, we introduce two principles for applying transformation-based defenses in learning robust RL agents: (1) autoencoder-styled denoising to reconstruct the original state and (2) bounded transformations (bit-depth reduction and vector quantization (VQ)) to achieve close transformed inputs. The transformations are applied to the state before feeding it into the policy network. Extensive experiments on multiple MuJoCo environments demonstrate that input transformation-based defenses, i.e. , VQ, defend against several adversaries in the state observations. The official code is available at https://github.com/tunglm2203/tirl

INDEX TERMS Transformed Input-robust Reinforcement Learning (TIRL), Bounded Transformation, Adversarial Attack, Autoencoder-styled Denoising, Bit-Depth Reduction, Vector Quantization (VQ).

## I. INTRODUCTION

Modern deep reinforcement learning (RL) agents [1]-[8] typically rely on deep neural networks (DNNs) as powerful function approximations. However, it has been discovered that even a well-trained RL agent can experience significant failures due to small adversarial perturbations in its input during deployment [9]-[13]. This vulnerability raises concerns about deploying such agents in safety-critical applications, such as autonomous driving [14]-[20]. Therefore, the development of techniques to help RL agents withstand adversarial attacks on state observations becomes crucial before their deployment in the real world [9], [21]-[26].

Several approaches have been suggested in the literature to safeguard against adversarial attacks on an agent's state observations. One avenue of research concentrates on improving the robustness of the agent's DNNs component, such as the policy network or Q-value network, by promoting properties like invariance and smoothness through regularization techniques [21], [27]-[29]. These regularizers yield policy outputs that generate similar actions under bounded input perturbations. However, these defenses do not account for the dynamics of the RL environment and can be vulnerable to stronger attacks [22]. An alternative approach involves adversarial training of the RL agent, in which an adversary is introduced to perturb the agent's input as it takes actions within the environment. The episodes collected under these adversarial conditions are then utilized for training, leading to a more robust RL agent. Perturbations in this context can be induced from the policy or value function [11], [13], [19], [30], or more recently, from another RL-based adversary [22], [31]. RL-based adversaries are typically acquired through online learning. Although training with RL-based adversaries can yield robust agents, it often requires additional samples and computations due to the training of the additional agent involved. Furthermore, online attacks may lead to several unsafe behaviors, posing a potential risk to the control system if adversarial training occurs in a physical environment rather than a simulated one.

The approaches above can be seen as robust training-based defenses, wherein the RL agent undergoes optimization to concurrently fulfill the requirements of an RL task and meet the robust performance criteria. Meanwhile, within the realm

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-2457ef5e-c0c0-4564-ab84-da255aa63903.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a66c4601-5084-45ea-bf2f-af674c26da6c.jpg)

of image classification, an alternative strategy for countering adversarial attacks is recognized as input transformationbased defenses [32]-[42]. These defenses aim to eliminate adversarial perturbations from the input by applying transformations before presenting it to the classifier. The transformational process commonly relies on denoising techniques to purify perturbations [33], [35], [39]-[43] or employs image preprocessing techniques to mitigate the impact of adversaries [32], [34], [36]-[38]. Moreover, as this strategy exclusively modifies the input and not the model itself, input transformation-based defenses could benefit RL agents without substantially altering the underlying RL algorithms, serving as a plug-and-play module. Despite their potential to mitigate adversarial attacks, many of these transformations are tailored to image data [32], [36]-[38] and may not easily extend to vector inputs such as low-dimensional states in continuous control systems. This paper explores a new input transformation-based defensive approach, referred to as T ransformed I nput-robust RL TIRL ( ) , against adversarial attacks on state observations by applying input transformations suitable for low-dimensional states in continuous control tasks. To validate the effectiveness of the input transformations, various techniques are investigated, including bitdepth reduction, vector quantization, and autoencoder-style denoisers in continuous control tasks. Our findings indicate that these novel defenses exhibit notable resilience against adversarial attacks. Our contributions are summarized as follows:

- · We propose a new method, Transformed Input-robust RL (TIRL) with two principles, i.e. , (1) Bounded Transformation and (2) Autoencoder-styled Denoising , to use input transformations to counter adversarial attacks on state observations in deep RL.
- · Based on the two proposed principles, we explore the best transformation, i.e. , vector quantization (VQ), which is beneficial for the RL agent in continuous control tasks for the first time.
- · Extensive experiments on continuous control tasks from MuJoCo environments demonstrates that these defenses can be surprisingly effective against existing attacks.

## II. RELATED WORK

Adversarial perturbed states. Since the discovery of adversarial examples in image classification [44], corresponding vulnerabilities in deep reinforcement learning (RL) were first demonstrated in [9], [10]. Huang et al. [9] evaluated the robustness of Deep Q-Network (DQN) agents by employing a weak Fast Gradient Sign Method (FGSM) attacker [45] to initiate attacks at every step. Lin et al. [10] focused on attacking specific steps within trajectories, using a planner to devise perturbations that steer the agent toward a target state. Pattanaik et al. [13] introduced a more potent attack strategy that leverages both the policy and the Q function, departing from perturbations based solely on the policy. Zhang et al. [21] later introduced the state-adversarial Markov decision process to formally define adversarial attacks on state obser- vations, showing that optimal attacks can be learned as an RL problem. Building upon this foundation, Zhang et al. [22] and Sun et al. [31] introduced RL-based adversaries for black-box and white-box attacks, respectively. Although these attackers appear powerful, their effectiveness often relies on specific assumptions, such as requiring knowledge of the victim's policy [31]. Consequently, the introduced defenses [21], [31], [46] might be vulnerable to a more sophisticated, yet unknown, attacker. In contrast, we focus on developing defense strategies that are agnostic to the attacker's characteristics.

Adversarial training. The robustness of RL agents can be improved by training with adversarial samples [9], [11], [12], [47]. In the Atari domain, Kos and Song [11] and Behzadan et al. [12] used weak attacks to generate perturbations for training DQN agents, achieving limited improvement. Alternatively, regularization-based methods have emerged to enhance the robustness of deep neural network components in RL algorithms. Zhang et al. [21] introduced a hinge loss regularizer to control the smoothness of the Q function under bounded perturbations, while Russo and Proutiere [48] proposed Lipschitz regularization. In continuous control tasks, Huang et al. [9] generated attacks using both the policy and Q function, then used the resulting trajectories for training. However, recent work [21] shows this approach is unreliable for enhancing robustness against new attacks. Smoothness regularization has also been proposed to strengthen policy model robustness in both online [21] and offline RL [29]. Based on the theory of optimal attacks from [21], Zhang et al. [22] introduced a training paradigm that alternates between learning the RL agent and a black-box RL-based adversary to develop robustness. Similarly, Sun et al. [31] applied this approach with a white-box RL-based adversary, yielding a more robust RL agent. Training robust RL agents with additional RL-based adversaries requires extra samples and modifications to the underlying algorithms, making the process more complex and sample-intensive [22], [31]. Our approach, however, focuses on modifying the agent's input rather than altering the algorithm. This simplifies training and offers greater flexibility when incorporating other RL algorithms, providing a more practical and effective solution for developing robust RL agents.

Input Transformation-based Defenses. Studies have used input transformations to mitigate adversarial attacks in image classification due to their simplicity and flexibility [33][43]. Traditional image processing techniques such as image cropping [36] and image rescaling [34] have been used to reduce the effectiveness of adversarial attacks. Powerful denoising models have also been employed to purify input perturbations. For example, Meng and Chen [33] used an autoencoder-style denoiser to reconstruct denoised images from randomly perturbed ones, while Samangouei et al. used GANs [49] for image reconstruction. Nie et al. [43] applied diffusion models [50] to refine adversarial examples before feeding them to classifiers. However, many of these transformations are specifically designed for image data [32], [36]-[38] and may not easily adapt to vector inputs, such

as low-dimensional states in continuous control systems. To alleviate those issues, this paper investigates a novel input transformation-based defense that mitigates the impact of adversarial attacks on state observations for RL agents.

## III. PRELIMINARIES

This section presents the general reinforcement learning framework and an RL algorithm, Soft Actor-Critic (SAC), before introducing our novel input-transformation-based defenses. To evaluate the effectiveness of our method with SAC, we also describe test-time adversarial attacks.

## A. REINFORCEMENT LEARNING

A reinforcement learning (RL) environment is modeled by a Markov decision process (MDP), defined as a tuple of ( S A , , p, r, γ ) , where S is the state space, A is the action space, p = Pr s ( t +1 | s , a t t ) is the transition probability distribution which is usually unknown, r : S×A×S → R is the reward function, and γ ∈ [0 , 1) is a discount factor. An agent executes actions based on a policy π : S → A . In this work, we use the terms state and state observation to indicate the underlying state of the environment and the state perceived by the agent, respectively. If there are no attacks, both of them are identical. The target of RL agent is to maximize the expected discounted return E π,p [ ∑ ∞ t =0 γ r s , a , s t ( t t t +1 )] , which is the expected cumulative sum of rewards when following the policy π in the MDP. The state value function can measure this objective as follows:

$$V ^ { \pi } ( s ) \coloneqq \mathbb { E } _ { \pi, p } \left [ \sum _ { t = 0 } ^ { \infty } \gamma ^ { t } r ( s _ { t }, a _ { t }, s _ { t + 1 } ) | s _ { 0 } = s \right ],$$

or the state-action value function:

$$Q ^ { \pi } ( s, a ) \coloneqq \mathbb { E } _ { \pi, p } \left [ \sum _ { t = 0 } ^ { \infty } \gamma ^ { t } r ( s _ { t }, a _ { t }, s _ { t + 1 } ) | s _ { 0 } = s, a _ { 0 } = a \right ]. \quad \begin{array} { c } \text{pul\ } \\ \text{c.\ } \end{array}$$

## B. TRAINING SOFT ACTOR-CRITIC

Soft Actor-Critic (SAC) [5] is an actor-critic off-policy RL algorithm widely used for solving continuous control tasks. This paper adopts SAC as the backbone of the proposed method for robustness evaluation due to its stability and sample efficiency in continuous control benchmarks. However, our method can also be easily extended to other algorithms, such as Proximal Policy Optimization (PPO) [51] and Twin Delayed Deep Deterministic Policy Gradient (TD3) [4].

SAC alternates between the soft policy evaluation and soft policy improvement steps to learn the optimal stochastic policy that maximizes the expected returns and entropy of actions. Denoting π θ as a stochastic policy parameterized by θ . In soft policy evaluation step, SAC learns two critic networks, denoted as Q ϕ 1 and Q ϕ 2 , parameterized by ϕ 1 and ϕ 2 , respectively, by minimizing the soft Bellman residual:

$$J _ { Q } ( \phi _ { i } ) = \mathbb { E } _ { ( s, a, s ^ { \prime }, r ) \sim \mathcal { R }, a ^ { \prime } \sim \pi _ { \theta } } \left [ ( Q _ { \phi _ { i } } ( s, a ) - y ) ^ { 2 } \right ], \ \ ( 1 ) \quad \inf _ { \text{F} \mathfrak { g }. 1 }$$

where, y = r + γ [min i =1 2 , Q ˆ ϕ i ( s , a ′ ′ ) -α log π θ ( a ′ | s ′ )] , R is the replay buffer, α is the temperature parameter. The

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-48d0a744-68fc-460c-8edb-a55634e4c24a.jpg)

ǁ

ǁ

ǁ

FIGURE 1: Transformed Input-robust RL (TIRL)-Vector Quantization (VQ) : Reinforcement learning with the state adversary at test time. The state s is adversarially perturbed by the adversary Ψ( ) s into ˜ s , which is then transformed by the transformation T before being fed to the agent.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a70bc7ac-71e6-4f77-84c1-60416f1b4d28.jpg)

parameter ˆ ϕ i denotes the exponential moving average (EMA) of ϕ i , which empirically improves training stability in offpolicy RL algorithms [4], [5]. In soft policy improvement step, the actor policy π θ is optimized by maximizing the following objective:

$$\colon & \text{the} \\ \text{ived} \quad \text{$J_{\pi}(\theta)=\mathbb{E}_{s\sim\mathcal{R}, a\sim\pi_{\theta} } \left [ \min _ { i = 1, 2 } Q_{\phi _ { i } } ( s, a ) - \alpha \log ( \pi _ { \theta } ( a | s ) ) \right ], \\ \text{nice} \quad \text{...} \dots \quad \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \end{cases}$$

where, the stochastic policy π θ ( a s | ) is a parametric tanhGaussian that given the state s , an action a is sampled from tanh ( µ s ( ) + σ s ϵ ( ) ) , with ϵ ∼ N (0 , 1) , and µ and σ are parametric mean and standard deviation. Finally, the temperature α is learned with following objective:

$$J ( \alpha ) = \mathbb { E } _ { s \sim \mathcal { R }, a \sim \pi _ { \theta } } \left [ - \alpha \log \pi _ { \theta } ( a | s ) - \alpha \bar { \mathcal { H } } \right ], \quad ( 3 )$$

where, ¯ H ∈ R is the target entropy hyperparameter that policy tries to match, which is usually set to -|A| in practice with A is the action space.

## C. TEST-TIME ADVERSARIAL ATTACKS

After training, the agent is deployed within its designated environment and operates according to a fixed pretrained policy π θ , i.e. , the parameter θ is frozen. During this testing phase, an adversary can observe interactions between the victim agent and the environment, including states, actions, and rewards, to mislead the agent. However, the adversary does not possess knowledge of the environment's dynamics nor the capability to alter the environment directly. We focus on a typical state adversary that targets the manipulation of the state observation of the RL agent [9]-[11], [13], [22], [27], [31], [48], [52], [53]. The MDP with a state adversary involved is formally defined by the state-adversarial Markov decision process (SA-MDP) introduced in [21]. Let Ψ : S → S represent a state adversary. At each time step, the adversary Ψ , equipped with a certain budget ϵ , is allowed to inject perturbations to state s returned by the environment before the agent perceives them, as illustrated in Fig. 1. It is important to note that the underlying states of the environment, returned rewards, and executed actions remain unaffected. This setting aligns with many realistic scenarios,

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a66c4601-5084-45ea-bf2f-af674c26da6c.jpg)

such as sensor measurement errors, noise in sensory signals, or man-in-the-middle (MITM) attacks on a deep RL system. For instance, in robotic manipulation, an attacker could inject imperceptible noises into the camera, capturing an object without altering the actual object's location. As in many of the existing works investigating adversarial robustness of deep RL [9], [13], [21], [22], [28], [31], we focus on the ℓ ∞ -norm attack, in which the adversary Ψ is constrained to perturb the state s ∈ S into another 'neighboring' state ˜ s within an ϵ -radius ℓ ∞ -norm ball around s . Formally, the perturbed state ˜ = Ψ( ) s s ∈ B ϵ ( s ) , with B ϵ ( s ) = { ˜ : s ∥ s - ∥ ˜ s ∞ ≤ } ϵ . The details of state adversaries are described in Section V-A.

## IV. TRANSFORMED INPUT-ROBUST RL (TIRL)

In this section, we theoretically analyze the effectiveness of input transformation-based defenses in improving the robustness of RL agents. Subsequently, two principles are proposed to design transformation defenses for RL agents: 1) Bounded Transformation and 2) Autoencoder-styled Denoising . Based on these principles, we explore potential transformations to counter adversarial attacks.

## A. EFFECTIVE INPUT TRANSFORMATION DEFENSES

We first characterize the agent's performance changes before and after attacks by measuring the difference in value functions between the policy in MDP and SA-MDP. Let π be the policy, V π ( s ) be its state value function in the regular MDP, and V π ◦ Ψ ( s ) be its state value function in the SA-MDP with the adversary Ψ . Prior work [21] provides an upper bound on the difference between value functions caused by an optimal adversary. Specifically, theorem 5 in [21] states that, under the optimal adversary Ψ ∗ in SA-MDP, the performance gap between V π ( s ) and V π ◦ Ψ ∗ ( s ) can be bounded as follows:

$$\max _ { s \in \mathcal { S } } \{ V ^ { \pi } ( s ) - V ^ { \pi \circ \Psi ^ { * } } ( s ) \} & \leq \kappa \max _ { s \in \mathcal { S } } \max _ { \tilde { s } \in \mathcal { B } _ { \epsilon } ( s ) } D _ { T V } ( \pi ( \cdot | s ), \pi ( \cdot | \tilde { s } ) ) \quad \text{ensuri} \\ \cdot \quad \sim \quad \nu \quad \nu + \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot$$

where, D TV ( π ( ·| s , π ) ( ·| ˜)) s represents the total variance distance between π ( ·| s ) and π ( ·| ˜) s , κ is a constant independent of π , B ϵ ( s ) is defined as in Section III-C, and π ◦ Ψ ∗ denotes the policy under perturbations, i.e. , π a ( | Ψ ( )) ∗ s . The detailed proof for Eq. 4, which is based on tools developed in constrained policy optimization [54], can be found in the Appendix of [21].

Based on the upper bound between value functions, [21], [27] proposed regularizing D TV ( π ( ·| s , π ) ( ·| ˜)) s during policy training. In contrast, we focus on simple yet effective input transformations to reduce the performance gap. Let T be a transformation that maps S → ¯ S , where ¯ S is a set of all transformed states. This transformation can be applied during the training and testing phases or exclusively during the testing phase. Our following proposition establishes the connection between the performance gap and the input transformations.

Proposition 1: Consider a K -Lipschitz continuous policy π parameterized by the Gaussian distribution with a constant variance independent of state for a regular MDP. Let the corresponding value function be V π ( s ) . Define T t and T d as the transformations applied during training and testing, respectively. Under the optimal adversary Ψ ∗ in SA-MDP, for all s ∈ S we have:

$$\begin{array} { c c c } \lambda & \lambda \\ \text{many} \\ \text{ess of} & \max _ { s \in \mathcal { S } } ^ { \times } \\ \text{$\alpha_{s \in \mathcal { S } } ^ { \pi \circ \mathcal { T } _ { d } \circ \Psi ^ { * } } ( s ) \} \leq \zeta \max _ { s \in \mathcal { S } } \max _ { \bar { s } \in \mathcal { B } _ { \epsilon } ( s ) } \| \mathcal { T } _ { t } ( s ) - \mathcal { T } _ { d } ( \bar { s } ) \| _ { 2 }, \\ \text{$\epsilon_{\infty}^{-}} \\ \text{to per-} \\ \text{other}, \, \zeta \text{ is a constant independent of $\pi$. Here, $\pi\circ \mathcal{T}_{t}$ and} \\ \text{within} & \pi \circ \mathcal { T } _ { d } \circ \Psi ^ { * } \text{ denote $\pi(\cdot| \mathcal { T } _ { t } ( s ) ) \text{ and $\pi(\cdot| \mathcal{T } _ { d } ( \Psi ( s ) ) ), respectively.} \end{array}$$

Proof: Based on Pinsker's inequality, we first upper bound the total variance distance in RHS of Eq. (4) by KullbackLeibler (KL) divergence:

$$\text{ss} _ { \text{ne} } \quad D _ { T V } ( \pi ( \cdot | s ), \pi ( \cdot | \tilde { s } ) ) \leq \sqrt { \frac { 1 } { 2 } D _ { K L } \left ( \pi ( \cdot | s ) \, \| \, \pi ( \cdot | \tilde { s } ) \right ) }. \quad ( 6 )$$

Assuming the state space has dimension d , and given that the policy is Gaussian with constant independence variance, we denote π ( ·| s ) ∼ N ( µ , s Σ ) s and π ( ·| ˜) s ∼ N ( µ , ˜ s Σ ) ˜ s , where µ , µ s ˜ s ∈ R d are produced by neural networks µ θ ( s , µ ) θ (˜) s , repectively, and Σ is a diagonal matrix independent of the state, i.e., Σ s = Σ ˜ s = Σ . Using the K -Lipschitz assumption of the policy network, we can bound the KL divergence as follows:

$$\text{:fore} \quad & \text{follows} \quad \\ \text{un-} \quad & \text{the} \quad & \text{$D_{K}L\left(\pi(\cdot|s) \left\|\pi(\cdot|\bar{s}) \right) = \frac{1}{2}(\log\frac{|\Sigma_{\bar{s}}|}-d+ttr(\Sigma_{\bar{s}}^{-1}\Sigma_{s})} \\ \text{with} \quad & \quad & \quad + ( \mu _ { \bar { s } } - \mu _ { s } ) ^ { \top } \Sigma _ { \bar { s } } ^ { - 1 } ( \mu _ { \bar { s } } - \mu _ { s } ) ) \\ \text{d on} \quad & \quad & \leq L \| \mu _ { \bar { s } } - \mu _ { s } \| _ { 2 } ^ { 2 } \\ \text{mal} \quad & \quad & \quad = L \| \mu _ { \theta } ( s ) - \mu _ { \theta } ( \bar { s } ) \| _ { 2 } ^ { 2 } \\ \text{nder} \quad & \quad & \leq L K \| s - \tilde { s } \| _ { 2 } ^ { 2 }. \\ \text{s} \quad & \quad \text{The first inequality holds because $\Sigma_{\bar{s}},\Sigma_{s}$ are positive, then} \\ \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \, \, \,$$

The first inequality holds because Σ ˜ s , Σ s are positive, then ensuring there exists a positive constant L ∈ R + that satisfies this inequality. The second inequality holds because the policy network is K -Lipschitz continuous. Let T t and T d be the transformations applied into state s and ˜ s , respectively. By combining Equation (4), (6), and (7), we obtain:

$$L H S \, \text{of} \, ( 4 ) \leq \zeta \max _ { s \in \mathcal { S } } \max _ { \tilde { s } \in \mathcal { B } _ { \epsilon } ( s ) } \| \mathcal { T } _ { t } ( s ) - \mathcal { T } _ { d } ( \tilde { s } ) \| _ { 2 } \quad ( 8 )$$

$$\text{in} \, \text{the} \quad \text{ where } \zeta \coloneqq \frac { 1 } { \sqrt { 2 } } \kappa \sqrt { L K } \,. \, \text{The proof is complete.} \quad \quad \Box$$

## B. INPUT TRANSFORMATION PRINCIPLES

Proposition 1 says that the performance gap can be bounded by the difference between the 'origin' state and 'perturbed' state in the transformed space, i.e. , ∥T t ( s ) -T d (˜) s ∥ 2 . It suggests two distinct principles for reducing the performance gap through transformations. We summarize our two principles as follows:

- 1) Bounded Transformation : If T t and T d are used for both training and testing, we can design T t and T d such that, after transformation, the difference between the state and its perturbed version is small, i.e. , max ˜ s ∈B ϵ ( s ) ∥T t ( s ) -T d (˜) s ∥ 2 is minimized.

FIGURE 2: Top: Illustration of using bit-depth reduction to quantize the 1-D state space. The state is assigned to the closest point. Bottom: The current state and its perturbed versions may be assigned the same value if ϵ is not too large. A larger bin width ( bW ) led to greater robustness.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-090c166b-2deb-437a-a468-4e747c5f5a07.jpg)

- 2) Autoencoder-styled denoising : If T t is the identity function, i.e. , T t is not used during training, we need to design T d such that it minimizes max ˜ s ∈B ϵ ( s ) ∥ s -T d (˜) s ∥ 2 . In other words, T d needs to reconstruct origin state s from the perturbed state ˜ s . This naturally leads to the selection of denoisers for T d .

In the autoencoder-styled denoising approach, we can theoretically maintain the performance under attacks, V π ◦T ◦ d Ψ ∗ ( s ) , close to the natural performance V π ( s ) as long as the denoiser can effectively reconstruct the original state s from the perturbed state ˜ s . However, this approach might be vulnerable to fully white-box gradient-based attacks, where adversaries have access to the entire policy's architecture, including its parameters and the denoiser. In the bounded transformation approach, the strength of these defenses lies in their non-differential nature, making them more resilient to white-box attacks. However, due to the use of transformations during training, it is not guaranteed that the natural performance under the transformation, V π ◦T t ( s ) , is identical to V π ( s ) , as some information in the state may be lost. Nevertheless, as long as T t retains the essential information from the original space, we can expect V π ◦T t ( s ) to be close to V π ( s ) , as empirically observed in our experiments. We present input transformation-based defenses following these two principles in the following parts.

## C. VARIOUS INPUT TRANSFORMATIONS

We investigate various input transformations to satisfy the two principles. In each principle, we can summarize as follows: (1) Bounded Transformation (bit-depth reduction and vector quantization (VQ) to achieve close transformations of origin and perturbed state and (2) Autoencoder-styled Denoising, i.e., VAED to reconstruct origin state.

## 1) Bounded Transformation: Bit-Depth Reduction

The bit-depth reduction (BDR) transformation, initially explored as a defense in image classification tasks [36], [37], is adopted here for our control tasks. The BDR performs uniform quantization for each input dimension, assigning each value to the nearest quantized point (Fig. 2 top). If the adversary's budget ϵ is not too large, the state s and its perturbed state ˜ s within the ϵ -radius can be assigned the same value after transformation (Fig. 2 bottom). The BDR adheres to the

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-99a097fb-8da7-433a-85a0-3bfc89001c6e.jpg)

FIGURE 3: Illustration of using vector quantization to reduce the space of adversarial attacks in a 2-D state space. The green dots represent the centroids of clusters, while the gray dotted lines mark the boundaries between clusters. Each state is assigned to closest centroid. Fewer clusters result in a sparser state space, leading to greater robustness.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-aa4e08af-cff5-43b6-ae73-8f26fa5a2d2e.jpg)

bounded transformation principle, keeping ∥T t ( s ) -T d (˜) s ∥ 2 small overall. The original design of BDR for images does not directly apply to continuous vector states due to varying value ranges and the continuous nature of the data. To address the limited flexibility, we redesigned BDR by partitioning each dimension into equally spaced bins, controlled by the bin width ( bW ) parameter, and rounding real-valued scalars to the nearest bin value. To balance robustness and natural performance, we adjust the bW parameter: increasing the bin width enhances defense against attacks but may reduce overall performance and vice versa.

## 2) Bounded Transformation: Vector Quantization

We propose a new sparse input transformation using vector quantization (VQ) to discretize the entire state space into discrete vectors more strongly, as shown in Fig. 3. To perform VQ for state space, we employ the K-means algorithm. Specifically, given a set of state vectors { s i } N i =1 , the K-means algorithm partitions the N points into K clusters. Subsequently, each state is then represented by its nearest centroid in the codebook C = [ c 1 , c 2 , . . . , c K ] . The training of Kmeans is achieved by minimizing the following objective:

$$\min _ { C, \{ p _ { i } \} } \sum _ { i = 1 } ^ { N } \| s _ { i } - C p _ { i } \| _ { 2 } ^ { 2 }$$

where p i ∈ { 0 1 , } K is a one-hot vector with the value 1 indicating the index of the centroid nearest to data point s i and K controls the codebook size. In the backward pass, the VQ module is treated as an identity function, referred to as straight-through gradient estimation [55]. The above objective is also convenient for learning the codebook online, which is suitable for concurrent training with RL algorithms since the data collected by the RL agent continuously evolves during training time. Unlike BDR, after transformation, VQ can produce a more 'sparse' state space, thus significantly reducing the space of attacks, leading to the agent being more robust against adversarial attacks.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a66c4601-5084-45ea-bf2f-af674c26da6c.jpg)

- 3) Autoencoder-styled Denoising

Training denoisers requires perturbed states, which are additionally collected by interacting with the environment under attack, making the process inefficient. To obtain stable denoising results, we only apply the autoencoder-styled denoiser [33], [35] after completing the training of the RL agent. Moreover, sampling data under online adversarial attacks can result in potentially unsafe behaviors, especially in a physical system. For instance, in manipulation tasks using a robotic arm, collecting samples under attacks might lead to the breakage of objects. To mitigate these challenges, we leverage samples from the replay buffer collected during the training of RL agents for denoiser training. To generate perturbed states, we use Action Diff [21] and Min Q [13] attacks, as they only require a policy or Q -value network and do not necessitate interaction with the environments. This strategy helps reduce costs for acquiring additional data and avoids unsafe behaviors. In this approach, we must select an appropriate ϵ -radius for generating perturbations. If the ϵ is too large, the denoiser might be unable to reconstruct the original states.

## Algorithm 1 TIRL for training the agent with SAC

- 1: Input : T training steps, initialize policy network π θ and critic networks Q ϕ 1 , Q ϕ 2 with parameters θ , ϕ 1 , and ϕ 2 , the target critic networks ˆ ϕ 1 ← ϕ , ϕ 1 ˆ 2 ← ϕ 2 , initialize replay buffer R . Transformation T and its hyperparameter, i.e. , bW for BDR or K for VQ.
- 2: for t = 0 to T do
- 3: Execute action a i ∼ π θ ( T ( s i )) into environment.
- 4: Store transition { s , a t t , r t , s t +1 } into R .
- 6: y j = r j + γ [min i =1 2 , Q ˆ ϕ i ( s , a ′ j ′ ) -α log π θ ( a ′ |T ( s ′ j ))] , for all j ∈ [ N ] , and a ′ ∼ π θ ( ·|T ( s ′ j )) .
- 5: Sample a mini-batch of N samples { s , a j j , r j , s ′ j } from R .
- 7: Train critic:

$$\Phi _ { \Phi } \Phi _ { i } ) = \frac { 1 } { N } \sum _ { j } ( Q _ { \phi _ { i } } ( s _ { j }, a _ { j } ) - y _ { j } ) ^ { 2 }$$

- 8: Train policy: J θ ( ) = 1 N ∑ j min i =1 2 , Q ϕ i ( s , a j ) -α log( π θ ( a |T ( s j ))) where, a ∼ π θ ( ·|T ( s j )) .
- 9: if TIRL-VQ is used then
- 10: Update codebook C for TIRL-VQ using Eq. (9).
- 11: end if
- 12: end for

## Algorithm 2 TIRL for the agent at test time

- 1: Input : N episodes for evaluation, pretrained policy network π θ from Alg. 1. Adversary Ψ and ϵ -radius. Transformation T .
- 2: for i = 0 to N do
- 3: while not done do

4:

Get state s from environment

- 5: Get perturbed state ˜ = Ψ( ) s s

6:

Execute action

a

∼

π

θ

(

T

(˜))

s

- 7: end while
- 8: end for

The pseudo-code for applying BDR or VQ during agent training is shown in Algorithm 1. Note that we only apply these transformations to the input of the policy network.

into environment.

After training is complete, the transformations are applied to mitigate adversarial attacks, as demonstrated in Algorithm 2.

## V. EXPERIMENTS

In this section, we evaluate the efficacy of our defenses across continuous control tasks. Section V-A details the setup and adversaries used to evaluate the robustness. The experiment in Section V-B evaluates performance in a gray-box setting, where adversaries can access the model architecture and parameters but are unaware of the applied transformations. The experiment in Section V-C focuses on a white-box setting, where the adversaries are aware of the defense strategies.

## A. EXPERIMENTAL SETUP

Environments and Baseline. We evaluate the effectiveness defenses on five MuJoCo environments, as commonly used in the literature [4], [21], [57], including Walker2d, Hopper, Ant, Inverted Pendulum, and Reacher, as shown in Fig. 4. We use SAC as a base RL algorithm with the implementation from [58], the hyperparameters for the five environments similar to those in [5]. When applying the input transformations, we maintain the same hyperparameters for the underlying RL algorithm and only conduct a grid search for the transformation's hyperparameter. We run Walker2d and Ant 3 × 10 6 steps, Hopper 10 6 steps, and Inverted Pendulum and Reacher 2 × 10 5 steps.

Adversaries. We attack the trained policies with multiple existing attack methods, as used in [9], [21], [22], [31], including Random , Action Diff , Min Q , Robust Sarsa (RS) , and Policy Adversarial Actor Director (PA-AD) . Specifically, given an attack budget ϵ , a state s is adversarially perturbed into state ˜ s by adversaries as follows:

- · Random uniformly samples perturbed states within an ϵ -radius ℓ ∞ -norm ball.
- · Action Diff [21] is an attack method relied on the agent's policy. It directly finds the perturbed states within an ϵ -radius ℓ ∞ -norm ball to satisfy: max ˜ s ∈B ϵ ( s ) D KL ( π θ ( ·| s ) || π θ ( ·| ˜)) s with D KL is the Kullback-Leibler divergence.
- · Min Q [13] relies on the agent's policy and Q function to perform attacks. It generates the perturbed states within an ϵ -radius ℓ ∞ -norm ball to satisfy: min ˜ s ∈B ϵ ( s ) Q ϕ ( s, π θ (˜)) s
- · Robust Sarsa (RS) [21] is performed similar to Min Q attack with a robust action-value function.
- · Policy Adversarial Actor Director (PA-AD) [31] is currently the most powerful attack on RL agents, as it learns the optimal adversary using RL. Specifically, given a pretrained policy π θ , PA-AD learns an adversarial policy π adv , which takes the state s as input to generate the adversarial state ˜ s . The adversarial policy is trained using Proximal Policy Optimization (PPO) [51].

Generally, the strength of attackers is ranked in ascending order as follows: Random &lt; Action Diff &lt; Min Q &lt; RS &lt; PA-AD . For Action Diff , Min Q , and RS , we use projected gradient descent (PGD) to generate the optimal perturbed

FIGURE 4: The five MuJoCo environments in OpenAI Gym [56] are used to evaluate the robustness for SAC-based TILR.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-9f2e6819-0faf-4958-bafa-24aa1d389524.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-d50893bc-a263-4ba1-98a5-bcca5b7f60fc.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-7ec10d7b-629c-49af-9781-c47d7b102064.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-7d230439-083b-4dd7-8804-272a6780f72f.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-9cc14cb9-68e1-4e24-8860-3fbe14ca3c61.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a81dc5b9-86b6-4b85-b9f0-0feedc2591b3.jpg)

TABLE 1: Main hyperparameter of TIRL.

| TIRL / Env.   | Ant      | Hopper   | Walker2d   | Pendulum   | Reacher   |
|---------------|----------|----------|------------|------------|-----------|
| BDR ( bW )    | 0.05     | 0.15     | 0.1        | 0.05       | 0.4       |
| VQ ( K )      | 5 × 10 5 | 5 × 10 5 | 10 5       | 3 × 10 5   | 2 × 10 5  |
| AED ( ϵ )     | 0.15     | 0.1      | 0.05       | 0.3        | 0.25      |
| VAED ( ϵ )    | 0.15     | 0.1      | 0.05       | 0.3        | 0.25      |

state, as in [10], [13], [21], [59]. For example, the perturbed state ˜ s is obtained by PGD in Action Diff attack as follows:

mean square error: ∥ s -D ω 2 ( E ω 1 (˜) s ∥ 2 2 , where E ω 1 and D ω 2 are the encoder and decoder, respectively. Note that the denoiser is only used to denoise the perturbed inputs of the policy network during testing. We use the Adam optimizer [60] for training the denoiser, with a learning rate of 3 × 10 -4 . For the Min Q attacker, we search the ϵ -radius in { 0 25 0 3 . , . , 0 35 0 4 . , . } for Inverted Pendulum, and { 0 05 . , 0 1 . , 0 15 . , 0 2 . , 0 25 . } for the other environments.

$$\tilde { s } ^ { k + 1 } = \tilde { s } ^ { k } + \eta \, \text{proj} [ \nabla _ { \tilde { s } ^ { k } } D _ { K L } ( \pi _ { \theta } ( \cdot | s ) | | \pi _ { \theta } ( \cdot | \tilde { s } ^ { k } ) ) ] \ \ ( 1 0 ) \quad \ \text{aut}$$

where ˜ s 0 = s , k = 0 , . . . , H -1 , proj [ ] . is a projection to B ϵ ( s ) , η is the step size, and H is the number of iterations. Through experiments, we use 10-step PGD for optimization for these attacks ( i.e. , H = 10 ) with a step size η = 0 1 . . For PA-AD, we follow the setup in [31] to evaluate pretrained SAC policies. The architecture of the adversarial policy is similar to the SAC policy, consisting of two 256dimensional hidden layers followed by ReLU activations, except for the last layer, which uses Tanh activation. The ϵ -radius of attackers for each environment is shown in the tables.

## Input Transformations for Robust RL

- · TIRL-BDR: We apply the BDR transformation to the input of the policy network of SAC, while leaving the input of the Q networks unchanged. Note that BDR is applied in both training (in a regular environment) and testing (with adversaries). The hyperparameter bW is searched in { 0 1 . , 0 25 0 4 . , . , 0 5 . } for Reacher, and { 0 05 0 1 . , . , 0 15 . , 0 2 . } for the other environments.
- · TIRL-VQ: Similar to BDR, the VQ transformation is applied only to the input of the policy network and is used during both training and testing. The codebook is learned concurrently with the agent while training by optimizing Eq.( 9); during testing, the learned codebook is fixed. The codebook size K is searched in { 2 × 10 5 , 5 × 10 5 , 10 6 } for Ant, Hopper, and Walker2d, and { 5 × 10 4 , 1 × 10 5 , 2 × 10 5 , 3 × 10 5 , 5 × 10 5 } for the other environments.
- · TIRL-AED: We use an autoencoder-based denoiser to reconstruct an original state from a perturbed state. To train the denoiser, we collect a set of N states { s i } N i =1 from a pretrained policy's replay buffer, then use Min Q attacks to generate perturbed states { ˜ s i } N i =1 . The autoencoder is trained to reconstruct the original states by optimizing the
- · TIRL-VAED: We follow a similar setting as TIRL-AED, except for the training objective. We utilize the variational autoencoder [61] objective: -βD KL ( E ω 1 ( z s | ˜) ∥ p z ( )) + E E ω 1 ( z s | ˜) [log D ω 2 ( s z | )] , where p z ( ) is the standard normal distribution, and β is set to 1 × 10 -4 . We also search the ϵ -radius in the same range as TIRL-AED.

We summarize the best hyperparameters for transformations across environments in the Table 1.

## B. EVALUATION WITH GRAY-BOX ATTACKS

In this scenario, adversaries can access the policy architecture and its parameters but not the applied transformations. When generating perturbations, the clean state is fed directly into the policy network, bypassing transformations. For example, in the Action Diff attack, the adversarial state ˜ s is generated using the formula: ˜ s k +1 = ˜ s k + η proj [ ∇ ˜ s k D KL ( π θ ( ·| s ) || π θ ( ·| ˜ )] s k , ignoring any transformations applied during policy training. Min Q and RS are analogous to Action Diff . In RL-based attacks like PAAD, the transformation is also excluded from the pre-trained policy when learning the adversarial policy.

The performance of both the vanilla SAC agent and the TIRL with various transformations under different attacks is presented in Table 2. The results illustrate the effectiveness of the investigated transformations in mitigating the impact of attacks. The TIRL-BDR improves performance by 31% over the vanilla SAC across five environments, achieving the highest robustness in the Walker2d and Inverted Pendulum environments at 45% and 44% , respectively. The TIRL-VQ significantly enhances robustness by 55% over the vanilla SAC, with the most substantial improvements observed in the Ant ( 117% ) and Hopper ( 56% ) environment among transformations. In the Reacher environment, the TIRL-VAED excels with the highest performance, reaching 40% . When comparing the TIRL-AED and TIRL-VAED, both demonstrate comparable performance. Overall, in the gray-box setting, the TIRL-VQ transformation appears to be the most effective among the transformations considered. However, it is essen-

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a66c4601-5084-45ea-bf2f-af674c26da6c.jpg)

TABLE 2: Average episode return ± standard deviation of SAC agents and SAC-TIRL with various transformation defenses in the gray-box setting. Results are averaged over 5 runs, with evaluations conducted over 50 episodes per run. Following [7], we bold all scores within 5 percent of the maximum in each attack and environment ( ≥ 0.95 · max).

| Environment                       | Method        | Natural Returns   | Random       | Action Diff   | Min Q         | RS            | PA-AD         |   Average |
|-----------------------------------|---------------|-------------------|--------------|---------------|---------------|---------------|---------------|-----------|
| Ant (state-dim: 111) ϵ : 0.15     | SAC           | 6859 ± 126        | 5637 ± 293   | 454 ± 335     | 370 ± 279     | 463 ± 216     | -962 ± 341    |    2136.8 |
|                                   | SAC-TIRL-BDR  | 6543 ± 506        | 5445 ± 477   | 1113 ± 310    | 1071 ± 564    | 910 ± 329     | 899 ± 367     |    2663.5 |
|                                   | SAC-TIRL-VQ   | 4785 ± 164        | 4737 ± 205   | 4667 ± 405    | 4595 ± 90     | 4485 ± 498    | 4569 ± 237    |    4639.7 |
|                                   | SAC-TIRL-AED  | 6002 ± 430        | 5059 ± 499   | 1766 ± 803    | 1956 ± 949    | 1567 ± 887    | 1384 ± 697    |    2955.7 |
|                                   | SAC-TIRL-VAED | 5957 ± 804        | 5034 ± 525   | 2054 ± 784    | 2207 ± 752    | 1459 ± 467    | 1646 ± 851    |    3059.5 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC           | 3476 ± 163        | 2967 ± 414   | 1570 ± 261    | 1557 ± 325    | 1098 ± 194    | 1028 ± 279    |    1949.3 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-BDR  | 3516 ± 90         | 3015 ± 427   | 2620 ± 695    | 2320 ± 760    | 1687 ± 432    | 1774 ± 935    |    2488.7 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-VQ   | 3253 ± 182        | 3123 ± 288   | 3177 ± 218    | 3188 ± 300    | 2500 ± 46     | 2997 ± 595    |    3039.7 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-AED  | 3195 ± 409        | 2303 ± 383   | 1729 ± 325    | 1956 ± 221    | 1461 ± 47     | 1499 ± 565    |    2023.8 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-VAED | 3189 ± 665        | 2914 ± 289   | 1749 ± 176    | 1606 ± 277    | 1593 ± 367    | 1704 ± 291    |    2125.8 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC           | 6242 ± 383        | 6067 ± 433   | 3700 ± 655    | 3524 ± 600    | 3854 ± 1359   | 1782 ± 818    |    4194.8 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-BDR  | 6266 ± 417        | 6243 ± 392   | 6142 ± 387    | 6144 ± 401    | 5651 ± 688    | 5968 ± 156    |    6069   |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-VQ   | 5455 ± 459        | 5298 ± 479   | 5400 ± 547    | 5333 ± 277    | 5225 ± 607    | 5211 ± 453    |    5320.3 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-AED  | 6211 ± 367        | 5938 ± 458   | 4374 ± 797    | 4215 ± 1291   | 4909 ± 1122   | 3280 ± 1332   |    4821.2 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-VAED | 6184 ± 210        | 5704 ± 189   | 4596 ± 635    | 3975 ± 1090   | 4690 ± 1260   | 3240 ± 1589   |    4731.5 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC           | 1000 ± 0          | 916 ± 169    | 638 ± 304     | 736 ± 317     | 89 ± 76       | 188 ± 94      |     594.5 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-BDR  | 1000 ± 0          | 1000 ± 0     | 953 ± 66      | 963 ± 52      | 434 ± 94      | 789 ± 97      |     856.5 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-VQ   | 1000 ± 0          | 1000 ± 0     | 738 ± 230     | 814 ± 263     | 797 ± 134     | 703 ± 95      |     842   |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-AED  | 1000 ± 0          | 1000 ± 0     | 949 ± 40      | 869 ± 155     | 474 ± 68      | 489 ± 65      |     796.8 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-VAED | 1000 ± 0          | 1000 ± 0     | 775 ± 285     | 756 ± 388     | 499 ± 92      | 534 ± 99      |     760.7 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC           | -3.68 ± 0.07      | -7.77 ± 0.66 | -32.31 ± 2.0  | -29.59 ± 2.51 | -21.52 ± 0.53 | -21.36 ± 0.72 |     -19.4 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-BDR  | -3.76 ± 0.03      | -6.99 ± 0.32 | -26.95 ± 1.76 | -24.93 ± 1.35 | -17.58 ± 2.62 | -17.30 ± 0.52 |     -16.3 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-VQ   | -4.46 ± 0.03      | -7.57 ± 0.17 | -19.02 ± 1.87 | -18.60 ± 1.29 | -13.0 ± 0.39  | -14.1 ± 1.21  |     -12.8 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-AED  | -3.67 ± 0.09      | -8.22 ± 0.61 | -21.8 ± 3.0   | -21.46 ± 2.13 | -11.49 ± 1.73 | -14.90 ± 0.98 |     -13.6 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-VAED | -3.68 ± 0.10      | -7.04 ± 0.34 | -16.59 ± 1.61 | -16.51 ± 1.59 | -10.71 ± 1.56 | -14.72 ± 1.41 |     -11.5 |

tial to note that applying VQ impacts natural returns due to the loss of information in quantized states. This highlights the trade-off between balancing robustness and maintaining natural performance.

## C. EVALUATION WITH WHITE-BOX ATTACKS

While the previous experiment demonstrates that input transformation defenses are relatively effective in mitigating adversarial attacks, we now conduct experiments using white-box attacks in a more challenging setting. In this scenario, adversaries can access the policy architecture, parameters, and applied transformations. Specifically, in the Action Diff , Min Q , and RS attacks, adversaries are allowed to backpropagate through the transformations while generating perturbations. For example, the Action Diff generates an adversarial state as follows: ˜ s k +1 = ˜ s k + η proj [ ∇ ˜ s k D KL ( π θ ( ·|T ( s )) || π θ ( ·|T (˜ ))] s k . Similarly, RLbased adversaries can use the complete pretrained policy, including its transformations, in RL-based attacks to train the adversarial policy.

Table 3 presents the performance results in this scenario. The results show that TIRL-VQ maintains the highest robustness among all transformations, achieving a 50% improvement over vanilla SAC. Although TIRL-BDR's improvement decreases, it still partially mitigates the attacks' effects, showing a 20% improvement across five environments. In contrast, the autoencoder-style denoising approach exhibits a performance decline in this setting, resulting in outcomes similar to vanilla SAC. This decline is because the denoisers are composed of deep neural network components, which adversaries can easily exploit to generate more robust perturbations. Additionally, we observe that TIRL-VQ performs best in 4 out of 5 environments in the white-box setting, compared to 2 out of 5 environments in the gray-box setting. This indicates that VQ's performance hardly deteriorates as the adversary's strength increases.

## D. COMPARISON WITH PRIOR WORK

In this experiment, we compare our defenses with prior robust training-based methods, including SAC-SA [21] and PA-ATLA [31]. We conduct experiments on the Ant, Hopper, and Walker2d environments, which were also used in previous works. The results are presented in Table 4. In the Ant environment, we observe that SAC-TIRL-VQ performs comparably to PA-ATLA and better than SAC-SA. In the Walker2d environment, SAC-TIRL-VQ and TIRL-BDR are comparable to SAC-SA and outperform PA-ATLA. Across all three environments, SAC-TIRL-VQ achieves an average return of 4216 1 . , SAC-SA achieves an average return of 4196 1 . , and PA-ATLA achieves an average return of 3823 6 . . Overall, SAC-TIRL-VQ shows comparable improvement to SAC-SA across environments, with an approximate 52%

TABLE 3: Average episode return ± standard deviation of SAC agents and SAC-TIRL with various transformation defenses in the white-box setting. Results are averaged over 5 runs, with evaluations conducted over 50 episodes per run. Following [7], we bold all scores within 5 percent of the maximum in each attack and environment ( ≥ 0.95 · max).

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-0b40359e-e9a9-4710-9e2e-9852d192d0ad.jpg)

FIGURE 5: The ablation study of different bin width ( bW ) for BDR transformation in Hopper environment. We evaluate the robustness under different attacks with various ϵ scales.

| Environment                       | Method        | Natural Reward   | Random       | Action Diff   | Min Q         | RS            | PA-AD         |   Average |
|-----------------------------------|---------------|------------------|--------------|---------------|---------------|---------------|---------------|-----------|
| Ant (state-dim: 111) ϵ : 0.15     | SAC           | 6859 ± 126       | 5637 ± 293   | 454 ± 335     | 370 ± 279     | 463 ± 216     | -962 ± 341    |    2136.8 |
|                                   | SAC-TIRL-BDR  | 6543 ± 506       | 5445 ± 477   | 877 ± 54      | 767 ± 174     | 1073 ± 375    | 719 ± 513     |    2570.7 |
|                                   | SAC-TIRL-VQ   | 4785 ± 164       | 4737 ± 205   | 4699 ± 308    | 4456 ± 236    | 4379 ± 129    | 4103 ± 161    |    4526.5 |
|                                   | SAC-TIRL-AED  | 6002 ± 430       | 5059 ± 499   | 721 ± 153     | 819 ± 153     | 479 ± 393     | -154 ± 302    |    2154.3 |
|                                   | SAC-TIRL-VAED | 5957 ± 804       | 5034 ± 525   | 799 ± 395     | 864 ± 341     | 590 ± 314     | -150 ± 308    |    2182.3 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC           | 3476 ± 163       | 2967 ± 414   | 1570 ± 261    | 1557 ± 325    | 1098 ± 194    | 1028 ± 279    |    1949.3 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-BDR  | 3516 ± 90        | 3015 ± 427   | 2620 ± 695    | 2300 ± 634    | 1613 ± 432    | 1521 ± 811    |    2430.8 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-VQ   | 3253 ± 182       | 3123 ± 288   | 2947 ± 334    | 2706 ± 129    | 2369 ± 301    | 2634 ± 491    |    2838.7 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-AED  | 3195 ± 409       | 2303 ± 383   | 1374 ± 392    | 1527 ± 307    | 1234 ± 471    | 1176 ± 182    |    1801.5 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-VAED | 3189 ± 665       | 2914 ± 289   | 1445 ± 396    | 1864 ± 299    | 1446 ± 213    | 1097 ± 241    |    1992.5 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC           | 6242 ± 383       | 6067 ± 433   | 3700 ± 655    | 3524 ± 600    | 3854 ± 1359   | 1782 ± 818    |    4194.8 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-BDR  | 6266 ± 417       | 6243 ± 392   | 5180 ± 908    | 5265 ± 1028   | 5198 ± 550    | 4019 ± 1558   |    5361.8 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-VQ   | 5455 ± 459       | 5298 ± 479   | 5560 ± 282    | 4823 ± 524    | 5370 ± 349    | 5193 ± 558    |    5283.2 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-AED  | 6211 ± 367       | 5938 ± 458   | 4117 ± 1130   | 3942 ± 927    | 4876 ± 947    | 3011 ± 633    |    4682.5 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-VAED | 6184 ± 210       | 5704 ± 189   | 4158 ± 1291   | 3752 ± 1484   | 4639 ± 1260   | 3143 ± 858    |    4596.7 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC           | 1000 ± 0         | 916 ± 169    | 638 ± 304     | 736 ± 317     | 89 ± 76       | 188 ± 94      |     594.5 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-BDR  | 1000 ± 0         | 1000 ± 0     | 934 ± 47      | 946 ± 77      | 272 ± 235     | 223 ± 97      |     729.2 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-VQ   | 1000 ± 0         | 1000 ± 0     | 820 ± 181     | 744 ± 338     | 621 ± 360     | 597 ± 97      |     797   |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-AED  | 1000 ± 0         | 1000 ± 0     | 386 ± 436     | 398 ± 428     | 324 ± 125     | 248 ± 111     |     559.3 |
| Pendulum (state-dim: 4) ϵ : 0.3   | SAC-TIRL-VAED | 1000 ± 0         | 1000 ± 0     | 245 ± 29      | 241 ± 27      | 240 ± 15      | 121 ± 15      |     474.5 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC           | -3.68 ± 0.07     | -7.77 ± 0.66 | -32.31 ± 2.0  | -29.59 ± 2.51 | -21.52 ± 0.53 | -21.36 ± 0.72 |     -19.4 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-BDR  | -3.76 ± 0.03     | -6.99 ± 0.32 | -27.37 ± 1.97 | -25.37 ± 0.87 | -18.12 ± 1.67 | -19.91 ± 2.03 |     -16.9 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-VQ   | -4.46 ± 0.03     | -7.57 ± 0.17 | -15.83 ± 0.05 | -16.25 ± 1.25 | -15.2 ± 1.29  | -14.90 ± 2.16 |     -12.4 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-AED  | -3.67 ± 0.09     | -8.22 ± 0.61 | -22.15 ± 1.74 | -23.08 ± 1.88 | -13.21 ± 2.06 | -15.90 ± 1.22 |     -14.4 |
| Reacher (state-dim: 11) ϵ : 1.0   | SAC-TIRL-VAED | -3.68 ± 0.10     | -7.04 ± 0.34 | -18.2 ± 1.24  | -19.5 ± 2.25  | -16.01 ± 1.22 | -14.21 ± 1.64 |     -13.1 |

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-8d7f5622-be77-4796-8988-9c7504da2848.jpg)

improvement compared to vanilla SAC. It is worth noting that, unlike SAC-SA and PA-ATLA, we do not use adversarial examples during training. Moreover, our method only modifies the model's input, making it possible to combine with robust training-based defenses for further robustness enhancement. We leave this investigation for future work.

quently, with each pretrained model, we evaluate robustness as a function of adversary strength ( i.e. , the ϵ -radius) for each of the three attackers: Random Action Diff , , and Min Q .

## E. ABLATION EXPERIMENTS

In this section, we conduct an ablation study on hyperparameters for TIRL-BDR and TIRL-VQ transformations. We first train the agents with different hyperparameters. Subse-

Fig. 5 illustrates the robustness of TIRL-BDR in the Hopper environment with varying values of bin width ( bW ). As observed, if bW is small ( e.g. , 0.05), it resembles the robustness of vanilla SAC, particularly noticeable in Action Diff and Min Q attacks. This is because a smaller bW essentially reflects the original state space, thus having less impact on countering perturbations caused by attacks. However, a value that is too large, bW , dramatically drops natural performance

TABLE 4: Average episode return ± standard deviation of different defenses. Results are averaged over 5 runs, with evaluations conducted over 50 episodes per run. Following [7], we bold all scores within 5 percent of the maximum in each attack and environment ( ≥ 0.95 · max). Note that Min Q is not applicable for PA-ATLA since it does not contain a Q network.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a66c4601-5084-45ea-bf2f-af674c26da6c.jpg)

FIGURE 6: The abluation study of different codebook size ( K ) for VQ transformation in Reacher environment. The agents are evaluated under different attacks with various ϵ scales.

| Environment                       | Method       | Natural Reward   | Random      | Action Diff   | Min Q       | RS         | PA-AD       |   Average |
|-----------------------------------|--------------|------------------|-------------|---------------|-------------|------------|-------------|-----------|
| Ant (state-dim: 111) ϵ : 0.15     | SAC-TIRL-BDR | 6543 ± 506       | 5445 ± 477  | 877 ± 54      | 767 ± 174   | 1073 ± 375 | 719 ± 513   |    2570.7 |
| Ant (state-dim: 111) ϵ : 0.15     | SAC-TIRL-VQ  | 4785 ± 164       | 4737 ± 205  | 4699 ± 308    | 4456 ± 236  | 4379 ± 129 | 4103 ± 161  |    4526.5 |
| Ant (state-dim: 111) ϵ : 0.15     | SAC-SA       | 6626 ± 675       | 5263 ± 1657 | 4357 ± 1734   | 3735 ± 1647 | 3414 ± 850 | 1021 ± 943  |    4069.3 |
| Ant (state-dim: 111) ϵ : 0.15     | PA-ATLA      | 5469 ± 106       | 5469 ± 158  | 5328 ± 169    | -           | 4124 ± 291 | 2986 ± 864  |    4675.2 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-BDR | 3516 ± 90        | 3015 ± 427  | 2620 ± 695    | 2300 ± 634  | 1613 ± 432 | 1521 ± 811  |    2430.8 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-TIRL-VQ  | 3253 ± 182       | 3123 ± 288  | 2947 ± 334    | 2706 ± 129  | 2369 ± 301 | 2634 ± 491  |    2838.7 |
| Hopper (state-dim: 11) ϵ : 0.075  | SAC-SA       | 3652 ± 359       | 3522 ± 375  | 3394 ± 675    | 2987 ± 359  | 2869 ± 862 | 2504 ± 653  |    3154.7 |
| Hopper (state-dim: 11) ϵ : 0.075  | PA-ATLA      | 3449 ± 237       | 3325 ± 239  | 3145 ± 546    | -           | 3002 ± 129 | 2521 ± 325  |    3086.6 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-BDR | 6266 ± 417       | 6243 ± 392  | 5180 ± 908    | 5265 ± 1028 | 5198 ± 550 | 4019 ± 1558 |    5361.8 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-TIRL-VQ  | 5455 ± 459       | 5298 ± 479  | 5560 ± 282    | 4823 ± 524  | 5370 ± 349 | 5193 ± 558  |    5283.2 |
| Walker2d (state-dim: 17) ϵ : 0.05 | SAC-SA       | 5895 ± 305       | 5884 ± 419  | 5502 ± 499    | 5275 ± 234  | 5124 ± 383 | 4505 ± 684  |    5364.3 |
| Walker2d (state-dim: 17) ϵ : 0.05 | PA-ATLA      | 4178 ± 529       | 4129 ± 78   | 4024 ± 572    | -           | 3966 ± 307 | 2248 ± 131  |    3709   |

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-01d4efd5-2ac5-41f7-ad06-8927ae3e5cad.jpg)

due to inaccurate representations.

Fig. 6 showcases the results for various codebook sizes of the VQ transformation in the Reacher environment. Across all considered values, robustness in policy performance remains consistently strong. However, our primary objective is to identify smaller codebook sizes that yield higher natural rewards. This choice is motivated by the increasing computational cost of larger codebook sizes in the VQ transformation. Notably, in our experiments on the Hopper environment, the wall-clock time for training SAC-VQ with K = 5 × 10 5 is approximately 2.5 times longer compared to SAC-BDR. This cost differential becomes even more pronounced in higher-dimensional environments. Consequently, when selecting hyperparameters for VQ, we recommend commencing with smaller values and incrementally adjusting them until achieving a satisfactory balance between robustness and computational efficiency.

## F. COMPUTATIONAL COST COMPARISON

We compare the training and testing times for different transformations on a single machine equipped with a Tesla V100 16GB GPU. Note that TIRL-AED and TIRL-VAED are trained after completing the training of RL agents, which takes approximately 15-20 minutes for training the denoiser.

TABLE 5: Average training and testing times in the Ant environment. Note that only Bounded Transformation is used during RL agent training, while the Autoencoder-styled Denoising is applied after the agent's training is complete.

|               | Running time                    | Running time                      |
|---------------|---------------------------------|-----------------------------------|
| Method        | Training (hours per 10 6 steps) | Testing (seconds per 50 episodes) |
| SAC           | 5.28                            | 91                                |
| SAC-TIRL-BDR  | 5.38                            | 91.5                              |
| SAC-TIRL-VQ   | 10.54                           | 109.5                             |
| SAC-TIRL-AED  | -                               | 92.5                              |
| SAC-TIRL-VAED | -                               | 92.5                              |

The results are presented in Table 5. Although TIRL-VQ is effective in mitigating adversarial attacks, its training time with SAC is longer, primarily due to the K-means algorithm. This poses a limitation when extending our method to highdimensional states, such as images. Future investigations could explore more advanced variants of VQ to optimize efficiency further and expand the scope of potential applications.

## VI. CONCLUSIONS

Reinforcement Learning (RL) agents demonstrating proficiency in a training environment exhibit vulnerability to

adversarial perturbations in input observations during deployment. This underscores the importance of building a robust agent before its real-world deployment. To alleviate the challenging point, prior works focus on developing robust training-based procedures, encompassing efforts to fortify the deep neural network component's robustness or subject the agent to adversarial training against potent attacks. In this work, we proposed a novel method referred to as Transformed Input-robust RL (TIRL) , which explores another avenue to mitigate the impact of adversaries by employing input transformation-based defenses. Specifically, we introduced two principles for applying transformation-based defenses in learning robust RL agents: (1) autoencoder-styled denoising to reconstruct the original state and (2) bounded transformations (bit-depth reduction and vector quantization (VQ)) to achieve close transformed inputs. The transformations are applied to the state before feeding them into the policy network. Extensive experiments on multiple MuJoCo environments demonstrated that input transformation-based defenses, i.e. , VQ, defend against several adversaries in the state observations.

In this paper, we focus primarily on low-dimensional environments due to the high computational cost of applying VQ to high-dimensional environments like those in Atari games. Future research can address this challenge by developing more advanced and efficient VQ techniques. Additionally, our proposed TIRL method holds the potential to synergistically complement other robust training-based defenses, significantly enhancing the overall robustness of RL agents. This direction opens exciting opportunities for developing more resilient RL systems capable of operating effectively in complex and adversarial environments.

## REFERENCES

- [1] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., 'Human-level control through deep reinforcement learning,' Nature, pp. 529-533, 2015.
- [2] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, 'Trust region policy optimization,' in ICML, pp. 1889-1897, 2015.
- [3] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot, et al., 'Mastering the game of go with deep neural networks and tree search,' Nature, pp. 484-489, 2016.
- [4] S. Fujimoto, H. Hoof, and D. Meger, 'Addressing function approximation error in actor-critic methods,' in ICML, pp. 1587-1596, 2018.
- [5] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, 'Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor,' in ICML, pp. 1861-1870, 2018.
- [6] T. M. Luu and C. D. Yoo, 'Hindsight goal ranking on replay buffer for sparse reward environment,' IEEE Access, vol. 9, pp. 51996-52007, 2021.
- [7] I. Kostrikov, A. Nair, and S. Levine, 'Offline reinforcement learning with implicit q-learning,' ICLR, 2022.
- [8] T. M. Luu, T. Nguyen, T. Vu, and C. D. Yoo, 'Utilizing skipped frames in action repeats for improving sample efficiency in reinforcement learning,' IEEE Access, vol. 10, pp. 64965-64975, 2022.
- [9] S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel, 'Adversarial attacks on neural network policies,' arXiv preprint arXiv:1702.02284, 2017.
- [10] Y.-C. Lin, Z.-W. Hong, Y.-H. Liao, M.-L. Shih, M.-Y. Liu, and M. Sun, 'Tactics of adversarial attack on deep reinforcement learning agents,' arXiv preprint arXiv:1703.06748, 2017.
- [11] J. Kos and D. Song, 'Delving into adversarial attacks on deep policies,' arXiv preprint arXiv:1705.06452, 2017.
- [12] V. Behzadan and A. Munir, 'Vulnerability of deep reinforcement learning to policy induction attacks,' in MLDM, pp. 262-275, 2017.
- [13] A. Pattanaik, Z. Tang, S. Liu, G. Bommannan, and G. Chowdhary, 'Robust deep reinforcement learning with adversarial attacks,' AAMAS, 2018.
- [14] S. Shalev-Shwartz, S. Shammah, and A. Shashua, 'Safe, multiagent, reinforcement learning for autonomous driving,' arXiv preprint arXiv:1610.03295, 2016.
- [15] A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani, 'Deep reinforcement learning framework for autonomous driving,' arXiv preprint arXiv:1704.02532, 2017.
- [16] X. Pan, Y. You, Z. Wang, and C. Lu, 'Virtual to real reinforcement learning for autonomous driving,' arXiv preprint arXiv:1704.03952, 2017.
- [17] D. Gray, 'Introducing voyage deepdrive - unlocking the potential of deep reinforcement learning,' Nov 2019.
- [18] C. You, J. Lu, D. Filev, and P. Tsiotras, 'Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning,' Robotics and Autonomous Systems, vol. 114, pp. 118, 2019.
- [19] Y. Liang, Y. Sun, R. Zheng, and F. Huang, 'Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning,' NeurIPS, 2022.
- [20] X. He, B. Lou, H. Yang, and C. Lv, 'Robust decision making for autonomous vehicles at highway on-ramps: A constrained adversarial reinforcement learning approach,' IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 4, pp. 4103-4113, 2022.
- [21] H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C.-J. Hsieh, 'Robust deep reinforcement learning against adversarial perturbations on state observations,' NeurIPS, pp. 21024-21037, 2020.
- [22] H. Zhang, H. Chen, D. Boning, and C.-J. Hsieh, 'Robust reinforcement learning on state observations with learned optimal adversary,' ICLR, 2021.
- [23] S. Gupta, G. Singal, D. Garg, and S. Das, 'Rsac: a robust deep reinforcement learning strategy for dimensionality perturbation,' IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 6, no. 5, pp. 1157-1166, 2022.
- [24] J. Wu and Y. Vorobeychik, 'Robust deep reinforcement learning through bootstrapped opportunistic curriculum,' in International Conference on Machine Learning, pp. 24177-24211, PMLR, 2022.
- [25] Z. Liu, Z. Guo, Z. Cen, H. Zhang, J. Tan, B. Li, and D. Zhao, 'On the robustness of safe reinforcement learning under observational perturbations,' ICLR, 2023.
- [26] E. Korkmaz and J. Brown-Cohen, 'Detecting adversarial directions in deep reinforcement learning to make robust decisions,' ICML, 2023.
- [27] Q. Shen, Y. Li, H. Jiang, Z. Wang, and T. Zhao, 'Deep reinforcement learning with robust and smooth policy,' in ICML, pp. 8707-8718, 2020.
- [28] T. Oikarinen, W. Zhang, A. Megretski, L. Daniel, and T.-W. Weng, 'Robust deep reinforcement learning through adversarial loss,' NeurIPS, pp. 26156-26167, 2021.
- [29] R. Yang, C. Bai, X. Ma, Z. Wang, C. Zhang, and L. Han, 'Rorl: Robust offline reinforcement learning via conservative smoothing,' in NeurIPS, 2022.
- [30] V. Behzadan and A. Munir, 'Whatever does not kill deep reinforcement learning, makes it stronger,' arXiv preprint arXiv:1712.09344, 2017.
- [31] Y. Sun, R. Zheng, Y. Liang, and F. Huang, 'Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl,' ICLR, 2022.
- [32] G. K. Dziugaite, Z. Ghahramani, and D. M. Roy, 'A study of the effect of jpg compression on adversarial images,' arXiv:1608.00853, 2016.
- [33] D. Meng and H. Chen, 'Magnet: a two-pronged defense against adversarial examples,' in CCS, pp. 135-147, 2017.
- [34] J. Lu, H. Sibai, E. Fabry, and D. Forsyth, 'No need to worry about adversarial examples in object detection in autonomous vehicles,' arXiv:1707.03501, 2017.
- [35] F. Liao, M. Liang, Y. Dong, T. Pang, X. Hu, and J. Zhu, 'Defense against adversarial attacks using high-level representation guided denoiser,' in CVPR, pp. 1778-1787, 2018.
- [36] C. Guo, M. Rana, M. Cisse, and L. Van Der Maaten, 'Countering adversarial images using input transformations,' ICLR, 2018.
- [37] W. Xu, D. Evans, and Y. Qi, 'Feature squeezing: Detecting adversarial examples in deep neural networks,' NDSS, 2018.
- [38] A. Prakash, N. Moran, S. Garber, A. DiLillo, and J. Storer, 'Deflecting adversarial attacks with pixel deflection,' in CVPR, pp. 8571-8580, 2018.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-48d0a744-68fc-460c-8edb-a55634e4c24a.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00023v2-figure-a66c4601-5084-45ea-bf2f-af674c26da6c.jpg)

- [39] P. Samangouei, M. Kabkab, and R. Chellappa, 'Defense-gan: Protecting classifiers against adversarial attacks using generative models,' ICLR, 2018.
- [40] P. Gupta and E. Rahtu, 'Ciidefence: Defeating adversarial attacks by fusing class-specific image inpainting and image denoising,' in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 67086717, 2019.
- [41] G. Jin, S. Shen, D. Zhang, F. Dai, and Y. Zhang, 'Ape-gan: Adversarial perturbation elimination with gan,' in ICASSP, pp. 3842-3846, IEEE, 2019.
- [42] H. Salman, M. Sun, G. Yang, A. Kapoor, and J. Z. Kolter, 'Denoised smoothing: A provable defense for pretrained classifiers,' Advances in Neural Information Processing Systems, vol. 33, pp. 21945-21957, 2020.
- [43] W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar, 'Diffusion models for adversarial purification,' ICML, 2022.
- [44] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, 'Intriguing properties of neural networks,' arXiv preprint arXiv:1312.6199, 2013.
- [45] I. J. Goodfellow, J. Shlens, and C. Szegedy, 'Explaining and harnessing adversarial examples,' arXiv:1412.6572, 2014.
- [46] J. Sun, T. Zhang, X. Xie, L. Ma, Y. Zheng, K. Chen, and Y. Liu, 'Stealthy and efficient adversarial attacks against deep reinforcement learning,' in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 5883-5891, 2020.
- [47] K. Mo, W. Tang, J. Li, and X. Yuan, 'Attacking deep reinforcement learning with decoupled adversarial policy,' IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 1, pp. 758-768, 2022.
- [48] A. Russo and A. Proutiere, 'Optimal attacks on reinforcement learning policies,' ACC, 2021.
- [49] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial networks,' 2014.
- [50] J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' arXiv preprint arxiv:2006.11239, 2020.
- [51] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, 'Proximal policy optimization algorithms,' arXiv preprint arXiv:1707.06347, 2017.
- [52] T.-W. Weng, K. D. Dvijotham, J. Uesato, K. Xiao, S. Gowal, R. Stanforth, and P. Kohli, 'Toward evaluating robustness of deep reinforcement learning with continuous control,' in International Conference on Learning Representations, 2019.
- [53] M. Fischer, M. Mirman, S. Stalder, and M. Vechev, 'Online robustness training for deep reinforcement learning,' arXiv preprint arXiv:1911.00887, 2019.
- [54] J. Achiam, D. Held, A. Tamar, and P. Abbeel, 'Constrained policy optimization,' in International conference on machine learning, pp. 22-31, PMLR, 2017.
- [55] Y. Bengio, N. Léonard, and A. Courville, 'Estimating or propagating gradients through stochastic neurons for conditional computation,' arXiv:1308.3432, 2013.
- [56] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, 'Openai gym,' arXiv preprint arXiv:1606.01540, 2016.
- [57] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, 'Continuous control with deep reinforcement learning,' ICML, 2016.
- [58] T. Seno and M. Imai, 'd3rlpy: An offline deep reinforcement learning library,' Journal of Machine Learning Research, vol. 23, no. 315, pp. 1-20, 2022.
- [59] A. Kurakin, I. Goodfellow, and S. Bengio, 'Adversarial machine learning at scale,' arXiv preprint arXiv:1611.01236, 2016.
- [60] D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' Proc. Int. Conf. Learn. Represent. (ICLR), pp. 1-15, 2014.
- [61] D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' arXiv preprint arXiv:1312.6114, 2013.
Tung M. Luu Haeyong Kang Tri Ton Thanh Nguyen Chang D. Yoo
On the Perturbed States for Transformed Input-robust Reinforcement Learning
Reinforcement Learning (RL) agents demonstrating proficiency in a training
environment exhibit vulnerability to adversarial perturbations in input
observations during deployment. This underscores the importance of building a
robust agent before its real-world deployment. To alleviate the challenging
point, prior works focus on developing robust training-based procedures,
encompassing efforts to fortify the deep neural network component's robustness
or subject the agent to adversarial training against potent attacks. In this
work, we propose a novel method referred to as Transformed Input-robust RL
(TIRL), which explores another avenue to mitigate the impact of adversaries by
employing input transformation-based defenses. Specifically, we introduce two
principles for applying transformation-based defenses in learning robust RL
agents: (1) autoencoder-styled denoising to reconstruct the original state and
(2) bounded transformations (bit-depth reduction and vector quantization (VQ))
to achieve close transformed inputs. The transformations are applied to the
state before feeding it into the policy network. Extensive experiments on
multiple MuJoCo environments demonstrate that input transformation-based
defenses, i.e., VQ, defend against several adversaries in the state
observations. The official code is available at
https://github.com/tunglm2203/tirl