DEEP REINFORCEMENT LEARNING
Reinforcement Learning algorithms can learn complex behavioral patterns for sequential decision making tasks wherein an agent interacts with an environment and acquires feedback in the form of rewards sampled from it. Traditionally, such algorithms make decisions, i.e., select actions to execute, at every single time step of the agent-environment interactions. In this paper, we propose a novel framework, Fine Grained Action Repetition (FiGAR), which enables the agent to decide the action as well as the time scale of repeating it. FiGAR can be used for improving any Deep Reinforcement Learning algorithm which maintains an explicit policy estimate by enabling temporal abstractions in the action space. We empirically demonstrate the efficacy of our framework by showing performance improvements on top of three policy search algorithms in different domains: Asynchronous Advantage Actor Critic in the Atari 2600 domain, Trust Region Policy Optimization in Mujoco domain and Deep Deterministic Policy Gradients in the TORCS car racing domain.
1 INTRODUCTION
Reinforcement learning (RL) is used to solve goal-directed sequential decision making problems wherein explicit supervision in the form of correct decisions is not provided to the agent, but only evaluative feedback in the form of the rewards sampled from the environment. RL algorithms model goal-directed sequential decision making problems as Markov Decision Processes (MDP) [Sutton & Barto (1998)]. However, for problems with an exponential or continuous state space, tabular RL algorithms that maintain value or policy estimates for every state become infeasible. Therefore, there is a need to be able to generalize decision making to unseen states. Recent advances in representation learning through deep neural networks provide an efficient mechanism for such generalization [LeCun et al. (2015)]. Such a combination of representation learning through deep neural networks with reinforcement learning objectives has shown promising results in many sequential decision making domains such as the Atari 2600 domain [Bellemare et al. (2013); Mnih et al. (2015); Schaul et al. (2015); Mnih et al. (2016)], Mujoco simulated physics tasks domain [Todorov et al. (2012); Lillicrap et al. (2015)], the Robosoccer domain [Hausknecht et al. (2016)] and the TORCS domain [Wymann et al. (2000); Mnih et al. (2016)]. Often, MDP settings consist of an agent interacting with the environment at discrete time steps. A common feature shared by all the Deep Reinforcement Learning (DRL) algorithms above is that they repeatedly execute a chosen action for a fixed number of time steps k. If at represents the action taken at time step t, then for the said algorithms, a1 = a2 = · · · = ak, ak+1 = ak+2 = · · · = a2k and in general aik+1 = aik+2 = · · · = a(i+1)k, i ≥ 0. Action repetition allows these algorithms to compute the action once every k time steps and hence operate at higher speeds, thus achieving real-time performance. This also offers other advantages such as smooth action policies. More importantly, as shown in Lakshminarayanan et al. (2017) and Durugkar et al. (2016), macro-actions constituting the same action repeated k times could be interpreted as introducing temporal abstractions in the induced policies thereby enabling transitions between temporally distant advantageous states.
The time scale for action repetition has largely been static in DRL algorithms until now [Mnih et al. (2015; 2016); Schaul et al. (2015)]. Lakshminarayanan et al. (2017) are the first to explore dynamic time scales for action repetition in the DRL setting and show that it leads to significant improvement in performance on a few Atari 2600 games. However, they choose only two time scales and the experiments are limited to a few representative games. Moreover the method is limited to tasks with a discrete action space. We propose FiGAR, a framework that enables any DRL algorithm regardless of whether its action space is continuous or discrete, to learn temporal abstractions in the form of temporally extended macro-actions. FiGAR uses a structured and factored representation of the policy whereby the policy for choosing the action is decoupled from that for the action repetition selection. Note that deciding actions and the action repetitions independently enables us to find temporal abstractions without blowing up the action space, unlike Vezhnevets et al. (2016) and Lakshminarayanan et al. (2017). The contribution of this work is twofold. First, we propose a generic extension to DRL algorithms by coming up with a factored policy representation for temporal abstractions (see figure 1 for sequences of macro actions learnt in 2 Atari 2600 games). Second, we empirically demonstrate FiGAR’s efficiency in improving policy gradient DRL algorithms with improvements in performance over several domains: 31 Atari 2600 games with Asynchronous Advantage Actor Critic [Mnih et al. (2016)], 5 tasks in MuJoCo Simulated physics tasks domain with Trust Region Policy Optimization [Schulman et al. (2015)] and the TORCS domain with Deep Deterministic Policy Gradients [Lillicrap et al. (2015)].
2 RELATED WORK
Our framework is centered on a very general idea of only deciding when necessary. There have been similar ideas outside the RL domains. For instance, Gu et al. (2016) and Satija & Pineau (2016) explore Real Time Neural Machine Translation where the action at every time step is to decide whether to output a new token in the target language or not based on current context.
Transition Point Dynamic Programming (TPDP) [Buckland & Lawrence (1994)] algorithm is a modification to the tabular dynamic programming paradigm that can reduce the learning time and memory required for control of continuous stochastic dynamic systems. This is done by determining a set of transition points in the underlying MDP. The policy changes only at these transition point states. The algorithm learns an optimal set of transition point states by using a variant of Q-Learning to evaluate whether or not to add/delete a particular state from the set of transition points. FiGAR learns the transition points in the underlying MDP on the fly with generalization across the state space unlike TPDP which is tabular and infeasible for large problems.
The Dynamic Frameskip Deep Q-network [Lakshminarayanan et al. (2017)] proposes to use multiple time scales of action repetition by augmenting the Deep Q Network (DQN) [Mnih et al. (2015)] with separate streams of the same primitive actions corresponding to each time scale. This way, the time scale of action repetition is dynamically learned. Although this framework leads to a significant improvement in the performance on a few Atari 2600 games, it suffers from not being able to support multiple time scales due to potential explosion of the action space and is restricted to discrete action spaces. Durugkar et al. (2016) also explore learning macro-actions composed using the same action repeated for different time scales. However, their framework is limited to discrete action spaces and performance improvements are not significant.
Learning temporally extended actions and abstractions have been of interest in RL for a long time. Vezhnevets et al. (2016) propose Strategic Attentive Writer (STRAW) for learning macro-actions and building dynamic action-plans directly from reinforcement learning signals. Instead of outputting a single action after each observation, STRAW maintains a multi-step action plan. The agent periodically updates the plan based on observations and commits to the plan between the replanning steps. Although the STRAW framework represents a more general temporal abstraction than FiGAR, FiGAR should be seen as a framework that can compliment STRAW whereby the decision to repeat could now be hierarchical at plan and base action levels.
FiGAR is a framework that has a structured policy representation where the time scale of execution could be thought as parameterizing the chosen action. The only other work that explores parameterized policies in DRL is Hausknecht & Stone (2016) where discrete actions are parameterized by continuous values. In our case, discrete/continuous actions are parameterized by discrete values. The state spaces in Atari are also more sophisticated than the kind explored in Hausknecht et al. (2016).
FiGAR is also very naturally connected to the Semi-MDPs (SMDPs) framework. SMDPs are MDPs with durative actions. The assumption in SMDPs is that actions take some holding time to complete [Duff (1995); Mahadevan et al. (1997); Dietterich (2000)]. Typically, they are modeled with two distributions, one corresponding to the next state transition and the other corresponding to the holding time which denotes the number of time steps between the current action from the policy until the next action from the policy. The rewards over the entire holding time of an action is the credit assigned for picking the action. In our framework, we naturally have durative actions due to the policy structure where the decision consists of both the choice of the action and the time scale of its execution. Therefore, we convert the original MDP to an SMDP trivially. In fact, we give more structure to the SMDP because we are clear that we repeat the chosen action during the holding time, while what happens during the holding time is not specified in the SMDP framework. One can think of the part of the policy that outputs the probability distribution over the time scales as a holding time distribution. Therefore, our framework naturally fits into the SMDP definition with the action repetition rate characterizing the holding time. We also sum up the rewards over the holding time with an an appropriate discounting factor as in an SMDP framework.
3 BACKGROUND
3.1 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC
Actor critic algorithms execute policy gradient updates by maintaining parametric estimates for the policy πθa(a|s) and the value function Vθc(s) [Sutton & Barto (1998)]. The value function estimates are used to reduce the variance in the policy gradient updates.
Asynchronous Advantage Actor Critic (A3C) [Mnih et al. (2016)] learns policies based on an asynchronous n-step returns. The k learner threads execute k copies of the policy asynchronously and the parameter updates are sent to a central parameter server at regular intervals. This ensures that temporal correlations are broken between subsequent updates since the different threads possibly explore different parts of the state space in parallel. The objective function for policy improvement in A3C is: L(θa) = log πθa(at|st) (Gt − V (st)) where Gt is an estimate for the return at time step t. The A3C algorithm uses n-step returns for estimating Gt which is a biased estimate for Q(st, at). Hence one can think of Gt − V (st) as an estimate for A(st, at) which represents the advantage of taking action at in state st. The value
function Vθc(st) is updated by using n-step TD error as: L(θc) = ( V̂ (st)− Vθc(st) )2 where V̂ (st) is an estimate of the n-step return from the current state. In A3C j-step returns are used where j ≤ n and n is a fixed hyper-parameter. For simplicity assume that t ≤ n. Then the definition for V̂ (st) is:
V̂ (st) = n−1∑ j=t γt−jrj + γ n−tV (sn)
The policy and value functions are parameterized by Deep Neural Networks.
3.2 TRUST REGION POLICY OPTIMIZATION
TRPO [Schulman et al. (2015)] is a policy optimization algorithm. Constrained optimization of a surrogate loss function is proposed, with theoretical guarantees for monotonic policy improvement. The TRPO surrogate loss function L for potential next policies (π̃) is:
Lθold(θ̃) = η(π) + ∑ s ρπ(s) ∑ a π̃(a|s)Aπ(s, a)
where θold are the parameters of policy π and θ̃ are parameters of π̃. This surrogate loss function is optimized subject to the constraint: DmaxKL (π, π̃) ≤ δ which ensures that the policy improvement can be done in non-trivial step sizes and at the same time the new policy does not deviate much from the current policy due to the KL-divergence constraint.
3.3 DEEP DETERMINISTIC POLICY GRADIENTS
According to the Deterministic Policy Gradient (DPG) Theorem [Lever (2014)], the gradient of the performance objective (J) of the deterministic policy (µ) in continuous action spaces with respect to the policy parameters (θ) is given by:
∇θJ(µθ) = ∫ S ρµ(s)∇θµθ(s)∇aQµ(s, a)|a=µθ(s)ds
= Es∼ρµ [∇θµθ(s)∇aQµ(s, a)|a=µθ(s)] (1)
for an appropriately defined performance objective J . The DPG model built according to this theorem consists of an actor which outputs an action vector in the continuous action space and a critic model Q(s, a) which evaluates the action chosen at a state. The DDPG algorithm [Lillicrap et al. (2015)] extends the DPG algorithm by introducing non-linear neural network based function approximators for the actor and critic.
4 FIGAR: FINE GRAINED ACTION REPETITION
FiGAR provides a DRL algorithm with the ability to model temporal abstractions by augmenting it with the ability to predict the number of time steps for which an action chosen for execution is to be repeated. This prediction is conditioned on the current state in the environment.
The FiGAR framework can be used to extend any DRL algorithm (say Z) which maintains an explicit policy. Let Z ′ denote the extension of Z under FiGAR. Z ′ has two independent decoupled
Algorithm 1 Create FiGAR− Z 1: function MAKEFIGAR(DRLAlgorithm Z, ActionRepetitionSet W) 2: st ← state at time t 3: at ← action taken in st at time t 4: πa ← action policy of Z 5: fθa(st)← action network for realizing action policy πa 6: L(πa, st, at)← A’s objective function for improving πa 7: πx ← construct action repetition policy for FiGAR-Z. 8: fθx(st)← repetition network with output of size |W | for action repetition policy πx. 9: L(πx, st, at)← L evaluated at πx 10: T (st, at)← L(πx, st, at) ∗ L(πa, st, at) // Total Loss 11: return T, fθa , fθx
policy components. The policy πθa for choosing actions and the policy πθx for choosing action repetitions. Algorithm 1 describes the generic framework for deriving DRL algorithm Z ′ from algorithm Z. Let W stand for the set of all action repetitions that Z ′ would be able to perform. In tradition DRL algorithms, W = {c}, where c is a constant. This implies that the action repetition is static and fixed. In FiGAR, The set of action repetitions from which Z ′ can choose is W = {w1, w2, · · · , w|W |}. The central idea behind FiGAR is that the objective function used to update the parameters θaof πθa maintained by Z will be used to update the parameters θx of the action repetition policy πθx of Z
′ as well (illustrated by the sharing of L in Algorithm 1). In the first subsection, we desribe how Z ′ operates. In the next two sub-sections, we describe the instantiations of FiGAR extensions for 3 policy gradient DRL algorithms: A3C, TRPO and DDPG.
4.1 HOW FIGAR OPERATES
The following procedure describes how FiGAR variant Z ′ navigates the MDP that it is solving:
1. In the very first state s0 seen by Z ′, it predicts a tuple (a0, x0) of action to execute and number of time steps for which to execute it. a0 is decided based on πθa(s0) whereas x0 is decided based on πθx(s0). Each such tuple is known as an action decision.
2. We denote by sj the state of the agent after j such action decisions have been made. Similarly xj and aj denote the action repetition and the action chosen after j such action decisions. Note that xj ∈ {w1, w2, · · · , w|W |}, the set of all allowed action repetitions.
3. From time step 0 until x0 , Z ′ executes a0.
4. At time step x0, Z ′ again decides, based on current state s1 and policy components (πθa(s1), πθx(s1)), the tuple of action to execute and the number of times for which to execute it, (a1, x1).
5. It can seen that in general if Z ′ executes action ak for xk successive time steps, the next action is decided at time step t = k∑ i=0 xi on the basis of (πθa(sk+1), πθx(sk+1)), where sk+1 is the state
seen at time step t.
4.2 FIGAR-A3C
A3C uses fθa(sj) and fθc(sj) which represent the policy π(a|sj) and the value function V (sj) respectively. π(a|sj) is a vector of size equal to the action space of the underlying MDP while V (sj) is a scalar. FiGAR extends the A3C algorithm as follows:
1. With sj defined as in the previous sub-section, in addition to fθa(sj) and fθc(sj) , FiGARA3C defines a neural network fθx(sj). This neural network outputs a |W |-dimensional vector representing the probability distribution over the elements of the set W . The sampled time scale from this multinomial distribution decides how long the action decided with fθa(sj) is repeated. The actor is now composed of both fθa(sj) (action network) and fθx(sj) (repetition network).
2. The objective function for the actor is modified to be: L(θa, θx) = (log fθa(a|sj) + log fθx(x|sj))A(sj , a, x)
where A(sj , a, x) represents the advantage of executing action a for x time steps at state sj . This implies that for FiGAR-A3C the combination operator ∗ defined in Algorithm 1 is in fact scalar addition.
3. The objective function for the critic is the same except that estimated value function used in the target for the critic is changed as:
V̂ (sj) = n−1∑ k=j γyk−jrk + γ yn−jV (sn)
where we define y0 = 0, yk = yk−1 + xk, k ≥ 1 and action ak was repeated xk times when state sk was encountered. Note that the return used in target is based on n decision steps, steps at which a potential change in actions executed takes place. It is not based on n time steps.
Note that point 2 above implies that the action space has been extended by |W | and has a dimension of |A|+ |W |. It is only because of this factored representation of the FiGAR policy that the number of parameters do not blow up. If one were to extend the action space in a naive way by coupling the actions and the action repetitions, one would end up suffering the kind of action-space blowup as seen in [Lakshminarayanan et al. (2017); Vezhnevets et al. (2016)] wherein for being able to control with respect to |W | different action repetition levels (or |W |-length policy plans in the case of STRAW) , one would need to model |A|× |W | actions or action-values which would blow up the final layer size |W | times.
4.3 FIGAR-TRPO
Although fθa(sj) in A3C is generic enough to output continuous or discrete actions, we consider A3C only for discrete action spaces. Preserving the notation from the previous subsection, we describe FiGAR-TRPO where we consider the case of the output generated by the network fθa(sj) to be A dimensional with each dimension being independent and describing a continuous valued action. The stochastic policy is hence modeled as a multi-variate Gaussian with diagonal co-variance matrix. The parameters of the mean as well as the co-variance matrix are together represented by θa and the concatenated mean-covariance vector is represented by the function fθa(sj). FiGAR-TRPO is constructed as follows:
1. In TRPO,the objective function Lθold(θ̃) is constructed based on trajectories drawn according to the current policy. Hence, for FiGAR-TRPO the objective function is modified to be:
Lθa,old,θx,old(θ̃a)× ( Lθa,old,θx,old(θ̃x) )βar where θx are the parameters of sub-network fθx which computes the action repetition distribution. This implies that for FiGAR-TRPO the combination operator ∗ defined in Algorithm 1 is in some sense the scalar multiplication. βar controls the relative learning rate of the core-policy parameters and the action repetition parameters.
2. The constraint in TRPO corresponding to the KL divergence between old and new policies is modified to be:
DmaxKL (πa, π̃a) + βKLD max KL (πx, π̃x) ≤ δ
where πa denotes the Gaussian distribution for the action to be executed and πx denotes the multinomial softmax-based action repetition probability distribution. βKL controls the relative divergence of πx and πa from the new corresponding policies. See Appendix C for an explanation of the loss function used.
4.4 FIGAR-DDPG
In this subsection, we present an extension of DDPG under the FiGAR framework. DDPG consists of fθa(sj) which denotes a deterministic policy µ(s) and is a vector of size equal to the action space of the underlying MDP; and fθc(sj , aj) which denotes the critic network whose output is a single number, the estimated state-action value function Q(sj , aj). FiGAR framework extends the DDPG algorithm as follows:
1. fθx is introduced, similar to FiGAR-A3C. This implies that the complete policy for FiGARDDPG (πθa , πθx) is computed by the tuple of neural networks: (fθa , fθx) . Similar to DDPG [Lillicrap et al. (2015)], FiGAR-DDPG has no loss function for the actor. The actor receives gradients from the critic. This is because the actors proposed policy is directly fed to the critic and the critic provides the actor with gradients which the proposed policy follows for improvement. In FiGAR-DDPG the total policy π is a concatenation of vectors πa and πx. Hence the gradients for the total policy are also simply the concatenation of the gradients for the policies πa and πx.
2. To ensure sufficient exploration, the exploration policy for action repetition is an -greedy version of the behavioral action repetition policy. The action part of the policy, (fθa(sj)), continues to use temporally correlated noise for exploration, generated by an Ornstein-Uhlenbeck process (see Lillicrap et al. (2015) for details).
3. The critic is modeled by the equation f(sj , aj , xj) = fθc(sj , fθa(sj), fθx(sj))
As stated above, fθx is learnt by back-propagating the gradients produced by the critic with respect to fθx , in exactly the same way that fθa is learnt.
5 EXPERIMENTAL SETUP AND RESULTS
The experiments are designed to understand the answers to the following questions:
1. For different DRL algorithms, can FiGAR extensions learn to use the dynamic action repetition? 2. How does FiGAR impact the performance of the different algorithms on various tasks? 3. Is FiGAR able to learn control on several different kinds of Action Repetition sets W ?
In the next three sub-sections, we experiment with the simplest possible action repetition set W = {1, 2, · · · , |W |}. In the fourth sub-section, we understand the effects that changing the action repetition set W has on the policies learnt.
5.1 FIGAR-A3C ON ATARI 2600
This set of experiments was performed with FiGAR-A3C on the Atari 2600 domain. The hyperparameters were tuned on a subset of games (Beamrider, Breakout, Pong, Seaquest and Space Invaders) and kept constant across all games.
W is perhaps the most important hyper-parameter and depicts our confidence in the ability of a DRL agent to predict the future. Such a choice has to depend on the domain in which the DRL agent is operating. We only wanted to demonstrate the ability of FiGAR to learn temporal abstractions and hence instead of tuning for an optimal |W |, it was chosen to be 30, arbitrarily. The specific set of time scales we choose is 1, 2, 3, · · · , 30. FiGAR-A3C as well as A3C were trained for 100 million decision steps. They were evaluated in terms of the final policy learnt. Treating the score obtained by the A3C algorithm as baseline (b), we calculated the percentage improvement (i) offered by FiGARA3C (f) as: i = f−bb . Figure 2 plots this metric versus the game names. The improvement for Enduro and Atlantis is staggering and more than 900× and 35× respectively. Figure 2’s y-axis has been clipped at 1000% to make it more presentable. Appendix A contains the experimental details, the raw scores obtained by both the methods. Appendix B contains experiments on validating our setup.
To answer the first question we posed, experiments were conducted to record the percentage of times that a particular action repetition was chosen. Figure 3 presents the action repetition distribution across a selection of games, chosen arbitrarily. The values have been rounded to 2 decimal places and hence do not sum to 1 in each game. Each game was played for 10 episodes using the same policy used to calculate average scores in Figure 2. The two tables together show that FiGAR-A3C generally prefers lower action repetition but does come up with temporal abstractions in policy space (specially in games like Pong and Crazy Climber). Some such abstractions have been demonstrated in Figure 1. Such temporal abstractions do not always help general gameplay (Demon Attack). However, as can be seen from Figure 2, FiGAR-A3C outperforms A3C in 26 out of 33 games. One could potentially think of FiGAR as a deep exploration framework by using the learnt policy πθa for predicting actions at every time step and completely discarding the action-repetition policy
πθx , at evaluation time. Appendix F contains an empirical argument against such a usage of FiGAR and demonstrates that the temporal abstractions encoded by πθx are indeed important for game play performance.
5.2 FIGAR-TRPO ON MUJOCO TASKS
In this sub-section we demonstrate that FiGAR-TRPO can learn to solve the Mujoco simulated physics tasks reasonably successfully. Similar to FiGAR-A3C, |W | is chosen to be 30 arbitrarily.
The full policy (fθa , fθx) is trained jointly. The policies learnt after each TRPO optimization step (details in Appendix C) are compared to current best known policy to arrive at the overall best policy. The results in this sub-section are for this best policy. Table 1 compares the performance of TRPO and FiGAR-TRPO. The number in the brackets is the average action repetition chosen. As can be seen from the table, FiGAR learns either policies which are much faster to execute albeit at cost of slight loss in optimality or it learns policies similar to non-repetition case, performance being competitive with the baseline algorithm. This best policy was then evaluated on 100 episodes to arrive at average scores which are contained in Table 1. TRPO is a difficult baseline on the MuJoCo tasks domain. On the whole, FiGAR outperforms TRPO in 3 out of 5 domains, although the gains are marginal in most tasks. Appendix C contains experimental details. A video showing FiGAR-TRPO’s learned behavior policies can be found at http://youtu.be/JiaO2tBtH-k.
5.3 FIGAR-DDPG ON TORCS
FiGAR-DDPG was trained and tested on the TORCS domain. |W | was chosen to be 15 arbitrarily. FIGAR-DDPG manages to complete the race task flawlessly and manages to finish 20 laps of the circuit, after which the simulator stops. The total reward obtained by FiGAR-DDPG was 557929.68 as against 59519.70 obtained by DDPG. We also observed that FiGAR-DDPG learnt policies which were smoother than those learnt by DDPG. A video showing the learned driving behavior of the FiGAR-DDPG agent can be found at https://youtu.be/dX8J-sF-WX4. See Appendix D for experimental and architectural details.
5.4 EFFECT OF ACTION REPETITION SET ON FIGAR
This sub-section answers the third question raised at the beginning of this section in affirmative. We demonstrate that there is nothing sacrosanct about the set of action repetitions W = {1, 2, · · · , 30} on which FiGAR-A3C performed well, and that the good performance carries over to other action repetition sets.
To demonstrate the generality of FiGAR with respect to W , we chose a wide variety of action repetition sets W , trained and evaluated FiGAR-A3C variants which learn to repeat with respect to their respective Action Repetition sets. Table 3 describes the various FiGAR-variants considered for these experiments in terms of their action repetition set W .
Note that the hyper-parameters of the various variants of FiGAR-A3C were not tuned but rather the same ones obtained by tuning for FiGAR-30 were used. Table 2 contains a comparison of the raw scores obtained by the various FiGAR-A3C variants in comparison to the A3C baseline. It is clear that FiGAR is able to learn over any action repetition set W and the performance does not fall by a lot even when hyper-parameters tuned for FiGAR-30 are used for other variants. Appendix E
contains additional graphs showing the evolution of average game scores against number of training steps as well as a bar graph visualization of Table 2.
6 CONCLUSION, SHORTCOMINGS AND FUTURE WORK
We propose a light-weight framework (FiGAR) for improving current Deep Reinforcement Learning algorithms for policy optimization whereby temporal abstractions are learned in the policy space. The framework is generic and applicable to DRL algorithms concerned with policy gradients for continuous as well as discrete action spaces such as A3C, TRPO and DDPG. FiGAR maintains a structured policy wherein the action probability distribution is augmented with a probability distribution for choosing the time scale of repeating the chosen action. Our results demonstrate that FiGAR can be used to significantly improve the current policy gradient and Actor-Critic algorithms thereby learning better control policies across several domains by discovering optimal sequences of temporally elongated macro-actions.
Atari, TORCS and MuJoCo represent environments which are largely deterministic with a minimal degree of stochasticity in environment dynamics. In such highly deterministic environments we would expect FiGAR agents to build a latent model of the environment dynamics and hence be able to execute large action repetitions without dying. This is exactly what we see in a highly deterministic environment like the game Freeway. Figure 1 (a) demonstrates that the chicken is able to judge the speed of the approaching cars appropriately and cross the road in a manner which takes it to the goal without colliding with the cars and at the same time avoiding them narrowly.
Having said that, certainly the ability to stop an action repetition (or a macro-action) in general would be very important, especially in stochastic environments. In our setup, we do not consider the ability to stop executing a macro-action that the agent has committed to. However, this is a necessary skill in the event of unexpected changes in the environment while executing a chosen macro-action. Thus, stop and start actions for stopping and committing to macro-actions can be added to the basic dynamic time scale setup for more robust policies. We believe the modification could work for more general stochastic worlds like Minecraft and leave it for future work.
ACKNOWLEDGMENTS
We used the open source implementation of A3C at https://github.com/miyosuda/ async_deep_reinforce. We thank Volodymr Mnih for giving valuable hyper-parameter information. We thank Aravind Rajeswaran (University of Washington) for very helpful discussions regarding and feedback on the MuJoCo domain tasks. The TRPO implementation was a modification of https://github.com/aravindr93/robustRL. The DDPG implementation was a modification of https://github.com/yanpanlau/DDPG-Keras-Torcs. We thank ILDS (http://web.iitm.ac.in/ilds/) for the compute resources we used for running A3C experiments.
APPENDIX A: EXPERIMENTAL DETAILS FOR FIGAR-A3C
EXPERIMENTAL DETAILS AND RESULTS
We used the LSTM-variant of A3C [Mnih et al. (2016)] algorithm for FiGAR-A3C experiments. The async-rmsprop algorithm [Mnih et al. (2016)] was used for updating parameters with the same hyper-parameters as in Mnih et al. (2016). The initial learning rate used was 10−3 and it was linearly annealed to 0 over 100 million steps. The n used in n-step returns was 20. Entropy regularization was used to encourage exploration, similar to Mnih et al. (2016). The β for entropy regularization was found to be 0.02 after hyper-parameter tuning, both for the action-policy fθa and the action repetition policy fθx .
Since the Atari 2600 games tend to be quite complex, jointly learning a factored policy from random weight initializations proved to be less optimal as compared to a more stage-wise approach. The approach we followed for training FiGAR-A3C was to first train the networks using the regular A3C-objective function. This stage trains the action part of the policy fθa and value function fθc for a small number of iterations with a fixed action repetition rate (in this stage, gradients are not back-propagated for fθx and all action repetition predictions made are discarded). The next stage was to then train the entire architecture (fθa , fθx , fθc) jointly. This kind of a non-stationary training objective ensures that we have a good value function estimator fθc and a good action policy
estimator fθa before we start training the full policy (fθa , fθx) jointly. Every time FiGAR decides to execute action at for xt time steps, we say one step of action selection has been made. Since the number of time steps for which an action is repeated is variable, training time is measured in terms of action selections carried out. The first stage of the training was executed for 20 million (a hyper-parameter we found by doing grid search) action selections (called steps here onwards) and the next stage was executed for 80 million steps. In comparison the baseline ran for 100 million steps (action selections). Since a large entropy regularization was required to explore both components (fa and fx) of the policy-space, this also ends up meaning that the policies learnt are more diffused than one would like them to be. Evaluation was done after every 1 million steps and followed a strategy similar to -greedy. With = 0.1 probability, the action and action repetition was drawn from the output distribution ((fθa and fθx respectively) and with probability 1 − the action (and independently the action selection) with maximum probability was selected. This evaluation was done for 100 episodes or 100000 steps whichever was smaller, to arrive at an average score. Table 4 contains the raw scores obtained by the final FiGAR-A3C and A3C policies on 33 Atari 2600 games. The numbers inside the brackets depict the confidence interval at a confidence threshold of 0.95, calculated by averaging scores over 100 episodes. Table 5 contains scores for a competing method, STRAW [Vezhnevets et al. (2016)], which learns temporal abstractions by maintaining action plans, for the subset of games on which both FiGAR and STRAW were trained and tested. Note that the scores obtained by STRAW agents are averages over top 5 performing replicas. We can infer from Tables 4 and 5 that FiGAR and STRAW and competitive with each other, with FiGAR clearly out-performing STRAW in Breakout and STRAW clearing outperforming FiGAR in Frostbite.
Figure 4 demonstrates the evolution of the performance of FiGAR-A3C versus training progress. It also contains corresponding metrics for A3C to facilitate comparisons. In the 100 episode long evaluation phase we also keep track of the best episodic score. We also plot the best episode’s score versus time to get an idea of how bad the learnt policy is compared to the best it could have been.
ARCHITECTURE DETAILS
We used the same low level architecture as Mnih et al. (2016) which in turn uses the same low level architecture as Mnih et al. (2015), except that the pre-LSTM hidden layer had size 256 instead of 512 as in Mnih et al. (2016). Similar to Mnih et al. (2016) the Actor and Critic share all but one layer. Hence all but the final layer of fθa , fθx and fθc are the same. Each of the 3 networks has a different final layer with fθa and fθx having a softmax-non linearity as output non-linearity, to model the multinomial distribution and the fθc (critic)’s output being linear.
APPENDIX B: ADDITIONAL EXPERIMENTS FOR ATARI 2600
These additional experiments are geared at understanding the repercussions of the evaluation strategy chosen by us.
THE CHOICE OF WHETHER TO BE GREEDY OR STOCHASTIC
Note that in Appendix A, we state that for evaluating the policy learnt by the agent, we simply chose to sample from the output probability distributions with probability 0.1 and chose the optimal action/action repetition with probability 0.9. This choice of 0.1 might seem rather arbitrary. Hence we conducted experiments to understand how well the agent performs as we shift more and more from choosing the maximal action(0.1-greedy policy) towards sampling from output distributions (stochastic policy).
Figure 5 demonstrates that the performance of FiGAR-A3C does not deteriorate significantly, in comparison to A3C, even if we always sample from policy distributions, for most of the games. In the cases that there is a significant deterioration, we believe it is due to the diffused nature of the policy distributions (action and action repetition) learnt. Hence, although our choice of evaluation scheme might seem arbitrary, it is in fact reasonable.
PERFORMANCE VERSUS SPEED TRADEOFF
The previous discussion leads to a novel way to trade-off game-play performance versus speed. Figure 3 demonstrated that although FiGAR-A3C learns to use temporally elongated macro-actions, it does favor shorter actions for many games. Since the action repetition distribution πθx is diffused (as will be shown by Table 6), sampling from the distribution should help FiGAR choose larger action repetition rates probably at the cost of optimality of game play.
Table 6 demonstrates that this is exactly what FiGAR does. It was generated by playing 10 episodes, or 100000 steps, whichever is lesser and recording the fraction of times each action repetition was chosen. The policy used in populating table 6 was the stochastic policy (described in previous subsection). Contrast Table 6 to Table 7 which is an expanded version of Figure 3.
Both Figure 3 and Table 7 were created using the 0.1-greedy policy described in previous subsection. The reason that we compare the stochastic policy with the 0.1-greedy version instead of the fully-greedy version (wherein the optimal action and action repetition is always chosen) is that such a policy would end up being deterministic would not be good for evaluations. It can hence be seen that FiGAR learns to trade-off optimality of game-play for speed by choosing whether to sample from policy probability distributions (πθa and πθx ) with probability 1 and thus behave stochastically, or behave 0.1-greedily, and sample from the distributions with only a small probability. Table 6 can be compared to Figure 3 to understand how stochasticity in final policy affects action repetition chosen. A clear trend can be seen in all games wherein the stochastic variant of final policy learns to use longer and longer actions, albeit at a small cost of some loss in the optimality of game-play (as shown by Figure 5).
An expanded version of Figure 3 is presented as Table 7 for comparison with Table 6. As explained in Appendix A, the policy used for populating Table 7 is such that it picks a greedy action (or action repetition) with probability 0.9 and stochastically samples from output probability distributions with probability 0.1.
Table 8 contains the average action repetition chosen in each of the games for the two FiGARvariants. The same episodes used to populate Table 6 and 7 were used to fill Table 8. It can be seen that in most games, the Stochastic variant of policy learns to play at a higher speed, although this might result in some loss in optimality of game play, as demonstrated in Figure 5.
APPENDIX C: EXPERIMENTAL SETUP FOR FIGAR-TRPO
EXPERIMENTAL DETAILS
FiGAR-TRPO and the corresponding baseline algorithm operate on low dimensional feature vector observations. The TRPO (and hence FiGAR-TRPO) algorithm operates in two phases. In the first phase (P1), K trajectories are sampled according to current behavioral policy π to create the surrogate loss function. In the second phase (P2) a policy improvement step is performed by carrying out an optimization step on the surrogate loss function, subject to the KL-divergence constraint on the new policy. In our experiments, 500 such policy improvement steps were performed. K varies with the learning progress and the schedule on what value K would take in next iteration of P1 is defined linearly in terms of the return in the last iteration of P1. Hence if the return was large in previous iteration of P1, a small number of episodes are are used to construct the surrogate loss function in current iteration. The best policy was found by keeping track of the average returns seen during the training phase P1. This policy was then evaluated on 100 episodes to obtain the average score of the TRPO policy learnt. The most important hyper-parameters for FiGAR-TRPO are βar and βKL. By using a grid search on the set {0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28} we found the optimal hyper-parameters βar = 1.28 and βKL = 0.64. These were tuned on all the 5 tasks.
LOSS FUNCTION AND ARCHITECTURE
The tanh non-linearity is used throughout. The mean vector is realized using a 2-Hidden Layer neural network (mean network) with hidden layer sizes (128, 64). The standard deviation is realized using a Parameter layer (std-dev layer) which parameterizes the standard deviation but does not depend on the input. Hence the concatenation of the output of mean network and the std-dev layer forms the action policy fθa as described in Section 4. The Action Repetition function fθx is realized using a 2-Hidden Layer neural (act-rep network) network similar to the mean network albeit with smaller hidden layer sizes: (128, 64). However, its output non-linearity is a softmax layer of size 30 as dictated by the value of W . The action repetition network was kept small to ensure that FiGAR-TRPO does not have significantly more parameters than TRPO. The mean network, std-dev layer and act-rep network do not share any parameters or layers (See appendix G for experiments on FiGAR-TRPO with shared layers).
The surrogate loss function in TRPO when the Single Path method of construction is followed reduces to [Schulman et al. (2015)]:
Lθold(θ̃) = Es∼ρθold,a∼πθold [ πθ̃(a|s) πθold(a|s) Qθold(s, a) ] where q, the sampling distribution is just the old behavioral policy πθold (defining characteristic of Single-Path method) and ρ is the improper discounted state visitation distribution.
The surrogate loss function for a factored policy such as that of FiGAR-TRPO is:
Lθa,old,θx,old(θa, θx) = Es,a,x [ πθa(a|s) πθa,old(a|s) πθx(x|s) πθx,old(x|s) Qθa,old,θx,old(s, a, x) ] where s ∼ ρθa,θxold , a ∼ πθa,old , x ∼ πθx,old and πθa = fθa , πθa,old = fθa,old , πθx = fθx and πθx,old = fθx,old
This kind of a splitting of probability distributions happens because the action-policy fθa and the action-repetition policy fθx are independent probability distributions. The theoretically sound way to realize FiGAR-TRPO is to minimize the loss Lθa,old,θx,old(θa, θx). However, we found that in practice, optimizing a relaxed version of the objective function, that is,
Lθa,old,θx,old(θ̃a)× Lθa,old,θx,old(θ̃x)βar
works better. This leads to the FiGAR-TRPO objective defined in Section 4.3.
APPENDIX D: EXPERIMENTAL DETAILS FOR FIGAR-DDPG
EXPERIMENTAL DETAILS
The DDPG algorithm also operates on the low-dimensional (29 dimensional) feature-vector observations. The domain consists of 3 continuous actions, acceleration, break and steering. The W hyper-parameter used in main experiments was chosen to be 15 arbitrarily. Unlike Lillicrap et al. (2015), we did not find it useful to use batch normalization and hence it was not used. However, a replay memory was used of size 10000. Target networks were also used with soft updates being applied with τ = 0.001. Sine DDPG is an off-policy actor-critic method, we need to ensure that sufficient exploration takes place. Use of an Ornstein-Uhlenbeck process (refer to Lillicrap et al. (2015) for details) ensured that exporation was carried out in action-policy space. To ensure exploration in the action-repetition policy space, we adopted two strategies. First, an -greedy version of the policy was used during train time. The was annealed from 0.2 to 0 over 50000 training steps. The algorithm was run for 40000 training steps for baselines as well as FiGAR-DDPG. Second, with probability 1− , instead of picking the greedy action-repetition , we sampled from the output distribution fθx(s).
ARCHITECTURAL DETAILS
Through the architecture, the hidden layer non-linearity used was ReLU. All hidden layer weights were initialized using the He initialization [He et al. (2015)] The actor network consisted of a 2-hidden layer neural network with hidden sizes (300, 600) (call the second hidden layer representation h2. We learn two different output layers on top of this common hidden representation. fθa was realized by transforming h2 with an output layer of size 3. The output neuron corresponding to the action steering used tanh non linearity where as those corresponding to acceleration and break used the sigmoid non-linearity. The fθx network was realized by transforming h2 using a softmax output layer of size |W |. The output of the Actor network is a 3 + |W | = 18 dimensional vector. The critic network takes as input the state vector (29-dimensional) and the action vector (18- dimensional). The critic is a 3 hidden layer network of size (300, 600, 600). Similar to Lillicrap et al. (2015), actions were not included until the 2nd hidden layer of fθc . The final output is linear and is trained using the TD-error objective function, similar to Lillicrap et al. (2015)
APPENDIX E: DETAILS FOR FIGAR-VARIANTS
It is clear from Figure 6 that even though FiGAR A3C needs to explore in 2 separate action-spaces (those of primitive actions and the action repetitions), the training progress is not slowed down as a result of this exploration, for any FiGAR variant.
Table 2 contains final evaluation scores attained by various FiGAR variants. Figure 7 contains a bargraph visualization of the same table to demonstrate the advantage of all FiGAR variants relative to the baselines.
APPENDIX F: IMPORTANCE OF πθx
One could potentially use FiGAR at evaluation stage (after training has been completed) at an actionrepetition rate of 1 by picking every action according to πθa and completely discarding the learnt repetition policy πθx . Such a FiGAR variant is denoted as FiGAR-wo-πθx . We demonstrate that FiGAR-wo-πθx is worse than FiGAR on most games and hence the temporal abstractions learnt by and encoded in πθx are indeed non-trivial and important for gameplay performance. Table 9 contains the comparison between standard FiGAR agent and FiGAR-wo-πθx . Evaluation scheme is the same as Appendix A.
We observe that in 24 out of 33 games, πθx helps the agent learn temporal abstractions which result in a significant boost in performance compared to the FiGAR-wo-πθx agents.
APPENDIX G: SHARED REPRESENTATION EXPERIMENTS FOR FIGAR-TRPO
Section 5.2 contains results of experiments on FiGAR-TRPO. Appendix C contains the experimental setup for the same. Throughout these experiments on FiGAR-TRPO the policy components fθa and fθx do not share any representations. This appendix contains experimental results in the setting wherein (fθa and fθx) share all layers except the final one. This agent/network is denoted with the name FiGAR-shared-TRPO. All the hyper-parameters are the same as those in Appendix C except βar and βKL which were obtained through a grid-search similar to appendix C. These were tuned on all the 5 tasks. The values for these hyper-parameters that we found to be optimal are βar = 1.28 and βKL = 0.16. The same training and evaluation regime as appendix C was used. The performance of the best policy learnt is tabulated in Table 10
FiGAR-shared-TRPO on the whole does not perform much better than FiGAR-TRPO. In these TRPO experiments, the neural networks we used were rather shallow at only two hidden layers deep. Hence, we believe that sharing of layers thus leads to only small gains in terms of optimality of policy learnt.

This paper proposes a simple but effective extension to reinforcement learning algorithms, by adding a temporal repetition component as part of the action space, enabling the policy to select how long to repeat the chosen action for. The extension applies to all reinforcement learning algorithms, including both discrete and continuous domains, as it is primarily changing the action parametrization. The paper is well-written, and the experiments extensively evaluate the approach with 3 different RL algorithms in 3 different domains (Atari, MuJoCo, and TORCS).
Here are some comments and questions, for improving the paper:
The introduction states that "all DRL algorithms repeatedly execute a chosen action for a fixed number of time steps k". This statement is too strong, and is actually disproved in the experiments — repeating an action is helpful in many tasks, but not in all tasks. The sentence should be rephrased to be more precise.
In the related work, a discussion of the relation to semi-MDPs would be useful to help the reader better understand the approach and how it compares and differs (e.g. the response from the pre-review questions)
Experiments:
Can you provide error bars on the experimental results? (from running multiple random seeds)
It would be useful to see experiments with parameter sharing in the TRPO experiments, to be more consistent with the other domains, especially since it seems that the improvement in the TRPO experiments is smaller than that of the other two domains. Right now, it is hard to tell if the smaller improvement is because of the nature of the task, because of the lack of parameter sharing, or something else.
The TRPO evaluation is different from the results reported in Duan et al. ICML ’16. Why not use the same benchmark?
Videos only show the policies learned with FiGAR, which are uninformative without also seeing the policies learned without FiGAR. Can you also include videos of the policies learned without FiGAR, as a comparison point?
How many laps does DDPG complete without FiGAR? The difference in reward achieved seems quite substantial (557K vs. 59K).
Can the tables be visualized as histograms? This seems like it would more effectively and efficiently communicate the results.
Minor comments:
-- On the plot in Figure 2, the label for the first bar should be changed from 1000 to 3500.
-- “idea of deciding when necessary” - seems like it would be better to say “idea of only deciding when necessary"
-- "spaces.Durugkar et al.” — missing a space.
-- “R={4}” — why 4? Could you use a letter to indicate a constant instead? (or a different notation)


Deep neural networks have recently demonstrated excellent performance on various tasks. Despite recent advances, our understanding of these learning models is still incomplete, at least, as their unexpected vulnerability to imperceptibly small, non-random perturbations revealed. The existence of these so-called adversarial examples presents a serious problem of the application of vulnerable machine learning models. In this paper, we introduce the layerwise origin-target synthesis (LOTS) that can serve multiple purposes. First, we can use it as a visualization technique that gives us insights into the function of any intermediate feature layer by showing the notion of a particular input in deep neural networks. Second, our approach can be applied to assess the invariance of the learned features captured at any layer with respect to the class of the particular input. Finally, we can also utilize LOTS as a general way of producing a vast amount of diverse adversarial examples that can be used for training to further improve the robustness of machine learning models and their performance as well.
Deep neural networks have recently demonstrated excellent performance on various tasks. Despite recent advances, our understanding of these learning models is still incomplete, at least, as their unexpected vulnerability to imperceptibly small, non-random perturbations revealed. The existence of these so-called adversarial examples presents a serious problem of the application of vulnerable machine learning models. In this paper, we introduce the layerwise origin-target synthesis (LOTS) that can serve multiple purposes. First, we can use it as a visualization technique that gives us insights into the function of any intermediate feature layer by showing the notion of a particular input in deep neural networks. Second, our approach can be applied to assess the invariance of the learned features captured at any layer with respect to the class of the particular input. Finally, we can also utilize LOTS as a general way of producing a vast amount of diverse adversarial examples that can be used for training to further improve the robustness of machine learning models and their performance as well.
1 INTRODUCTION
Due to tremendous progress over the last several years, the most advanced deep neural networks (DNNs) have managed to approach and even surpass human level performance on a wide range of challenging machine learning tasks (Parkhi et al., 2015; Schroff et al., 2015; Szegedy et al., 2015; He et al., 2016). Despite the fact that we are able to design and train learning models that perform well, our understanding of these complex networks is still incomplete. This was highlighted by the discovery of the intriguing properties of machine learning models by Szegedy et al. (2014).
Gaining intuitive insights and, thus, building a better understanding of how these models work has a long history in the literature. For visual recognition tasks, various techniques have been proposed to address the problem of understanding what kind of features are captured and used by learning models (Erhan et al., 2009; Mahendran & Vedaldi, 2016) and how the different internal representations of object classes or input images can be better visualized (Zeiler & Fergus, 2014; Yosinski et al., 2015; Simonyan et al., 2014; Mahendran & Vedaldi, 2016).
While exploring the internal details of DNNs in order to further advance their performance via visualization is particularly relevant and the subject of this paper, recent research has also been focusing on the unpleasant properties revealed by Szegedy et al. (2014). Namely, machine learning models, including the best performing DNNs, suffer from highly unexpected instability as they can confidently misclassify adversarial examples that are formed by adding imperceptible, non-random perturbations to otherwise correctly classified inputs. As DNNs are expected to be robust to small perturbations to their inputs due to their excellent generalization capabilities, the existence of such adversarial perturbations challenges our understanding of DNNs, and questions the utility of such vulnerable learning models in real-world applications. Researchers proposed various techniques to reliably find adversarial perturbations (Szegedy et al., 2014; Goodfellow et al., 2015; Sabour et al., 2016; Rozsa et al., 2016), and demonstrated that adversarial examples can serve a good purpose as well, as they can be successfully used for training to improve both the overall performance and the robustness of machine learning models (Goodfellow et al., 2015; Rozsa et al., 2016).
In this paper, we introduce the layerwise origin-target synthesis (LOTS) that can be efficiently used for multiple purposes. First, it can be applied to visualize the internal representation of an input captured by the learning model at any particular layer. Second, LOTS is capable of forming a vast amount of diverse adversarial examples for each input and we show that such diversity can be beneficial for adversarial training. Finally, derived from the previous possible utilization, LOTS can also be used to explore and assess how stable a particular internal representation of an input is with respect to the class of the input. This paper introduces our novel approach, compares it to related work, and highlights its benefits and possible directions of future work.
2 RELATED WORK
While deep neural networks (DNNs) have achieved excellent performance on various tasks, there is an increasing interest to alleviate their vulnerability to adversarial examples. To better understand the sometimes unexpected behavior of learning models, visualizing the learned features is a common practice for gaining intuitive insights. As our work is related to the visualization of feature representations and adversarial instability, this section discusses the relationships to both areas.
2.1 VISUALIZATION
In order to design and train better performing machine learning models, it is useful to have a clearer understanding about the internal operations and behaviors of these complex models. Otherwise, research aiming to develop more advanced learning models remains restricted to trial-and-error.
Visualization of DNNs was pioneered by Erhan et al. (2009) who displayed high level features learned by various models at the unit level by finding the optimal stimulus in the image space that maximizes the activation of each particular unit. While their approach requires careful initialization, it cannot provide information about the invariance of the inspected units. To address this short-coming, Simonyan et al. (2014) demonstrated how image-specific class saliency maps can be obtained from the last fully connected layers of DNNs, showing areas of the image that are discriminative with respect to a given class. The authors also introduced a technique – image inverting – to generate artificial images for classes that maximize the selected class score. Related to saliency maps, Girshick et al. (2014) showed that identifying regions of images yielding high activations at higher layers can be successfully used for object detection. Zeiler & Fergus (2014) extended visualization to convolutional features. Furthermore, their approach not only finds patches of input images that stimulate a particular feature map, but is also capable of revealing structures within those patches by using a top-down projection.
Although these visualization techniques helped to gain insights into how and why DNNs might work, researchers proposed various approaches to further enhance the produced representations commonly called pre-images. Yosinski et al. (2015) visualized activations on each layer for an image or video, and introduced methods to produce qualitatively clearer and more interpretable visual representations. Similarly, Mahendran & Vedaldi (2016) presented regularized visualizations for maximized activations and inverted images in order to obtain natural looking pre-images.
Using LOTS we explore how the extracted features of an image at any given layer translate back to the input space, and how invariant those features are with respect to the class of a particular input.
2.2 ADVERSARIAL REPRESENTATIONS
Szegedy et al. (2014) revealed that machine learning models, including state-of-the-art DNNs, are vulnerable to adversarial examples that are formed by applying imperceptibly small, non-random perturbations to otherwise correctly recognized inputs leading to misclassifications. This discovery fundamentally challenged our understanding of DNNs, namely, the excellent performance achieved by these complex models was believed to be due to their capability of learning features from the training set that generalize well. The existence of adversarial examples highlights that machine learning models are in fact not robust, and adversarial instability needs to be addressed.
Since Szegedy et al. (2014) presented the problem and introduced the first method that is able to reliably find adversarial perturbations, various approaches were proposed in the literature. Compared to the computationally expensive box-constrained optimization technique (L-BFGS) of Szegedy et al.
(2014), a more lightweight, yet effective technique was introduced by Goodfellow et al. (2015). The proposed fast gradient sign (FGS) method relies on the sign of the gradient of loss which needs to be calculated only once in order to form an adversarial perturbation. The authors also demonstrated that by using FGS examples implicitly in an enhanced objective function, both the overall performance and the robustness of the trained models can be improved.
Both of the aforementioned adversarial example generation techniques rely on ascending the gradient of loss, namely, the formed perturbation causes misclassification by increasing the loss until the particular original class does not have the highest prediction probability. Non-gradient based methods were also proposed by researchers. The approach of Sabour et al. (2016) produces adversarial images that not only cause misclassifications but also mimic the internal representations of the targeted original inputs. Their technique also uses the computationally expensive L-BFGS optimization algorithm. Rozsa et al. (2016) introduced the non-gradient based hot/cold approach, which causes recognition errors by not only reducing the prediction probability of the original class of the input, but by aiming to magnify the probability of the specified targeted class. Therefore, this approach is capable of producing multiple adversarial examples for each input. The authors demonstrated that using a diverse set of such adversarial examples formed with perturbations with higher magnitude than the sufficient minimum necessary to cause misclassifications can outperform regular adversarial training. Finally, Rozsa et al. (2016) proposed a new psychometric called perceptual adversarial similarity score (PASS) to better measure adversarial quality, in other words, the distinguishability or similarity of original and adversarial image pairs in terms of human perception.
Our novel LOTS method can be considered as a general extension of the hot/cold approach to deeper layers, and it also shows similarities to the technique of Sabour et al. (2016) in terms of directly adjusting internal feature representations – without requiring the use of the L-BFGS algorithm.
3 APPROACH
One way of gaining insights into the operation of a deep neural network is by letting the network process a given image and – after modifying the output of the network – projecting this modification back to the input level, e.g, via backpropagation. While changing the output appears to be straightforward due to our clear(er) understanding of its meaning, e.g., the hot/cold approach (Rozsa et al., 2016) modifying logits, theoretically, the modification can happen at any layer or any neuron of the network. However, the interpretation of such modifications might be more difficult as the output of a given layer or neuron per se is not guaranteed to have a semantic meaning.
Formally, let us consider a network fw with weights w in a layered structure, i.e., having layers y(l), l = {1, . . . , L}, with their respective weights w(l). For a given input x, the output of the network can be formalized as:
fw(x) = y (L) ( y(L−1) ( . . . ( y(1)(x) ) . . . )) , (1)
while the internal representation of the given input x at layer l is:
f (l)w (x) = y (l)
( y(l−1) ( . . . ( y(1)(x) ) . . . )) . (2)
Our layerwise origin-target synthesis (LOTS) approach aims to adjust the internal representation of an input xo, the origin, to get closer to the internal representation associated with input xt, the target. We modify the internal feature representation of xo at a given layer l to step away from the origin and, in parallel, get closer to the target xt, and project this feature difference back to the input level. To do so, LOTS uses backpropagation operator Bl to estimate input changes on xo accordingly:
η(l)(xo, xt) = Bl ( f (l)w (xt)− f (l)w (xo) ) . (3)
We can use the direction defined by the backpropagated feature difference and form adversarial perturbations using a line-search – similar to the fast gradient sign (FGS) method (Goodfellow et al., 2015) and the hot/cold approach (Rozsa et al., 2016). Compared to previous techniques, LOTS has the potential to form dramatically more, and more diverse perturbations for each input due to the large amount of possible targets and the number of layers it can be used on. Also, LOTS is capable of attacking networks that extract deep features rather than doing classification.
Alternatively, LOTS can be used by targeting the origin itself, and magnifying or reducing the internal representation of input xo at layer l. This can be formalized as:
η̂(l)(xo) = Bl ( ±f (l)w (xo) ) . (4)
While this direction can be utilized to form adversarial perturbations, it can also be used to visualize the captured internal representations of the input at layer l. Furthermore, by exploring the magnitudes of perturbations necessary to cause misclassifications, we can assess the robustness of inspected layers with respect to the class of a particular input.
4 VISUALIZATION VIA LOTS
First, we focus on demonstrating the visualization capabilities of LOTS, exploring the captured internal representations of inputs on different DNNs. We investigate two publicly available deep neural networks: the 4-layer LeNet network from LeCun et al. (1998) trained on the MNIST dataset (LeCun et al., 1995), and the 16-layer VGG face recognition network from Parkhi et al. (2015).
To quantify the perturbed images, we use three metrics: L2 and L∞ norms of the perturbations, as well as the perceptual adversarial similarity score (PASS) (Rozsa et al., 2016) between origin xo and the perturbed image x±o . While L2 and L∞ norms focus strictly on the perturbations regardless of how visible or hidden those are on the distorted images, PASS was designed to quantify the similarity of original and perturbed image pairs with respect to human perception. Therefore, PASS is more suitable to measure adversarial quality with PASS=1 indicating perfect similarity. Note that throughout our experiments, we form perturbed images that have discrete pixel values in [0, 255].
4.1 INTERNAL REPRESENTATION OF HANDWRITTEN DIGITS
To visualize the internal representations of layers in LeNet, we apply Equation (4) on each layer. When we use the positive sign to form a perturbed image x+o , the captured internal representation is magnified and the classification of the modified digit will usually not change. On the other hand, by using the negative sign for forming an image x−o , we can display which kind of modifications would be required to inhibit the internal representation of origin xo at that particular layer. Consequently, by increasing the magnitude of the perturbation, the network is more likely to classify x−o differently than the origin xo as feature representations of the original class are fading away.
Figure 1 displays the visualized internal representations of an MNIST image with label 4, shown in Figure 1(a), captured in the LeNet network. The x+o samples are displayed in Figures 1(b) and 1(c). Note that magnifying the captured internal feature representations does not lead to altered classifications, therefore, these examples are optimized for visualization purposes. The x−o examples for the top layers of LeNet are presented in Figures 1(d) and 1(e), where perturbations have the sufficient minimal magnitude leading to a class label different than the origin’s. The perturbations shown in Figures 1(b) and 1(c) demonstrate that the convolutional layers have learned structural characteristics of the input, and increasing the captured internal representations of the origin gracefully translates to a thicker digit. Contrarily, the perturbations formed on the fully-connected layers
displayed in Figures 1(d) and 1(e) seem to have lost spatial focus by considering several locations in the image that appear to be unrelated for the classification of this particular digit.
4.2 INTERNAL REPRESENTATION OF FACES
LOTS is capable of displaying more difficult objects, and can be used to visualize more complex classes than the digits of MNIST. To demonstrate this, we generate x−o and x + o images for various layers of the VGG Face network (Parkhi et al., 2015). Figure 2 shows some distorted samples with the corresponding perturbations generated by modifying the internal representations captured at various layers of the VGG Face network, for an exemplar image taken from the VGG Face Dataset (Parkhi et al., 2015) shown in Figure 2(a). The first two rows of Figure 2 contain x−o samples, while the last row shows x+o representations.
We can observe that due to the large number of identities (classes) present in the VGG Face Dataset, perturbations lead to various class labels. Also, in opposition to the previously described MNIST samples, both x−o and x + o representations can yield altered classifications, which means that overemphasizing the captured features of a given identity also changes the classification of the network, however, these types of samples require perturbations with higher magnitudes.
In summary, the visualized internal feature representations of the origin suggest that lower convolutional layers of the VGG Face model have managed to learn and capture features that provide semantically meaningful and interpretable representations to human observers. Although we can still recognize facial features and parts on visualized feature representations captured at higher layers, closer to the top layer those become harder to interpret. This can be due to the fact that closer to
last layer the model needs features that allow for differentiating identities from one another, hence, those features can be small for visualization, and even semantically meaningless for us.
5 LOTS OF ADVERSARIAL EXAMPLES
Now, let us exploit the instability of DNNs and show how LOTS can be applied to generate adversarial examples. For a given origin image xo, we aim to form an imperceptibly small perturbation which makes the DNN classify the perturbed image differently than the origin. In order to do so, we can exploit Equation (3) to compute a direction using the feature difference of the origin and target at any given layer, and use it to form the adversarial image such that the perturbation has the lowest magnitude necessary for altering the classification or, if desired, reaching the targeted class.
For adversarial example generation, we use the same networks as before: we commence our experiments forming adversarial perturbations for MNIST digits that exploit the vulnerability of the LeNet model, we compare LOTS with previous techniques, then we turn to the more challenging task of manipulating internal representations of faces to cause misclassifications on the VGG Face model.
5.1 ADVERSARIAL EXAMPLES OF HANDWRITTEN DIGITS
Let us consider an image of digit 4 as the origin that we would like to turn to a 9 – at least, when LeNet classifies it. After selecting an applicable image of 9 for being the target, we can use the extracted feature representations of the target and the origin to form the adversarial perturbation. As shown in Figure 3, we use the same origin image as in Figure 1(a) and a selected target image (cf. Figure 3(b)) that looks relatively similar to the origin, but is clearly from another class.
When generating the adversarial example xto by manipulating the internal feature representations captured at the first convolution layer (CONV1), we can clearly see that the perturbation focuses on the difference between the origin 4 and the target 9, as shown in Figure 3(c). Moving to higher layers (CONV2, IP1 and IP2), we can observe that perturbations lose the focus more and more on the global structure. Interestingly, the perturbations have smaller L2 and L∞ norms, but also lower PASS scores, which indicates that the perturbed images generated at lower layers are less distinguishable from the origin, in other words, they are better in terms of adversarial quality.
In order to compare LOTS with previous adversarial generation techniques, we have conducted experiments with LeNet to measure the adversarial quality of the produced samples and – after using them for training – their effect on error rates and adversarial robustness.
We have trained a LeNet model on the MNIST Training dataset (60K digits), denoted as the Basic LeNet model, and we have also generated 2M images with InfiMNIST (Loosli et al., 2007) and trained a network for 100K iterations with the same hyperparameters distributed with Caffe (Jia et al., 2014). To compare adversarial generation techniques with respect to adversarial quality, we have formed perturbed images with FGS and FGV methods, the hot/cold (HC) approach, and LOTS on the MNIST Test dataset (10K digits), and collected four metrics: the success rate showing the percentage of all attempts the particular technique can produce a perturbation which changes the classification, PASS quantifying the adversarial quality, and L2 and L∞ norms of perturbations. For LOTS, we have limited the targets by selecting one image per class – yielding “only” 36 possible
perturbations for each input. Compared to other techniques, LOTS maintains a high success rate while the achieved adversarial quality is lower than FGV and HC samples have, as we can see in Table 1. However, we have found that using LOTS with FGV or HC targets can produce perturbed images with adversarial quality surpassing the originating targets – indicated by higher PASS scores – but these “improved” samples do not have any further benefits when used for adversarial training.
To analyze the performances of adversarial generation techniques when the formed samples are used for training, we have generated perturbed images for the MNIST Training dataset on our Basic LeNet model and then used those images for fine-tuning their originating learning model. Since we have various numbers of perturbed images for each adversarial type, we have fine-tuned Basic LeNet using the regular hyperparameters for different number of iterations: 20K iterations for FGS (app. 52K images) and FGV (app. 51K images) samples, 50K iterations for HC (app. 538K images), and 100K iterations for LOTS (app. 2.1M images) samples. For LOTS, we have selected one image per class and used the same 10 images as targets. Considering all possible targets, the overall number of perturbed images would be beyond 10 billion. By looking at the results listed in Table 1, we can observe that fine-tuning with LOTS samples achieves the lowest error rate (0.65%) on the MNIST Test dataset among models trained with adversarial examples – slightly worse than InfiMNIST – and this network is the most robust considering the collected metrics of all four adversarial types as well. Finally, we have compared LOTS samples formed by manipulating the captured feature representations at various layers. We have found that, in general, fine-tuning with LOTS samples from higher layers require adversarial generation techniques to form stronger, more visible perturbations as indicated by the decreasing PASS scores and increasing L2 and L∞ norms on models denoted as LOTS CONV1, LOTS CONV2, LOTS IP1, and LOTS IP2.
5.2 ADVERSARIAL EXAMPLES OF FACES
A greater threat to automatic face recognition systems is presented by the possibility of generating adversarial examples from face images. In DNN-based face recognition systems, usually the last layer of the network (containing the identities) is disregarded, and the output of the penultimate layer (e.g., FC7 of the VGG Face model) is stored as a representation of the face. When an adversary steals this representation of a target (without requiring the possession of the target image) and also has access to the original network, he can generate an image that looks like himself to a human operator, but is identified as the target identity by the network.
Given the internal representation of a target image xt representing identity t (Sean Bean in Figure 4(b)) at a given layer l of the network, and an origin image xo including its internal representation of identity o (Hugh Laurie in Figure 4(a)), we can use Equation (3), and form the adversarial image xto classified as identity t. As shown in Figure 4, the adversarial images are basically indistinguishable from the original images as indicated by the very high PASS scores, yet the network classifies them incorrectly as the target.
Finally, LOTS can be modified to perform better, however, it would be computationally more expensive. Namely, instead of taking the direction defined by the backpropagated feature difference with respect to the origin once and using it throughout the whole line-search, we can consider calculating the direction multiple times. This “step-and-adjust” optimization could potentially improve both the quality of the produced adversarial examples and the capability of LOTS to reach the target.
6 CONCLUSION
In this paper, we have presented our novel layerwise origin-target synthesis (LOTS) algorithm which can be efficiently used for multiple purposes as we have demonstrated on two well-known deep neural networks (DNNs): the hand-written digit recognition network LeNet and the face recognition network from VGG. First, we can visualize the captured internal structure of an input at any layer of DNNs and we can also gain intuitive insights into the interpretation of a given input by DNNs by magnifying what the network at a particular layer “thinks” is or is not a good representation of the object. Second, the stability of the captured internal feature representations can be assessed with respect to the class of the input by exploring how robust they are to perturbations aiming at magnifying or reducing them. Third, we have demonstrated that LOTS is capable of producing a large number of diverse adversarial examples for each input by modifying the internal structure of the original input to mimic the particular target. This application of LOTS can help us get a deeper understanding about the intrinsic features that differentiate classes in various layers of DNNs. Finally, we have conducted large-scale experiments to compare our approach with other adversarial generation techniques and have concluded that using the large number of diverse perturbed examples produced via LOTS for adversarial training outperforms previous methods with respect to both the achieved performance and the improved adversarial robustness.
ACKNOWLEDGMENTS
This research is based upon work funded in part by NSF IIS-1320956 and in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014-14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.

This paper proposes the Layerwise Origin Target Synthesis (LOTS) method, which entails computing a difference in representation at a given layer in a neural network and then projecting that difference back to input space using backprop. Two types of differences are explored: linear scalings of a single input’s representation and difference vectors between representations of two inputs, where the inputs are of different classes.
In the former case, the LOTS method is used as a visualization of the representation of a specific input example, showing what it would mean, in input space, for the feature representation to be supressed or magnified. While it’s an interesting computation to perform, the value of the visualizations is not very clear.
In the latter case, LOTS is used to generate adversarial examples, moving from an origin image just far enough toward a target image to cause the classification to flip. As expected, the changes required are smaller when LOTS targets a higher layer (in the limit of targetting the last layer, results similar to the original adversarial image results would be obtained).
The paper is an interesting basic exploration and would probably be a great workshop paper. However, the results are probably not quite compelling enough to warrant a full ICLR paper.
A few suggestions for improvement:
 - Several times it is claimed that LOTS can be used as a method for mining for diverse adversarial examples that could be used in training classifiers more robust to adversarial perturbation. But this simple experiment of training on LOTS generated examples isn’t tried. Showing whether the LOTS method outperforms, say, FGS would go a long way toward making a strong paper.
 - How many layers are in the networks used in the paper, and what is their internal structure? This isn’t stated anywhere. I was left wondering whether, say, in Fig 2 the CONV2_1 layer was immediately after the CONV1_1 layer and whether the FC8 layer was the last layer in the network.
 - In Fig 1, 2, 3, and 4, results of the application of LOTS are shown for many intermediate layers but miss for some reason applying it to the input (data) layer and the output/classification (softmax) layer. Showing the full range of possible results would reinforce the interpreatation (for example, in Fig 3, are even larger perturbations necessary in pixel space vs CONV1 space? And does operating directly in softmax space result in smaller perturbations than IP2?)
 - The PASS score is mentioned a couple times but never explained at all. E.g. Fig 1 makes use of it but does not specify such basics as whether higher or lower PASS scores are associated with more or less severe perturbations. A basic explanation would be great.
 - 4.2 states “In summary, the visualized internal feature representations of the origin suggest that lower convolutional layers of the VGG Face model have managed to learn and capture features that provide semantically meaningful and interpretable representations to human observers.” I don’t see that this follows from any results. If this is an important claim to the paper, it should be backed up by additional arguments or results.
1/19/17 UPDATE AFTER REBUTTAL:
Given that experiments were added to the latest version of the paper, I'm increasing my review from 5 -> 6. I think the paper is now just on the accept side of the threshold.