COMBATING DEEP REINFORCEMENT LEARNING’S SISYPHEAN CURSE WITH INTRINSIC FEAR
To use deep reinforcement learning in the wild, we might hope for an agent that can avoid catastrophic mistakes. Unfortunately, even in simple environments, the popular deep Q-network (DQN) algorithm is doomed by a Sisyphean curse. Owing to the use of function approximation, these agents eventually forget experiences as they become exceedingly unlikely under a new policy. Consequently, for as long as they continue to train, DQNs may periodically relive catastrophic mistakes. Many real-world environments where people might be injured exhibit a special structure. We know a priori that catastrophes are not only bad, but that agents need not ever get near to a catastrophe state. In this paper, we exploit this structure to learn a reward shaping that accelerates learning and guards oscillating policies against repeated catastrophes. First, we demonstrate unacceptable performance of DQNs on two toy problems. We then introduce intrinsic fear, a new method that mitigates these problems by avoiding dangerous states. Our approach incorporates a second model trained via supervised learning to predict the probability of catastrophe within a short number of steps. This score then acts to penalize the Q-learning objective, shaping the reward function away from catastrophic states.
1 INTRODUCTION
Following success on Atari games (Mnih et al., 2015) and the board game Go (Silver et al., 2016), many researchers have begun exploring practical applications of deep reinforcement learning (DRL). With DRL, we might hope to replace simple hand-coded control policies with continuously learning agents. Some investigated applications include robotics (Levine et al., 2016), dialogue systems (Fatemi et al., 2016; Lipton et al., 2016), energy management (Night, 2016), and self-driving cars (Shalev-Shwartz et al., 2016). Amid this push to apply DRL, we might ask, can we trust these agents in the wild?
Agents acting in real world environments might possess the ability to cause catastrophic outcomes. Consider a self-driving car that might hit pedestrians, or a domestic robot that might injure a child. We might hope to prevent DRL agents from ever making catastrophic mistakes. But doing so requires extensive prior knowledge of the environment in order to constrain the exploration of policy space (Garcıa & Fernández, 2015). Such knowledge may not always exist. In this paper, we don’t assume prior knowledge of the state-space.
Already, we should note that multiple, conflicting definitions of safety and catastrophe exist, a problem that invites further philosophical consideration. We will return to this matter in Section 3. For now, we introduce a specific but plausible notion of a catastrophe. Suppose that catastrophes are a subset of states that are terminal, yielding a return of 0 and that returns are greater than or equal to
∗http://zacklipton.com
0 for non-catastrophic states. Moreover, we assume that an optimal policy never even comes near a catastrophe state. Note that this doesn’t hold in general RL problems. In a cliff world, the pot of gold might lie on the ledge of a cliff. But for a self-driving car, we don’t suspect that it ever should have to come within a split-second decision of committing murder. Finally, while we don’t assume prior knowledge of which states are dangerous, we do assume the existence of catastrophe detector. After encountering a catastrophic state, an agent can realize this and take action to avoid dangerous states in the future.
Given this definition, we address two challenges: First, can we expect DRL agents, after experiencing some number of catastrophic failures, to avoid perpetually making the same mistakes? Second, can we use our prior knowledge that catastrophes should be kept at distance to accelerate learning of a DRL agent? Our experiments show that even on toy problems, the deep Q-network (DQN), a basic algorithm behind many of today’s state of the art DRL systems, struggles on both counts. Even in toy environments, DQNs may encounter thousands of catastrophes before avoiding them and may periodically repeat old errors so long as they continue to learn. We call this latter problem the Sisyphean curse.
This poses a formidable obstacle to using DQNs in the real world. How can we hand over responsibility for any consequential actions (control of a car, say) to a DRL agent if it may be doomed to periodically remake every kind of mistake, however grave, so long as it continues to learn? Imagine a self-driving car that had to periodically hit a few pedestrians in order to remember that it’s undesirable.
Note that traditional tabular agents do not suffer from the Sisyphean curse. In the tabular setting, an RL agent never forgets the learned dynamics of its environment, even as its policy evolves. Thus, if the Markovian assumption holds, eventual convergence to a globally optimal policy is guaranteed. Unfortunately, the tabular approach becomes infeasible in high-dimensional, continuous state spaces.
The trouble owes to the use of function approximation (Murata & Ozawa, 2005). When training a DQN, we successively update a neural network based on experiences. These experiences might be sampled in an online fashion, from a trailing window (experience replay buffer), or uniformly from all past experiences. Regardless of which mode we use to train the network, eventually, states that a learned policy never encounters will come to form an infinitesimally small region of the training distribution. At such time, our networks are subject to the classic problem of catastrophic interference McCloskey & Cohen (1989); McClelland et al. (1995). Nothing prevents the DQN’s policy from drifting back towards one that revisits catastrophic, but long-forgotten mistakes.
More formally, we could characterize the failures as:
• Training under distribution D, our agent produces a safe policy πs that avoids catastrophes • Collecting data generated under πs yields a distribution D′ • Training under D′, the agent produces πd, a policy that once again experiences catastrophes
In this paper, we illustrate the brittleness of modern deep reinforcement learning algorithms. We introduce a simple pathological problem called Adventure Seeker. This problem consists of a one-dimensional continuous state, two actions, simple dynamics and a clear analytic solution. Nevertheless, the DQN fails. We then show that similar dynamics exist in the classic RL environment Cart-Pole. Finally, we present preliminary findings on the Atari game Seaquest, suggesting that the intrinsic fear model may also be useful for improving performance in more complex environments.
To combat these problems, we propose intrinsic fear (Figure 1). In this heuristic approach, alongside the DQN, we train a supervised danger model. The danger model predicts which states are likely to lead to a catastrophe within some number of steps kr. The output of this model (a probability) is then scaled by a fear factor and used to penalize the Q-learning target. Our approach bears some resemblance to intrinsic motivation Chentanez et al. (2004). But instead of perturbing the reward function to encourage the discovery of novel states, we perturb it to discourage the rediscovery of catastrophic states.
2 BACKGROUND: DEEP Q-LEARNING
For context we briefly review deep Q-learning. Over a series of turns, an agent interacts with its environment via a Markov decision process, or MDP, (S,A, T ,R, γ). At each step t, an agent
observes a state s ∈ S. The agent then chooses an action a ∈ A according to some policy π. In turn, the environment transitions to a new state st+1 ∈ S according to transition dynamics T (st+1|st, at) and generates a reward rt with expectationR(s, a).. This cycle continues until each episode terminates.
The goal of an agent is to maximize the cumulative discounted return ∑T t=0 γ
trt. Temporaldifferences (TD) methods (Sutton, 1988) such as Q-learning (Watkins & Dayan, 1992) model the Q-function, which gives the optimal discounted total reward of a state–action pair; the greedy policy w.r.t. the Q-function is optimal (Sutton & Barto, 1998). Most problems of practical interests have large state spaces, thus the Q-function has to be approximated by parametric models such as neural networks.
In deep Q-learning, this is typically accomplished by alternately collecting experiences by acting greedily with respect to Q(s, a; θQ) and updating the parameters θQ. Updates proceed as follows. For a given experiences (st, at, rt, st+1), we minimize the squared Bellman error:
L = (Q(st, at; θQ)− yt)2 (1)
for yt = rt + γ · maxa′ Q(st+1, a′; θQ). Traditionally, the parameterised Q(s, a; θ) is trained by stochastic approximation, estimating the loss on each experience as it is encountered, yielding the update:
θt+1 ←θt + α(yt −Q(st, at; θt))∇Q(st, at; θt). (2)
Q-learning methods also require an exploration strategy, and for simplicity we consider only the -greedy heuristic. However, our techniques apply equally for Thompson-sampling based exploration. For a thorough overview of RL fundamentals, we refer the reader to Sutton & Barto (1998).
A few tricks are often useful to stabilize Q-learning with function approximation. Of particular relevance to this work is experience replay (Lin, 1992): the RL agent maintains a buffer of past experiences, applying TD-learning on randomly selected mini-batches of experience to update the Qfunction. This technique has proven effective to make Q-learning more stable and more data-efficient (Lin, 1992; Mnih et al., 2015).
3 SAFE REINFORCEMENT LEARNING
At the outset we ought to consider a more formal treatment of safe reinforcement learning. Precisely what do we mean by safety, catastrophe, or danger? The literature offers several notions of safety in reinforcement learning. Garcıa & Fernández (2015) provide a comprehensive review of this literature, suggesting that existing approaches can be divided into two categories.
The first approach consists of modifying the objective function. Traditionally, in a Markov decision process, a reinforcement learner seeks to maximize expected return. Hans et al. (2008) defines a fatality as a return below some threshold τ . In this view, an agent might seek to minimize the worst
case scenario instead of maximizing the expected return. Along these lines, Heger (1994) suggested the Q̂-learning objective:
Q̂ = min(Q̂(st, at), rt+1 + γ max at+1∈A Q̂(st+1, at+1))
Other relevant papers suggest modifying the objective to address risk. While classic Q-learning objective addresses only expected returns, alternative objectives penalize policies for returns with high-variance Garcıa & Fernández (2015). Other suggest incorporating external knowledge into the exploration process. In the extreme case, we might confine our policy search to the subset of policies known to be safe.
We see the following problem. The first category of methods acts only on the summary statistics of the return. This can be beneficial because it requires no foreknowledge. But the drawback is that they fundamentally change the objective everywhere, even in states that we might not be worried about. On the other hand, processes relying on expert knowledge may presume an unreasonable level of foreknowledge.
In this paper, we propose a new formulation of the safety problem. We suppose there exists a subset C ⊂ S of catastrophic states. These states have minimum return, and thus resemble the fatalities of Hans et al. (2008). Additionally we assume that it’s reasonable for policy to avoid even getting close to these states.
In other words, there exist good policies that never get within a few hops of a catastrophe state. Just as a child shouldn’t run with scissors, a self-driving car should never be one split-second decision away from killing a passenger or pedestrian. Note, this imposes some prior knowledge. In a general, unknown environment, it might be that a good policy should get very close to a fatality in order to maximize reward.
We don’t assume advanced knowledge of C, but we do assume the agent possesses a catastrophe detector. In the case of a self-driving car, this might be an impact detector. Even if we can’t anticipate all the situations which might lead to an accident, we can recognize one when it has already happened and take measures to avoid danger zones around these states in the future.
The notion of danger zone suggests some measure of proximity. Informally, we conceive of danger as the likelihood of entering a catastrophe state c ∈ C within a short number of steps. Note, this probability depends on the policy. In this paper, we collect the experiences used to train the danger model from the entire training period. So the likelihood here is determined based on the observations of many policies over the course of training. Explicitly modeling danger zones can be useful for two reasons. First, it provides a mechanism to avoid oscillating policy learners from repeatedly encountering catastrophe states. Second, reward-shaping to avoid danger zones can significantly reduce the sample complexity of exploration.
4 TWO PATHOLOGICAL FAILURE CASES
We now present two pathological environments. In the Adventure Seeker (Figure 2a) environment, we imagine a player placed on a hill, sloping upward to the right. At each turn, the player can move to the right (up the hill) or left (down the hill). The environment adjusts the player’s position accordingly, adding some random noise. Between the left and right edges of the hill, the player gets more reward for spending time higher on the hill. But if the player goes too far to the right, she will fall off (a catrastrophic state), terminating the episode and receiving a return of 0.
Formally, the state consists of a single continuous variable s ∈ [0, 1.0], denoting the player’s position. The starting position for each episode is chosen uniformly at random in the interval [.25, .75]. The available actions consist only of {−1,+1} (left and right). Given an action at in state st, T (st+1|st, at) gives successor state st+1 ← st + .01 · at + η where η ∼ N (0, .012). At each turn, the player gets reward equal to st (proportional to height). The player falls off the hill, entering the catastrophic terminating state, whenever an action would result in successor state st+1 > 1.0 or st+1 < 0.0.
This game admits an obvious analytic solution. There exists some threshold above which the agent should always choose to go left, and below which it should always go right. Even an infant could grasp the gist of this solution. And yet a state-of-the-art DQN model learning online or with experience
replay successively plunges to its death. To be clear, the DQN does learn a near-optimal thresholding policy quickly. But over the course of continued training, the agent oscillates between a reasonable thresholding policy and one which always moves right, regardless of the state. The pace of this oscillation evens out and all networks (over multiple runs) quickly reach a constant catastrophe per turn rate (Figure 3a) that does not attenuate with continued training. How could we ever trust a system that can’t solve Adventure Seeker to make consequential real-world decisions?
Cart-Pole (Figure 2b) is a classic RL environment in which an agent tries to balance a pole atop a cart. Qualitatively, the game exhibits four distinct catastrophe modes. The pole could fall down to the right or fall down to the left. Additionally, the cart could run off the right boundary of the screen or run off the left.
Formally, at each time, the agent observes a four-dimensional state vector (x, v, θ, ω) consisting respectively of the cart position, cart velocity, pole angle, and the pole’s angular velocity. At each time step, the agent chooses an action, applying a force of either −1 or +1. For every time step that the pole remains upright and the cart remains on the screen, the agent receives a reward of 1. If the pole falls, the episode terminates, giving a return of 0 from the penultimate state. In experiments, we use the implementation CartPole-v0 contained in the openAI gym Brockman et al. (2016). Like Adventure Seeker, this problem admits an analytic solution. A perfect policy should never drop the pole. But, as with Adventure Seeker, a DQN converges to a constant rate of catastrophes per turn.
In addition to these pathological cases, we also present some preliminary experiments extending these results to the Atari game environment. We consider Seaquest, a game in which a player swims under water. The player can navigate up, down, left, right, and shoot a gun. Periodically, as the oxygen gets low, she must rise to the surface for oxygen. Additionally, fish swim across the screen. The player gains points each time she shoots a fish. Colliding with a fish or running out of oxygen result in death. The player has 3 lives, and the final death is a terminal state. In this paper, because the environment doesn’t explicitly report intermediate deaths, we consider only the final death to be the catastrophe state. In the future work, we might hack the game to include all deaths.
5 INTRINSIC FEAR
We now introduce intrinsic fear (Algorithm 1), a novel mechanism for avoiding catastrophes when learning online with function approximation. In our approach, we maintain both a DQN and a separate, supervised danger model. Our danger model provides an auxiliary source of reward, penalizing the Q-learner for entering dangerous states.
The goal in learning this smooth danger zone is twofold. First the knowledge that regions around certain states should be avoided represents a source of prior knowledge that can accelerate learning in some environments. Second, by using danger zones to shape rewards away from catastrophic states, we allow oscillating policies to drift close (but not too close) to catastrophe states, giving them opportunity to adjust trajectories away, without having to re-experience the catastrophe. We draw some inspiration from the idea of a parent scolding a child for running around with a knife. The child can learn to adjust its behavior without actually having to stab someone.
We make the following conservative assumption. While we don’t know anything in advance about the state space, we possess a catastrophe detector which can identify each catastrophe as it happens.
We also assume that the agent collects a buffer of previously observed states. After it detects each catastrophe, the agent can refer back to the trajectory that led up to it.
Algorithm 1: Training DQN with Intrinsic Fear 1 function Train( Q(·, ·; ·), d(·; ·), T , λ, kλ, kr);
Input :Two model architectures: Q (a DQN) and d (the danger model), time limit T , fear factor λ, fear phase-in length kλ, fear radius kr
Output :Learned parameters θQ and θd 2 Initialize parameters θQ and θd randomly 3 Initialize replay memory De, danger states Dd, and safe states Ds 4 for t in 1:T do 5 With probability select random action at 6 Otherwise, select action at = maxa′ Q(st, a′; θQ) 7 Execute action at in environment, observing reward rt and successor state st+1 8 Store transition (st, at, rt, st+1) in D 9 if st+1 is a terminal state then
10 Add states st−kr through st to Dd 11 Add states st−ne through st−kr−1 to Ds 12 end 13 Sample random minibatch of transitions (sτ , aτ , rτ , sτ+1) from D 14 Set λt ← min(λ, λ·tkλ )
15 Set yt ← { rt − λt, for terminal sτ+1 rt +maxa′ Q(st+1, a ′; θQ)− λ · d(st+1; θd) for non-terminal st+1 } 16 Apply SGD step θQ ← θQ − η · ∇θQ(yτ −Q(sτ , aτ ; θQ))2 17 Sample random mini-batch sj with 50% of examples from Dd and 50% from Ds
18 Set yj ← {
1, for sj ∈ Dd 0, for sj ∈ Ds } 19 Apply SGD step θd ← θd − η · ∇θd lossd(yj , d(sj ; θd)) 20 end
The technique works as follows: In addition to the DQN, we maintain a binary classifier that we term a danger model. In our case, it is a neural network of the same architecture as the DQN (but for the output layer). In general, it could be any supervised model. The danger model’s purpose is to identify the likelihood that any state will lead to catastrophe within k moves. In the course of training, our agent adds each experience (s, a, r, s′) to the experience replay buffer. As each catastrophe is reached at the nth turn of an episode, we add the kr (fear radius) states leading up to the catastrophe to a list of danger states. We add the preceding n− kr states to a list of safe states. When n < kr, all states for that episode are added to the list of danger states.
Then after each turn, in addition to making one update to the Q-network, we make one mini-batch update to the danger model. To make this update, we sample 50% of states from the danger states, assigning them label 1 and 50% of states from the safe states, assigning them label 0.
For each update to the DQN, we perturb the TD target yt. Instead of updating Q(st, at; θQ) towards rt +maxa′ Q(st+1, a ′; θQ), we introduce the intrinsic fear to the model via the target:
yIFt = rt +max a′ Q(st+1, a ′; θQ)− λ · d(st+1; θd)
where d(s; θd) is the danger model and λ is a fear factor determining the scale of the impact of intrinsic fear on the Q update.
Note that our method perturbs the objective function. As such, one might be reasonably concerned that it might imply a different optimal policy. We can address this concern in the following way. Let us assume the optimal policy never enters a danger zone of radius kr. Let us also assume that over course of training, the danger model approaches perfect accuracy, meaning that all safe states s are given danger score d(s) = 0. Then because our penalty is equal to 0 for all safe states and >= 0 for all danger states, the optimal policy receives the same return as under the original reward function. However, the subset of suboptimal policies that enter danger zones receive strictly worse returns.
Therefore, as the danger model approaches perfect performance wrt false positives, the optimal policy should be unchanged.
We also note that the danger zone depends on the policy. We could determine its size objectively by reference to the worst possible policy. In practice, however, we have found that we achieve better results by setting a wider fear radius than this worst-case analysis suggests.
Finally, we observe that the setting of the fear radius encodes prior knowledge of a problem. However, this prior knowledge is expressed far more succinctly. It’s not obvious how we could communicate information about the high-dimensional, continuous state space with similar efficiency.
Over the course of our experiments, we discovered the following pattern. To be effective, the fear radius kr should be large enough that the model can detect danger states at a safe distance where they can still be avoided. When the fear radius is too small, the danger probability is only nonzero at a points from which catastrophes are inevitable and intrinsic fear doesn’t help. On the other hand, when the fear radius is too large, all states are predicted to be danger states. For example, early in training, all Cart-Pole experiences are shorter than 20 experiences. So a fear radius of 20 leads our models model to diverge. To overcome this problem, we gradually phase in the fear factor from 0 to λ reaching full strength at predetermined time step kλ. In our Cart-Pole experiments we phase in over 1M steps.
6 EXPERIMENTS
To assess the effectiveness of the intrinsic fear model, we evaluate both a standard DQN (DQNNoFear) and one enhanced by intrinsic fear (DQN-Fear). In both cases, we use multilayer perceptrons (MLPs) with a single hidden layer and 128 hidden nodes. We train all MLPs by stochastic gradient descent using the Adam optimizer Kingma & Ba (2015) to adaptively tune the learning rate. Some might wonder whether the problems we observe truly owe to distributional shift or if they actually stem from the well-known problems of off-policy learning. To show that learning on-policy learning does not mitigate these issues, we also present results for an expected-SARSA variant of the DQN for Adventure Seeker and Cart-Pole.
Because, for the Adventure Seeker problem, an agent can escape from danger with only a few time steps of notice, we set the fear radius kr to 5. We phase in the fear factor quickly, reaching full strength in just 1000 moves. On this problem we set the fear factor λ to 40.
For Cart-Pole, we set a wider fear radius of kr = 20. We initially tried training this model with a shorter fear radius but made the following observation. Some models would learn well surviving for millions of experiences, with just a few hundred catastrophes. This compared to a DQN (Figure 3) which would typically suffer 4000-5000 catastrophes. When examining the output from the danger models on successful vs unsuccessful runs, we noticed that the unsuccessful models would output danger of probability greater than .5 for precisely the 5 moves before a catastrophe. But by that time it would be too late for an agent to correct course. In contrast, on the more successful runs, the danger model would often output predictions in the range .1− .5. We suspect that this gradation between mildly dangerous states and those where danger is imminent provided a richer reward signal to the DQN. Accordingly, we lengthened the fear radius to 20 so that some states might be ambiguous (could show up in either danger list Dd or safe list Ds. On both the Adventure Seeker and Cart-Pole environments, the DQNs augments by intrinsic fear far outperform their otherwise identical counterparts (Figure 3). We compared this approach against some traditional approaches, like memory based methods for preferentially sampling failure cases but they could not improve over the DQN.
On the Atari game Seaquest, we use a fear radius of 10 and a fear factor of .1. Interestingly, the models trained with Intrinsic Fear have indistinguishable catastrophe rates but achieve significantly higher reward. Using a much higher fear factor and wider radius we are able to significantly lower the catastrophe rate but this improvement comes at the expense of total reward. These results suggest an interesting interplay between the various reward signals that warrants further exploration.
7 RELATED WORK
The paper addresses safety in RL, intrinsically motivated RL, and the stability of Q-learning with function approximation under distributional shift. Our work also has some connection reward shaping. While space constraints preclude adequate treatment of all prior work, we attempt to highlight the most relevant papers here.
The Stability of RL with Function Approximation: The potential oscillatory or divergent behavior of Q-learners with function approximation has been previously identified (Boyan & Moore, 1995; Baird et al., 1995; Gordon, 1996). But these papers address neither AI safety nor RL. Murata & Ozawa (2005) addresses the problem of catastrophic forgetting owing to distributional shift in RL with function approximation, proposing a memory-based solution. Outside of reinforcement learning, a number of papers address problems related to non-stationary data, including a book on covariate shift (Sugiyama & Kawanabe, 2012).
Safety in Reinforcement Learning: Several papers address safety in RL. Garcıa & Fernández (2015) provide a thorough review on the topic, dividing the existing works into those which perturb the objective function and those which use external knowledge to improve the safety of exploration. Hans et al. (2008) defines a fatal state as one from which one receives a return below some safety threshold τ . They propose a solution comprised of two components. One, the safety function, identifies unsafe states. The other, denoted the backup model, is responsible for navigating away from the critical state. Their work does not address function approximation, instead focusing on a domain where a tabular approach is viable. Moldovan & Abbeel (2012) gives a definition of safety based on ergodicity. They consider a fatality to be a state from which one cannot return to the start state. Shalev-Shwartz et al. (2016) provides theoretical analysis considering how strong a penalty should be to discourage accidents. They also consider hard constraints to ensure safety, an approach that differs from ours. None of the above works address the case where distributional shift dooms an agent to perpetually revisit catastrophic failure modes.
Intrinsically Motivated Reinforcement Learning: A number of papers have investigated the idea of intrinsically motivated reinforcement learners. Intrinsic motivation refers to an intrinsically assigned reward, in contrast to the extrinsic reward that comes from the environment. Typically
intrinsic motivation is proposed as a way to encourage exploration of an environment (Schmidhuber, 1991; Bellemare et al., 2016) and to acquire a modular set of skills Chentanez et al. (2004). In principle, such motivation can lead agents to explore intelligently even when extrinsic rewards are sparse. Some papers refer to the intrinsic reward for discovery as curiosity. Like classic work on intrinsic motivation, our methods operate by perturbing the reward function. But instead of assigning bonuses to encourage discovery of novel transitions, we assign penalties to discourage re-discovery of catastrophic transitions.
8 DISCUSSION
Our experiments suggest that DQNs may be too brittle for use in real-world applications where harm can come of actions. While it’s easy to visualize these problems on toy examples, similar dynamics are embedded in more complex domains. Consider a domestic robot acting as a barber. The robot might receive positive feedback for giving a closer shave. This reward encourages closer contact at a steeper angle. Of course, the shape of this reward function belies the catastrophe lurking just past the optimal shave. Similar dynamics might be found in a vehicle which is rewarded for traveling faster but could risk an accident with excessive speed.
These scenarios are not so unlike the pathological case presented in Adventure Seeker. We might even say these shortcomings seem analogous to the fundamental inability of linear models to solve XOR. But while XOR can be solved by simply incorporating hidden layers of representation, it’s less obvious how to overcome these pitfalls.
These early successes of the intrinsic fear model suggest that for some classes of problems, we might avoid perpetual catastrophe and still enjoy the benefits of function approximation. In this work we assume the ability to recognize a catastrophe once it has happened. But we don’t assume anything in advance about the precise form that the danger states might take in the state space. Our approaches seem appropriate for problems with some notion of proximity. A self-driving car can’t run over a passenger in the next second if no people are in close proximity. But out methods might not be appropriate for other problems. For example, in a problem where a single action, from any state, can produce a catastrophe with high probability, our methods might fail.
This work represents a first step towards combating AI safety issues stemming from the use of function approximation in deep reinforcement learning. In follow-up work, we hope to formalize the notion of danger presented here. We hope to critically examine competing definitions of catastrophe and of danger, to organize them into a taxonomy and to develop theory addressing the most promising ones. We also hope to explore the effectiveness of our technique on more complex domains. In precisely what sorts of environments should our approach work? Under what conditions can we expect it to fail?

This paper addresses an important and timely topic in a creative way. I consider it to have three flaws (and one good idea).
1) insufficient context of what is known and had been studied before (in shallow RL), for example within the field of “robust RL”. A good place to start might be with the work of Shie Mannor.
2) an ill-defined general problem setup. Does it make sense to do post-hoc labeling of certain actions as “catastrophic” if the agent is not informed about that metric during learning? Training a system to do one thing (maximize reward), but then evaluating it with a different metric is misleading. On the training metric, it could even be that the baseline outperforms the new algorithm? So I’d want to see plots for “average reward” in fig 3 as well. Also, what would the baseline learn if it was given large negative rewards for entering these otherwise invisible “danger states”?
3) a somewhat ad-hoc solution, that introduces new domain-specific hyperparameters (k_r, k_lambda and lambda) a second deep network and and two additional replay memories. In terms of results, I’m also unsure whether I can trust the results, given the long-standing track-record of cart-pole being fully solved by many methods: is DQN an outlier here? Or is the convnet not an appropriate function-approximator? Actually: which exact variant “state-of-the-art” variant of DQN are you using?
The good idea that I encourage the authors to pursue further is D_d, this set of rare but dangerous states, that should be kept around in some form. I see it as an ingredient for continual learning that most typical methods lack -- it is also one of the big differences between RL and supervised learning, where such states would generally be discarded as outliers.
Given my comment a couple of weeks ago, and the prompt response (“we implemented expected SARSA”), I would have expected that the paper had been revised with the new results by now? In any case, I’m open to discussing all these points and revising my opinion based on an updated version of the paper.
Minor comment: the bibliography is done sloppily, with missing years, conference venues and missing/misspelled author lists, e.g. “Sergey et al. Levine”. I also think it is good form to cite the actual conference publications instead of arXiv where applicable.


FRACTALNET: ULTRA-DEEP NEURAL NETWORKS WITHOUT RESIDUALS
We introduce a design strategy for neural network macro-architecture based on selfsimilarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of highperformance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.
1 INTRODUCTION
Residual networks (He et al., 2016a), or ResNets, lead a recent and dramatic increase in both depth and accuracy of convolutional neural networks, facilitated by constraining the network to learn residuals. ResNet variants (He et al., 2016a;b; Huang et al., 2016b) and related architectures (Srivastava et al., 2015) employ the common technique of initializing and anchoring, via a pass-through channel, a network to the identity function. Training now differs in two respects. First, the objective changes to learning residual outputs, rather than unreferenced absolute mappings. Second, these networks exhibit a type of deep supervision (Lee et al., 2014), as near-identity layers effectively reduce distance to the loss. He et al. (2016a) speculate that the former, the residual formulation itself, is crucial.
We show otherwise, by constructing a competitive extremely deep architecture that does not rely on residuals. Our design principle is pure enough to communicate in a single word, fractal, and a simple diagram (Figure 1). Yet, fractal networks implicitly recapitulate many properties hard-wired into previous successful architectures. Deep supervision not only arises automatically, but also drives a type of student-teacher learning (Ba & Caruana, 2014; Urban et al., 2017) internal to the network. Modular building blocks of other designs (Szegedy et al., 2015; Liao & Carneiro, 2015) resemble special cases of a fractal network’s nested substructure.
For fractal networks, simplicity of training mirrors simplicity of design. A single loss, attached to the final layer, suffices to drive internal behavior mimicking deep supervision. Parameters are randomly initialized. As they contain subnetworks of many depths, fractal networks are robust to choice of overall depth; make them deep enough and training will carve out a useful assembly of subnetworks.
The entirety of emergent behavior resulting from a fractal design may erode the need for recent engineering tricks intended to achieve similar effects. These tricks include residual functional forms with identity initialization, manual deep supervision, hand-crafted architectural modules, and studentteacher training regimes. Section 2 reviews this large body of related techniques. Hybrid designs could certainly integrate any of them with a fractal architecture; we leave open the question of the degree to which such hybrids are synergistic.
Our main contribution is twofold:
• We introduce FractalNet, the first simple alternative to ResNet. FractalNet shows that explicit residual learning is not a requirement for building ultra-deep neural networks.
• Through analysis and experiments, we elucidate connections between FractalNet and an array of phenomena engineered into previous deep network designs.
As an additional contribution, we develop drop-path, a novel regularization protocol for ultradeep fractal networks. Without data augmentation, fractal networks, trained with drop-path and dropout (Hinton et al., 2012), exceed the performance of residual networks regularized via stochastic depth (Huang et al., 2016b). Though, like stochastic depth, it randomly removes macro-scale components, drop-path further exploits our fractal structure in choosing which components to disable.
Drop-path constitutes not only a regularization strategy, but also provides means of optionally imparting fractal networks with anytime behavior. A particular schedule of dropped paths during learning prevents subnetworks of different depths from co-adapting. As a consequence, both shallow and deep subnetworks must individually produce correct output. Querying a shallow subnetwork thus yields a quick and moderately accurate result in advance of completion of the full network.
Section 3 elaborates the technical details of fractal networks and drop-path. Section 4 provides experimental comparisons to residual networks across the CIFAR-10, CIFAR-100 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al., 2009) datasets. We also evaluate regularization and data augmentation strategies, investigate subnetwork student-teacher behavior during training, and benchmark anytime networks obtained using drop-path. Section 5 provides synthesis. By virtue of encapsulating many known, yet seemingly distinct, design principles, selfsimilar structure may materialize as a fundamental component of neural architectures.
2 RELATED WORK
Deepening feed-forward neural networks has generally returned dividends in performance. A striking example within the computer vision community is the improvement on the ImageNet (Deng et al., 2009) classification task when transitioning from AlexNet (Krizhevsky et al., 2012) to VGG (Simonyan & Zisserman, 2015) to GoogLeNet (Szegedy et al., 2015) to ResNet (He et al., 2016a). Unfortunately, greater depth also makes training more challenging, at least when employing a firstorder optimization method with randomly initialized layers. As the network grows deeper and more non-linear, the linear approximation of a gradient step becomes increasingly inappropriate. Desire to overcome these difficulties drives research on both optimization techniques and network architectures.
On the optimization side, much recent work yields improvements. To prevent vanishing gradients, ReLU activation functions now widely replace sigmoid and tanh units (Nair & Hinton, 2010). This subject remains an area of active inquiry, with various tweaks on ReLUs, e.g. PReLUs (He et al., 2015), and ELUs (Clevert et al., 2016). Even with ReLUs, employing batch normalization (Ioffe & Szegedy, 2015) speeds training by reducing internal covariate shift. Good initialization can also ameliorate this problem (Glorot & Bengio, 2010; Mishkin & Matas, 2016). Path-SGD (Neyshabur et al., 2015) offers an alternative normalization scheme. Progress in optimization is somewhat orthogonal to our architectural focus, with the expectation that advances in either are ripe for combination.
Notable ideas in architecture reach back to skip connections, the earliest example of a nontrivial routing pattern within a neural network. Recent work further elaborates upon them (Maire et al., 2014; Hariharan et al., 2015). Highway networks (Srivastava et al., 2015) and ResNet (He et al., 2016a;b) offer additional twists in the form of parameterized pass-through and gating. In work subsequent to our own, Huang et al. (2016a) investigate a ResNet variant with explicit skip connections. These methods share distinction as the only other designs demonstrated to scale to hundreds of layers and beyond. ResNet’s building block uses the identity map as an anchor point and explicitly parameterizes an additive correction term (the residual). Identity initialization also appears in the context of recurrent networks (Le et al., 2015). A tendency of ResNet and highway networks to fall-back to the identity map may make their effective depth much smaller than their nominal depth.
Some prior results hint at what we experimentally demonstrate in Section 4. Namely, reduction of effective depth is key to training extremely deep networks; residuals are incidental. Huang et al. (2016b) provide one clue in their work on stochastic depth: randomly dropping layers from ResNet during training, thereby shrinking network depth by a constant factor, provides additional performance benefit. We build upon this intuition through drop-path, which shrinks depth much more drastically.
The success of deep supervision (Lee et al., 2014) provides another clue that effective depth is crucial. Here, an auxiliary loss, forked off mid-level layers, introduces a shorter path during backpropagation. The layer at the fork receives two gradients, originating from the main loss and the auxiliary loss, that are added together. Deep supervision is now common, being adopted, for example, by GoogLeNet (Szegedy et al., 2015). However, irrelevance of the auxiliary loss at test time introduces the drawback of having a discrepancy between the actual objective and that used for training.
Exploration of the student-teacher paradigm (Ba & Caruana, 2014) illuminates the potential for interplay between networks of different depth. In the model compression scenario, a deeper network (previously trained) guides and improves the learning of a shallower and faster student network (Ba & Caruana, 2014; Urban et al., 2017). This is accomplished by feeding unlabeled data through the teacher and having the student mimic the teacher’s soft output predictions. FitNets (Romero et al., 2015) explicitly couple students and teachers, forcing mimic behavior across several intermediate points in the network. Our fractal networks capture yet another alternative, in the form of implicit coupling, with the potential for bidirectional information flow between shallow and deep subnetworks.
Widening networks, by using larger modules in place of individual layers, has also produced performance gains. For example, an Inception module (Szegedy et al., 2015) concatenates results of convolutional layers of different receptive field size. Stacking these modules forms the GoogLeNet architecture. Liao & Carneiro (2015) employ a variant with maxout in place of concatenation. Figure 1 makes apparent our connection with such work. As a fractal network deepens, it also widens. Moreover, note that stacking two 2D convolutional layers with the same spatial receptive field (e.g. 3ˆ 3) achieves a larger (5ˆ 5) receptive field. A horizontal cross-section of a fractal network is reminiscent of an Inception module, except with additional joins due to recursive structure.
3 FRACTAL NETWORKS
We begin with a more formal presentation of the ideas sketched in Figure 1. Convolutional neural networks serve as our running example and, in the subsequent section, our experimental platform. However, it is worth emphasizing that our framework is more general. In principle, convolutional layers in Figure 1 could be replaced by a different layer type, or even a custom-designed module or subnetwork, in order to generate other fractal architectures.
Let C denote the index of the truncated fractal fCp¨q. Our network’s structure, connections and layer types, is defined by fCp¨q. A network consisting of a single convolutional layer is the base case:
f1pzq “ convpzq (1)
We define successive fractals recursively:
fC`1pzq “ rpfC ˝ fCqpzqs ‘ rconvpzqs (2)
where ˝ denotes composition and ‘ a join operation. When drawn in the style of Figure 1, C corresponds to the number of columns, or width, of network fCp¨q. Depth, defined to be the number of conv layers on the longest path between input and output, scales as 2C´1. Convolutional networks for classification typically intersperse pooling layers. We achieve the same by using fCp¨q as a building block and stacking it with subsequent pooling layers B times, yielding total depth B ¨ 2C´1. The join operation ‘ merges two feature blobs into one. Here, a blob is the result of a conv layer: a tensor holding activations for a fixed number of channels over a spatial domain. The channel count corresponds to the size of the filter set in the preceding conv layer. As the fractal is expanded, we collapse neighboring joins into a single join layer which spans multiple columns, as shown on the right side of Figure 1. The join layer merges all of its input feature blobs into a single output blob.
Several choices seem reasonable for the action of a join layer, including concatenation and addition. We instantiate each join to compute the element-wise mean of its inputs. This is appropriate for convolutional networks in which channel count is set the same for all conv layers within a fractal block. Averaging might appear similar to ResNet’s addition operation, but there are critical differences:
• ResNet makes clear distinction between pass-through and residual signals. In FractalNet, no signal is privileged. Every input to a join layer is the output of an immediately preceding conv layer. The network structure alone cannot identify any as being primary.
• Drop-path regularization, as described next in Section 3.1, forces each input to a join to be individually reliable. This reduces the reward for even implicitly learning to allocate part of one signal to act as a residual for another.
• Experiments show that we can extract high-performance subnetworks consisting of a single column (Section 4.2). Such a subnetwork is effectively devoid of joins, as only a single path is active throughout. They produce no signal to which a residual could be added.
Together, these properties ensure that join layers are not an alternative method of residual learning.
3.1 REGULARIZATION VIA DROP-PATH
Dropout (Hinton et al., 2012) and drop-connect (Wan et al., 2013) modify interactions between sequential network layers in order to discourage co-adaptation. Since fractal networks contain additional macro-scale structure, we propose to complement these techniques with an analogous coarse-scale regularization scheme.
Figure 2 illustrates drop-path. Just as dropout prevents co-adaptation of activations, drop-path prevents co-adaptation of parallel paths by randomly dropping operands of the join layers. This discourages the network from using one input path as an anchor and another as a corrective term (a configuration that, if not prevented, is prone to overfitting). We consider two sampling strategies:
• Local: a join drops each input with fixed probability, but we make sure at least one survives. • Global: a single path is selected for the entire network. We restrict this path to be a single
column, thereby promoting individual columns as independently strong predictors.
As with dropout, signals may need appropriate rescaling. With element-wise means, this is trivial; each join computes the mean of only its active inputs.
In experiments, we train with dropout and a mixture model of 50% local and 50% global sampling for drop-path. We sample a new subnetwork each mini-batch. With sufficient memory, we can simultaneously evaluate one local sample and all global samples for each mini-batch by keeping separate networks and tying them together via weight sharing.
While fractal connectivity permits the use of paths of any length, global drop-path forces the use of many paths whose lengths differ by orders of magnitude (powers of 2). The subnetworks sampled by drop-path thus exhibit large structural diversity. This property stands in contrast to stochastic depth regularization of ResNet, which, by virtue of using a fixed drop probability for each layer in a chain, samples subnetworks with a concentrated depth distribution (Huang et al., 2016b).
Global drop-path serves not only as a regularizer, but also as a diagnostic tool. Monitoring performance of individual columns provides insight into both the network and training mechanisms, as Section 4.3 discusses in more detail. Individually strong columns of various depths also give users choices in the trade-off between speed (shallow) and accuracy (deep).
3.2 DATA AUGMENTATION
Data augmentation can reduce the need for regularization. ResNet demonstrates this, achieving 27.22% error rate on CIFAR-100 with augmentation compared to 44.76% without (Huang et al., 2016b). While augmentation benefits fractal networks, we show that drop-path provides highly effective regularization, allowing them to achieve competitive results even without data augmentation.
3.3 IMPLEMENTATION DETAILS
We implement FractalNet using Caffe (Jia et al., 2014). Purely for convenience, we flip the order of pool and join layers at the end of a block in Figure 1. We pool individual columns immediately before the joins spanning all columns, rather than pooling once immediately after them.
We train fractal networks using stochastic gradient descent with momentum. As now standard, we employ batch normalization together with each conv layer (convolution, batch norm, then ReLU).
4 EXPERIMENTS
The CIFAR, SVHN, and ImageNet datasets serve as testbeds for comparison to prior work and analysis of FractalNet’s internal behavior. We evaluate performance on the standard classification task associated with each dataset. For CIFAR and SVHN, which consist of 32ˆ 32 images, we set our fractal network to have 5 blocks (B “ 5) with 2ˆ 2 non-overlapping max-pooling and subsampling applied after each. This reduces the input 32ˆ 32 spatial resolution to 1ˆ 1 over the course of the entire network. A softmax prediction layer attaches at the end of the network. Unless otherwise noted, we set the number of filter channels within blocks 1 through 5 as p64, 128, 256, 512, 512q, mostly matching the convention of doubling the number of channels after halving spatial resolution.
For ImageNet, we choose a fractal architecture to facilitate direct comparison with the 34-layer ResNet of He et al. (2016a). We use the same first and last layer as ResNet-34, but change the middle of the network to consist of 4 blocks (B “ 4), each of 8 layers (C “ 4 columns). We use a filter channel progression of p128, 256, 512, 1024q in blocks 1 through 4.
4.1 TRAINING
For experiments using dropout, we fix drop rate per block at p0%, 10%, 20%, 30%, 40%q, similar to Clevert et al. (2016). Local drop-path uses 15% drop rate across the entire network.
1Densely connected networks (DenseNets) are concurrent work, appearing subsequent to our original arXiv paper on FractalNet. A variant of residual networks, they swap addition for concatenation in the residual functional form. We report performance of their 250-layer DenseNet-BC network with growth rate k “ 24.
2This deeper (4 column) FractalNet has fewer parameters. We vary column width: p128, 64, 32, 16q channels across columns initially, doubling each block except the last. A linear projection temporarily widens thinner columns before joins. As in Iandola et al. (2016), we switch to a mix of 1ˆ 1 and 3ˆ 3 convolutional filters.
Method Top-1 (%) Top-5 (%) VGG-16 28.07 9.33 ResNet-34 C 24.19 7.40 FractalNet-34 24.12 7.39
Table 2: ImageNet (validation set, 10-crop).
Cols. Depth Params. Error (%) 1 5 0.3M 37.32 2 10 0.8M 30.71 3 20 2.1M 27.69 4 40 4.8M 27.38 5 80 10.2M 26.46 6 160 21.1M 27.38
Table 3: Ultra-deep fractal networks (CIFAR-100++). Increasing depth greatly improves accuracy until eventual diminishing returns. Contrast with plain networks, which are not trainable if made too deep (Table 4).
Model Depth Train Loss Error (%) Plain 5 0.786 36.62 Plain 10 0.159 32.47 Plain 20 0.037 31.31 Plain 40 0.580 38.84 Fractal Col #1 5 0.677 37.23 Fractal Col #2 10 0.141 32.85 Fractal Col #3 20 0.029 31.31 Fractal Col #4 40 0.016 31.75 Fractal Full 40 0.015 27.40
Table 4: Fractal structure as a training apparatus (CIFAR-100++). Plain networks perform well if moderately deep, but exhibit worse convergence during training if instantiated with great depth. However, as a column trained within, and then extracted from, a fractal network with mixed drop-path, we recover a plain network that overcomes such depth limitation (possibly due to a student-teacher effect).
We run for 400 epochs on CIFAR, 20 epochs on SVHN, and 70 epochs on ImageNet. Our learning rate starts at 0.02 (for ImageNet, 0.001) and we train using stochastic gradient descent with batch size 100 (for ImageNet, 32) and momentum 0.9. For CIFAR/SVHN, we drop the learning rate by a factor of 10 whenever the number of remaining epochs halves. For ImageNet, we drop by a factor of 10 at epochs 50 and 65. We use Xavier initialization (Glorot & Bengio, 2010).
A widely employed (Lin et al., 2013; Clevert et al., 2016; Srivastava et al., 2015; He et al., 2016a;b; Huang et al., 2016b; Targ et al., 2016) scheme for data augmentation on CIFAR consists of only horizontal mirroring and translation (uniform offsets in r´4, 4s), with images zero-padded where needed after mean subtraction. We denote results achieved using no more than this degree of augmentation by appending a “+” to the dataset name (e.g. CIFAR-100+). A “++” marks results reliant on more data augmentation; here exact schemes may vary. Our entry in this category is modest and simply changes the zero-padding to reflect-padding.
4.2 RESULTS
Table 1 compares performance of FractalNet on CIFAR and SVHN with competing methods. FractalNet (depth 20) outperforms the original ResNet across the board. With data augmentation, our CIFAR-100 accuracy is close to that of the best ResNet variants. With neither augmentation nor regularization, FractalNet’s performance on CIFAR is superior to both ResNet and ResNet with stochastic depth, suggesting that FractalNet may be less prone to overfitting. Most methods perform similarly on SVHN. Increasing depth to 40, while borrowing some parameter reduction tricks (Iandola et al., 2016), reveals FractalNet’s performance to be consistent across a range of configuration choices.
Experiments without data augmentation highlight the power of drop-path regularization. On CIFAR100, drop-path reduces FractalNet’s error rate from 35.34% to 28.20%. Unregularized ResNet is far behind (44.76%) and ResNet with stochastic depth (37.80%) does not catch up to our unregularized starting point of 35.34%. CIFAR-10 mirrors this story. With data augmentation, drop-path provides a boost (CIFAR-10), or does not significantly influence FractalNet’s performance (CIFAR-100).
Note that the performance of the deepest column of the fractal network is close to that of the full network (statistically equivalent on CIFAR-10). This suggests that the fractal structure may be more important as a learning framework than as a final model architecture.
Table 2 shows that FractalNet scales to ImageNet, matching ResNet (He et al., 2016a) at equal depth. Note that, concurrent with our work, refinements to the residual network paradigm further improve the state-of-the-art on ImageNet. Wide residual networks (Zagoruyko & Komodakis, 2016) of 34-layers reduce single-crop Top-1 and Top-5 validation error by approximately 2% and 1%, respectively, over
ResNet-34 by doubling feature channels in each layer. DenseNets (Huang et al., 2016a) substantially improve performance by building residual blocks that concatenate rather than add feature channels.
Table 3 demonstrates that FractalNet resists performance degradation as we increase C to obtain extremely deep networks (160 layers for C “ 6). Scores in this table are not comparable to those in Table 1. For time and memory efficiency, we reduced block-wise feature channels to p16, 32, 64, 128, 128q and the batch size to 50 for the supporting experiments in Tables 3 and 4. Table 4 provides a baseline showing that training of plain deep networks begins to degrade by the time their depth reaches 40 layers. In our experience, a plain 160-layer completely fails to converge. This table also highlights the ability to use FractalNet and drop-path as an engine for extracting trained networks (columns) with the same topology as plain networks, but much higher test performance.
4.3 INTROSPECTION
With Figure 3, we examine the evolution of a 40-layer FractalNet during training. Tracking columns individually (recording their losses when run as stand-alone networks), we observe that the 40-layer column initially improves slowly, but picks up once the loss of the rest of the network begins to stabilize. Contrast with a plain 40-layer network trained alone (dashed blue line), which never makes fast progress. The column has the same initial plateau, but subsequently improves after 25 epochs, producing a loss curve uncharacteristic of plain networks.
We hypothesize that the fractal structure triggers effects akin to deep supervision and lateral studentteacher information flow. Column #4 joins with column #3 every other layer, and in every fourth layer this join involves no other columns. Once the fractal network partially relies on the signal going through column #3, drop-path puts pressure on column #4 to produce a replacement signal when column #3 is dropped. This task has constrained scope. A particular drop only requires two consecutive layers in column #4 to substitute for one in column #3 (a mini student-teacher problem).
This explanation of FractalNet dynamics parallels what, in concurrent work, Greff et al. (2017) claim for ResNet. Specifically, Greff et al. (2017) suggest residual networks learn unrolled iterative estimation, with each layer performing a gradual refinement on its input representation. The deepest FractalNet column could behave in the same manner, with the remainder of the network acting as a scaffold for building smaller refinement steps by doubling layers from one column to the next.
These interpretations appear not to mesh with the conclusions of Veit et al. (2016), who claim that ensemble-like behavior underlies the success of ResNet. This is certainly untrue of some very deep networks, as FractalNet provides a counterexample: we can extract a single column (plain network topology) and it alone (no ensembling) performs nearly as well as the entire network. Moreover, the gradual refinement view may offer an alternative explanation for the experiments of Veit et al. (2016). If each layer makes only a small modification, removing one may look, to the subsequent portion of the network, like injecting a small amount of input noise. Perhaps noise tolerance explains the gradual performance degradation that Veit et al. (2016) observe when removing ResNet layers.
5 CONCLUSION
Our experiments with fractal networks provide strong evidence that path length is fundamental for training ultra-deep neural networks; residuals are incidental. Key is the shared characteristic of FractalNet and ResNet: large nominal network depth, but effectively shorter paths for gradient propagation during training. Fractal architectures are arguably the simplest means of satisfying this requirement, and match residual networks in experimental performance. Fractal networks are resistant to being too deep; extra depth may slow training, but does not impair accuracy.
With drop-path, regularization of extremely deep fractal networks is intuitive and effective. Drop-path doubles as a method of enforcing speed (latency) vs. accuracy tradeoffs. For applications where fast responses have utility, we can obtain fractal networks whose partial evaluation yields good answers.
Our analysis connects the internal behavior of fractal networks with phenomena engineered into other networks. Their substructure resembles hand-crafted modules used as components in prior work. Their training evolution may emulate deep supervision and student-teacher learning.
ACKNOWLEDGMENTS
We gratefully acknowledge the support of NVIDIA Corporation with the donation of GPUs used for this research.

Authors claim that the reviewers reflect "narrow view" of on the number of parameters and fractions of accuracy and  "complete focus on engineering" etc . Authors claims about focusing on engineering are open to debate as only the technicality in the paper is the fractal network architecture with intuitive claims. One can either formulate the system with rigorous analysis with clear assumptions (which would require satisfactory theoretical analysis and relatively small scale empirical sanity check without heavy experiments) or propose a comprehensive empirical analysis by designing careful experiments to support intuitive claims. Therefore, as the approach clearly does not belong to the first category,  such an approach needs strong experimental evidence to support intuitive claims without a doubt.  The rebuttal lists unsatisfactory answers with somehow manipulative arguments about narrow reviewing/ dates of the baselines( which will be discussed below)..
Authors state in the paper that " In experiments, fractal networks **match the state-of-the-art performance held by residual networks** on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks" and "Fractal architectures are arguably the simplest means of satisfying this requirement, and match or exceed residual networks in experimental performance". Therefore the main support behind all the claims is that fractal network match (or exceed) performance of state of art residual networks, indeed the empirical study only compares accuracy to some baselines without designing empirical analysis for the claims about differences to resnets (or ensemble explanation of resnet by Veit et al(NIPS 2016) ).
However even this comparison lacks many results as explained in my review.   Therefore the expectation of the fair comparison is completely reasonable and it is not about only focusing on fractions of accuracy or number of parameters. As a good example, Veit et al demonstrates systematical empirical support for their claims about analyzing residual networks (which seems to be on arxiv since May but still not cited on paper ). Table 4 and Figure 3 provide good preliminary sanity check but compares only to plain networks, do not support claims about differences between residual and fractal networks.
The paper claims that the fractal network scales to "ultra" deep networks, however authors can not report results on dozens of layers. Authors also claim that extra depth may slow training (not clear how much) but does not decrease accuracy but this is not clear as there are no results for dozens of layers, also in Table 3, error increases as depth increase to 160 layers.  Number of parameters of the proposed architecture is significantly greater than the state of art resnet variants as I explained in the review but the authors argue that it is slightly more parameters.
As an answer to lack of comparison to DenseNet, authors argue in rebuttal that DenseNet cites FractalNet but this can not be a reason for the lack of comparison as DenseNet was published in August and well-known as holding state of art results as a resnet variant.
Authors state in rebuttal that "Many of the variants only reported results on CIFAR/SVHN… it is a bit difficult to have already compared to results that did not exist at submission time." .  However there were clearly published ones, for example two results by Huang et al, 2016b on july (arxiv) ..
Regarding the "simplifying power" claim of authors.  It is not very convincing that it is simplifying. how can it simplify with a harder training procedure with many parameters that can not scale as good as baselines?
In essence, my evaluation did not change as rebuttal did not provide satisfactory clarification or improvement.