Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a suitable reward function takes considerable manual engineering and often requires additional and potentially visible sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide useful feedback on these implicit intermediate steps or sub-goals. To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify the key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit sub-goal supervision. The resulting reward functions, which are dense and smooth, can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward functions, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also demonstrate that our method can be used to learn a complex real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task.
Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a suitable reward function takes considerable manual engineering and often requires additional and potentially visible sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide useful feedback on these implicit intermediate steps or sub-goals. To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify the key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit sub-goal supervision. The resulting reward functions, which are dense and smooth, can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward functions, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also demonstrate that our method can be used to learn a complex real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task.
1 INTRODUCTION
Social learning, such as imitation, plays a critical role in allowing humans and animals to quickly acquire complex skills in the real world. Humans can use this weak form of supervision to acquire behaviors from very small numbers of demonstrations, in sharp contrast to deep reinforcement learning (RL) methods, which typically require extensive training data. In this work, we make use of two ideas to develop a scaleable and efficient imitation learning method: first, imitation makes use of extensive prior knowledge to quickly glean the “gist” of a new task from even a small number of demonstrations; second, imitation involves both observation and trial-and-error learning (RL). Building on these ideas, we propose a reward learning method for understanding the intent of a user demonstration through the use of pre-trained visual features, which provide the “prior knowledge” for efficient imitation. Our algorithm aims to discover not only the high-level goal of a task, but also the implicit sub-goals and steps that comprise more complex behaviors. Extracting such sub-goals can allow the agent to make maximal use of the information contained in the demonstration. Once the reward function has been extracted, the agent can use its own experience at the task to determine the physical structure of the behavior, even when the reward is provided by an agent with a substantially different embodiment (e.g. a human providing a demonstration for a robot).
∗Work done as part of the Google Brain Residency program (g.co/brainresidency).
To our knowledge, our method is the first reward learning technique that learns generalizable visionbased reward functions for complex robotic manipulation skills from only a few demonstrations provided directly by a human. Although prior methods have demonstrated reward learning with vision for real-world robotic tasks, they have either required kinesthetic demonstrations with robot state for reward learning (Finn et al., 2015), or else required low-dimensional state spaces and numerous demonstrations (Wulfmeier et al., 2016). The contributions of this paper are:
• A method for perceptual reward learning from only a few demonstrations of real-world tasks. Reward functions are dense and incremental, with automated unsupervised discovery of intermediate steps.
• The first vision-based reward learning method that can learn a complex robotic manipulation task from a few human demonstrations in real-world robotic experiments.
• A set of empirical experiments that show that the learned visual representations inside a pre-trained deep model are general enough to be directly used to represent goals and subgoals for manipulation skills in new scenes without retraining.
1.1 RELATED WORK
Deep reinforcement learning and deep robotic learning work has previously examined learning reward functions based on images. One of the most common approaches to image-based reward functions is to directly specify a “target image” by showing the learner the raw pixels of a successful task completion state, and then using distance to that image (or its latent representation) as a reward function (Lange et al., 2012; Finn et al., 2015; Watter et al., 2015). However, this approach
has several severe shortcomings. First, the use of a target image presupposes that the system can achieve a substantially similar visual state, which precludes generalization to semantically similar but visually distinct situations. Second, the use of a target image does not provide the learner with information about which facet of the image is more or less important for task success, which might result in the learner excessively emphasizing irrelevant factors of variation (such as the color of a door due to light and shadow) at the expense of relevant factors (such as whether or not the door is open or closed). Analyzing a collection of demonstrations to learn a parsimonious reward function that explains the demonstrated behavior is known as inverse reinforcement learning (IRL) (Ng et al., 2000). A few recently proposed IRL algorithms have sought to combine IRL with vision and deep network representations (Finn et al., 2016b; Wulfmeier et al., 2016). However, scaling IRL to high-dimensional systems and open-ended reward representations is very challenging. The previous work closest to ours used images together with robot state information (joint angles and end effector pose), with tens of demonstrations provided through kinesthetic teaching (Finn et al., 2016b). The approach we propose in this work which can be interpreted as a simple and efficient approximation to IRL, can use demonstrations that consist of videos of a human performing the task using their own body, and can acquire reward functions with intermediate sub-goals using just a few examples. This kind of efficient vision-based reward learning from videos of humans has not been demonstrated in prior IRL work. The idea of perceptual reward functions using raw pixels was also explored by Edwards et al. (2016) which, while sharing the same spirit as this work, was limited to simple synthetic tasks and used single images as perceptual goals rather than multiple demonstration videos.
2 SIMPLE INVERSE REINFORCEMENT LEARNING WITH VISUAL FEATURES
The key insight in our approach is that we can exploit the semantically meaningful and powerful features in a pre-trained deep neural network to infer task goals and sub-goals using a very simple approximate inverse reinforcement learning method. The pre-trained network effectively transfers prior knowledge about the visual world to make imitation learning fast and robust. Our approach can be interpreted as a simple approximation to inverse reinforcement learning under a particular choice of system dynamics, as discussed in Section 2.1. While this approximation is somewhat simplistic, it affords an efficient and scaleable learning rule that avoids overfitting even when trained on a small number of demonstrations. As depicted in Fig. 1, our algorithm first segments the demonstrations into segments based on perceptual similarity, as described in Section 2.2. Intuitively, the resulting segments correspond to sub-goals or steps of the task. The segments can then be used as a supervision signal for learning steps classifiers, described in Section 2.3, which produces a single perception reward function for each step of the task. The combined reward function can then be used with a reinforcement learning algorithm to learn the demonstrated behavior. Although this method for extracting reward functions is exceedingly simple, its power comes from the use of highly general and robust pre-trained visual features, and our key empirical result is that such features are sufficient to acquire effective and generalizable reward functions for real-world manipulation skills.
We use the Inception network (Szegedy et al., 2015) pre-trained for ImageNet classification (Deng et al., 2009) to obtain the visual features for representing the learned rewards. It is well known that visual features in such networks are quite general and can be reused for other visual tasks. However, it is less clear if sparse subsets of such features can be used directly to represent goals and subgoals for real-world manipulation skills. Our experimental evaluation suggests that indeed they can, and that the resulting reward representations are robust and reliable enough for real-world robotic learning without any finetuning of the features. In this work, we use all activations starting from the first “mixed” layer that follows the first 5 convolutional layers (this layer’s activation map is of size 35x35x256 given a 299x299 input). While this paper focuses on visual perception, the approach is general and can be applied to other modalities (e.g. audio and tactile).
2.1 INVERSE REINFORCEMENT LEARNING WITH TIME-INDEPENDENT GAUSSIAN MODELS
Inverse reinforcement learning can be performed with a variety of algorithms (Ng et al., 2000), ranging from margin-based methods (Abbeel & Ng, 2004; Ratliff et al., 2006) to methods based on probabilistic models (Ramachandran & Amir, 2007; Ziebart et al., 2008). In this work, we use a very simple approximation to the MaxEnt IRL model (Ziebart et al., 2008), a popular probabilistic approach to IRL. We will use st to denote the visual feature activations at time t, which constitute the state, sit to denote the ith feature at time t, and τ = {s1, . . . , sT } to denote a sequence or trajectory of these activations in a video. In MaxEnt IRL, the demonstrated trajectories τ are assumed to be
drawn from a Boltzmann distribution according to:
p(τ) = p(s1, . . . , sT ) = 1
Z exp  T∑ t=1 R(st)  , (1) whereR(st) is the unknown reward function. The principal computational challenge in MaxEnt IRL is to approximate Z, since the states at each time step are not independent, but are constrained by the system dynamics. In deterministic systems, where st+1 = f(st, at) for actions at and dynamics f , the dynamics impose constraints on which trajectories τ are feasible. In stochastic systems, where st+1 ∼ p(st+1|st, at), we must also account for the dynamics distribution in Equation (1), as discussed by Ziebart et al. (2008). Prior work has addressed this using dynamic programming to exactly compute Z in small, discrete systems (Ziebart et al., 2008), or by using sampling to estimate Z for large, continuous domains (Kalakrishnan et al., 2010; Boularias et al., 2011; Finn et al., 2016b). Since our state representation corresponds to a large vector of visual features, exact dynamic programming is infeasible. Sample-based approximation requires running a large number of trials to estimate Z and, as shown in recent work (Finn et al., 2016a), corresponds to a variant of generative adversarial networks (GANs), with all of the accompanying stability and optimization challenges. Furthermore, the corresponding model for the reward function is complex, making it prone to overfitting when only a small number of demonstrations is available.
When faced with a difficult learning problem in extremely low-data regimes, a standard solution is to resort to simple, biased models, so as to minimize overfitting. We adopt precisely this approach in our work: instead of approximating the complex posterior distribution over trajectories under nonlinear dynamics, we use a simple biased model that affords efficient learning and minimizes overfitting. Specifically, we assume that all trajectories are dynamically feasible, and that the distribution over each activation at each time step is independent of all other activations and all other time steps. This corresponds to the IRL equivalent of a naı̈ve Bayes model: in the same way that naı̈ve Bayes uses an independence assumption to mitigate overfitting in high-dimensional feature spaces, we use independence between both time steps and features to learn from very small numbers of demonstrations. Under this assumption, the probability of a trajectory τ factorizes according to
p(τ) = T∏ t=1 N∏ i=1 p(sit) = T∏ t=1 N∏ i=1 1 Zit exp ( Ri(sit) ) ,
which corresponds to a reward function of the form Rt(st) = ∑N
i=1Ri(sit). We can then simply choose a form for Ri(sit) that can be normalized analytically, which in our case is a quadratic in sit, such that exp(Ri(sit))/Zit is a Gaussian distribution, and the original trajectory distribution is a naı̈ve Bayes model. While this approximation is quite drastic, it yields an exceedingly simple learning rule: in the most basic version, we have only to fit the mean and variance of each feature distribution, and then use the log of the resulting Gaussian as the reward.
2.2 DISCOVERY OF INTERMEDIATE STEPS
The simple IRL model in the previous section can be used to acquire a single quadratic reward function in terms of the visual features st. However, for complex multi-stage tasks, this model can be too coarse, making task learning slow and difficult. We therefore instead fit multiple quadratic reward functions, with one reward function per intermediate step or goal. These steps are discovered automatically in the first stage of our method, which is performed independently on each demonstration. If multiple demonstrations are available, they are pooled together in the feature selection step discussed in the next section, and could in principle be combined at the segmentation stage as well, though we found this to be unnecessary in our prototype. The intermediate steps model extends the simple independent Gaussian model in the previous section by assuming that
p(τ) = T∏ t=1 N∏ i=1 1 Zit exp ( Rigt(sit) ) ,
where gt is the index of the goal or step corresponding to time step t. Learning then corresponds to identifying the boundaries of the steps in the demonstration, and fitting independent Gaussian feature distributions at each step. Note that this corresponds exactly to segmenting the demonstration such that the variance of each feature within each segment is minimized.
In this work, we employ a simple recursive video segmentation algorithm as described in Algorithm 1. Intuitively, this method breaks down a sequence in a way that each frame in a segment is abstractly similar to each other frame in that segment. The number of segments is provided manually in this approach, though it would be straightforward to also utilize standard model selection criteria for choosing this number automatically. There exists a body of unsupervised video segmentation methods Yuan et al. (2007) which would likely enable a less constrained set of demonstrations to be used. While this is an important avenue of future work, we show that our simple approach is sufficient to demonstrate the efficacy of our method on a realistic set of demonstrations. We also investigate how to reduce the search space of similar feature patterns across videos in section 2.3. This would render discovery of video alignments tractable for an optimization method such as the one used in Joulin et al. (2014) for video co-localization.
The complexity of Algorithm 1 is O(nm) where n is the number of frames in a sequence and m the number of splits. Note that dynamic programming is not applicable to this algorithm because each sub-problem, i.e. how to split a sequence after the ith frame, depends on the segmentation chosen before the ith frame. We also experiment with a greedy binary version of this algorithm (Algorithm 2 detailed in section A.1): first split the entire sequence in two, then recursively split each new segment in two. While not exactly minimizing the variance across all segments, it is significantly more efficient (O(n2 logm)) and yields qualitatively sensible results.
Algorithm 1 Recursive similarity maximization, where AverageStd() is a function that computes the average standard deviation over a set of frames or over a set of values, Join() is a function that joins values or lists together into a single list, n is the number of splits desired and min size is the minimum size of a split.
function SPLIT(video, start, end, n,min size, prev std = []) if n = 1 then return [], [AVERAGESTD(video[start : end])] end if min std← None min std list← [] min split← [] for i← start+min size to end− ((n− 1) ∗min size)) do
std1← [AVERAGESTD(video[start : i])] splits2, std2← SPLIT(video, i, end, n− 1,min size, prev std+ std1) avg std← AVERAGESTD(JOIN(prev std, std1, std2)) if min std = None or avg std < min std then
min std← avg std min std list← JOIN(std1, std2) min split← JOIN(i, splits2)
end if end for return min split,min std list
end function
2.3 STEPS CLASSIFICATION
In this section we explore learning a steps classifier on top of the pre-trained deep model using a regular linear classifier and a custom feature selection classifier.
Intent understanding requires identifying highly discriminative features of a specific goal while remaining invariant to unrelated variation (e.g. lighting, color, viewpoint). The relevant discriminative features may be very diverse and more or less abstract, which motivates our intuition to tap into the activations of deep models at different depths. Deep models cover a large set of representations that can be useful, from spatially dense and simple features in the lower layers (e.g. large collection of detected edges) to gradually more spatially sparse and abstract features (e.g. few object classes).
We train a simple linear layer which takes as input the same mid to high level activations used for steps discovery and outputs a score for each step. This linear layer is trained using logistic regression and the underlying deep model is not fine-tuned. Given the large input (1,453,824 units) and the low data regime (11 to 19 videos of 30 to 50 frames each), we hypothesize that this model should severely overfit to the training data and perform poorly in validation and testing. We test and discuss that hypothesis in Section 3.1.2.
We also hypothesize that there exists a small subset of mid to high-level features that are sparse independent and can readily and compactly discriminate previously unseen inputs. We investigate that hypothesis using a simple feature selection method described in Appendix A.3. The existence of a small subset of discriminative features can be useful for reducing overfitting in low data regimes, but more importantly can allow drastic reduction of the search space for the unsupervised steps discovery. Indeed since each frame is described by millions of features, finding patterns of feature correlations across videos leads to a combinatorial explosion. However, the problem may become tractable if there exists a low-dimensional subset of features that leads to reasonably accurate steps classification. We test and discuss that hypothesis in Section 3.1.2.
2.4 USING PERCEPTUAL REWARDS FOR ROBOTIC LEARNING
In order to use our learned perceptual reward functions in a complete skill learning system, we must also choose a reinforcement learning algorithm and a policy representation. While in principle any reinforcement learning algorithm could be suitable for this task, we chose a method that is efficient for evaluation on real-world robotic systems in order to validate our approach. The method we use is based on the PI2 reinforcement learning algorithm (Theodorou et al., 2010). Our implementation, which is discussed in more detail in Appendix A.4, uses a relatively simple linear-Gaussian parameterization of the policy, which corresponds to a sequence of open-loop torque commands with fixed linear feedback to correct for perturbations. This method also requires initialization from example demonstrations to learn complex manipulation tasks efficiently. A more complex neural network policy could also be used (Chebotar et al., 2016), and more sophisticated RL algorithms could also learn skills without demonstration initialization. However, since the main purpose of this component is to validate the learned reward functions, we used this simple approach to test our rewards quickly and efficiently.
3 EXPERIMENTS
In this section, we discuss our empirical evaluation, starting with an analysis of the learned reward functions in terms of both qualitative reward structure and quantitative segmentation accuracy. We then present results for a real-world validation of our method on robotic door opening.
3.1 PERCEPTUAL REWARDS EVALUATION
We report results on two demonstrated tasks: door opening and liquid pouring. We collected about a dozen training videos for each task using a smart phone. As an example, Fig. 2 shows the entire training set used for the pouring task.
3.1.1 QUALITATIVE ANALYSIS
While a door opening sensor can be engineered using sensors hidden in the door, measuring pouring or container tilting would be quite complicated, would visually alter the scene, and is unrealistic for learning in the wild. Visual reward functions are therefore an excellent choice for complex physical phenomena such as liquid pouring. In Fig. 3, we present the combined reward functions for test
videos on the pouring task, and Fig. 10 shows the intermediate rewards for each sub-goal. We plot the predicted reward functions for both successful and failed task executions in Fig. 11. We observe that for “missed” executions where the task is only partially performed, the intermediate steps are correctly classified. Fig. 9 details qualitative results of unsupervised step segmentation for the door opening and pouring tasks. For the door task, the 2-segments splits are often quite in line with what one can expect, while a 3-segments split is less accurate. We also observe that the method is robust to the presence or absence of the handle on the door, as well as its opening direction. We find that for the pouring task, the 4-segments split often yields the most sensible break down. It is interesting to note that the 2-segment split usually occurs when the glass is about half full.
Failure Cases
The intermediate reward function for the door opening task which corresponds to a human hand manipulating the door handle seems rather noisy or wrong in 10b, 10c and 10e (”action1” on the y-axis of the plots).The reward function in 11f remains flat while liquid is being poured into the glass. The liquid being somewhat transparent, we suspect that it looks too similar to the transparent glass for the function to fire.
3.1.2 QUANTITATIVE ANALYSIS
We evaluate the quantitative accuracy of the unsupervised steps discovery in Table 1, while Table 2 presents quantitative generalization results for the learned reward on a test video of each task. For each video, ground truth intermediate steps were provided by human supervision for the purpose of evaluation. While this ground truth is subjective, since each task can be broken down in multiple ways, it is consistent for the simple tasks in our experiments. We use the Jaccard similarity measure (intersection over union) to indicate how much a detected step overlaps with its corresponding ground truth.
In Table 1, we compare our method against a random baseline. Because we assume the same step order in all demonstrations, we also order the random steps in time to provide a fair baseline. Note that the random baseline performs fairly well because the steps are distributed somewhat uniformly in time. Should the steps be much less temporally uniform, the random baseline would be expected to perform very poorly, while our method should maintain similar performance. We compare splitting between 2 and 3 steps and find that, for both tasks, 2 steps are easier to discover, probably because these tasks exhibit one strong visual change each while the other steps are more subtle. Note that our unsupervised segmentation only works when full sequences are available while our learned reward functions can be used in real-time without accessing future frames. Hence in these experiments we
evaluate the unsupervised segmentation on the training set only and evaluate the reward functions on the test set.
In Table 2, we evaluate the reward functions individually for each step on the test set. For that purpose, we binarize the reward function using a threshold of 0.5. The random baseline simply outputs true or false at each timestep. We observe that the learned feature selection and linear classifier functions outperform the baseline by about a factor of 2. It is not clear exactly what the minimum level of accuracy is required to successfully learn to perform these tasks, but we show in section 3.2.2 that the reward accuracy on the door task is sufficient to reach 100% success rate with a real robot. Individual steps accuracy details can be found in Table 3.
Surprisingly, the linear classifier performs well and does not appear to overfit on our relatively small training set. Although the feature selection algorithm is rather close to the linear classifier compared to the baseline, using feature selection to avoid avoiding is not neccesary. However the idea that a small subset of features (32 in this case) can lead to reasonable classification accuracy is verified and an important piece of information for drastically reducing the search space for future work in unsupervised steps discovery. Additionally, we show in Fig. 4 that the feature selection approach works well when the number of features n is in the region [32, 64] but collapses to 0% accuracy when n > 8192.
3.2 REAL-WORLD ROBOTIC DOOR OPENING
In this section, we aim to answer the question of whether our previously visualized reward function can be used to learn a real-world robotic motion skill. We experiment on a door opening skill, where we adapt a demonstrated door opening to a novel configuration, such as different position or orientation of the door. Following the experimental protocol in prior work (Chebotar et al., 2016), we adapt an imperfect kinesthetic demonstration which we ensure succeeds at least occasionally (about 10% of the time). These demonstrations consist only of robot poses, and do not include images. We then use a variety of different video demonstrations, which contain images but not robot poses, to learn the reward function. These videos include demonstrations with other doors, and even demonstrations provided by a human using their own body, rather than through kinesthetic teaching with the robot.
Figure 5 shows the experimental setup. We use a 7-DoF robotic arm with a two-finger gripper, and a camera placed above the shoulder, which provides monocular RGB images. For our baseline PI2 policy, we closely follow the setup of Chebotar et al. (2016) which uses an IMU sensor in the door handle to provide both a cost and feedback as part of the state of the controller. In contrast, in our approach we remove this sensor both from the state representation provided to PI2 and in our reward replace the target IMU state with the output of a deep neural network.
3.2.1 DATA
We experiment with a range of different demonstrations from which we derive our reward function, varying both the source demo (human vs robotic), the number of subgoals we extract, and the appearance of the door. We record monocular RGB images on a camera placed above the shoulder of the arm. The door is cropped from the images, and then the resulting image is re-sized such that the shortest side is 299 dimensional with preserved aspect ratio. The input into our convolutional feature extractor Szegedy et al. (2015) is the 299x299 center crop.
3.2.2 QUALITATIVE ANALYSIS
We evaluate our reward functions qualitatively by plotting our perceptual reward functions below the demonstrations with a variety of door types and demonstrators (e.g robot or human). As can be seen in Fig. 6 and in real experiments Fig. 7, we show that the reward functions are useful to a robotic arm while only showing human demonstrations as depicted in Fig. 12. Moreover we exhibit robustness variations in appearance.
3.2.3 QUANTITATIVE ANALYSIS
In comparing the success rate of visual reward versus a baseline PI2 method that uses the ground truth reward function obtained by instrumenting the door with an IMU. We run PI2 for 11 iterations with 10 sampled trajectories at each iteration. As can be seen in Fig. 7, we obtain similar convergence speeds to our baseline model, with our method also able to open the door consistently. Since our local policy is able to obtain high reward candidate trajectories, this is strong evidence that a perceptual reward could be used to train a global in same manner as Chebotar et al. (2016).
4 CONCLUSION
In this paper, we present a method for automatically identifying important intermediate goal given a few visual demonstrations of a task. By leveraging the general features learned from pre-trained deep models, we propose a method for rapidly learning an incremental reward function from human demonstrations which we successfully demonstrate on a real robotic learning task. We show that pre-trained models are general enough to be used without retraining. We also show there exists a small subset of pre-trained features that are highly discriminative even for previously unseen scenes and which can be used to reduce the search space for future work in unsupervised steps discovery. Another compelling direction for future work is to explore how reward learning algorithms can be combined with robotic lifelong learning. One of the biggest barriers for lifelong learning in the real world is the ability of an agent to obtain reward supervision, without which no learning is possible. Continuous learning using unsupervised rewards promises to substantially increase the variety and diversity of experience that is available for robotic reinforcement learning, resulting in more powerful, robust, and general robotic skills.
ACKNOWLEDGMENTS
We would like to thank Vincent Vanhoucke for helpful discussions and feedback. We would also like to thank Mrinal Kalakrishnan and Ali Yahya for indispensable guidance throughout this project.
A ALGORITHMS DETAILS
A.1 BINARY SEGMENTATION ALGORITHM
Algorithm 2 Greedy and binary algorithm similar to and utilizing Algorithm 1, where AverageStd() is a function that computes the average standard deviation over a set of frames or over a set of values, Join() is a function that joins values or lists together into a single list, n is the number of splits desired and min size is the minimum size of a split.
function BINARYSPLIT(video, start, end, n,min size, prev std = []) if n = 1 then return [], [] end if splits0, std0← SPLIT(video, start, end, 2,min size) if n = 2 then return splits0, std0 end if splits1, std1← BINARYSPLIT(video, start, splits0[0], CEIL(n/2),min size) splits2, std2← BINARYSPLIT(video, splits0[0] + 1, end, FLOOR(n/2),min size) all splits = [] all std = [] if splits1 6= [] then
JOIN(all splits, splits1) JOIN(all std, std1)
else JOIN(all std, std0[0]) end if if splits0 6= [] then
JOIN(all splits, splits0[0]) end if if splits2 6= [] then
JOIN(all splits, splits2) JOIN(all std, std2)
else JOIN(all std, std0[1]) end if return all splits, all std
end function
A.2 COMBINING INTERMEDIATE REWARDS
From the two previous sections, we obtain one reward function per intermediate step discovered by the unsupervised algorithm. These need to be combined so that the RL algorithm uses a single reward function which partially rewards intermediate steps but most rewards the final one. The initial step is ignored as it is assumed to be the resting starting state in the demonstrations. We opt for the maximum range of each reward be twice the maximum range of its preceding reward, and summing them as follow:
R(a) = n∑ i=2 Ri(a) ∗ 2(i−1) (2)
where n is the number of intermediate rewards detected and a an activations vector. An example of this combination is shown in Fig. 8.
A.3 FEATURE SELECTION ALGORITHM
Here we describe the feature selection algorithm we use to investigate the presence of a small subset of discriminative features in mid to high level layers of a pre-trained deep network. To select the most discriminative features, we use a simple scoring heuristic. Each feature i is first normalized by subtracting the mean and dividing by the standard deviation of all training sequences. We then rank them for each sub-goal according to their distance zi to the average statistics of the sets of positive and negative frames for a given goal:
zi = α ∣∣∣µ+i − µ−i ∣∣∣− (σ+i + σ−i ), (3)
where µ+i and σ + i are the mean and standard deviation of all “positive” frames and the µ − i and σ − i of all “negative” frames (the frames that do not contain the sub-goal). Only the top-M features are retained to form the reward function Rg() for the sub-goal g, which is given by the log-probability of an independent Gaussian distribution over the relevant features:
Rg(st) = 1
n M∑ j (sijt − µ+ijt) 2 σ+ijt 2 , (4)
where ij indexes the top-M selected features. We empirically choose α = 5.0 and M = 32 for our subsequent experiments. At test time, we do not know when the system transitions from one goal to another, so instead of time-indexing the goals, we instead combine all of the goals into a single time-invariant reward function, where later steps yield higher reward than earlier steps, as described in Appendix A.2.
A.4 PI2 REINFORCEMENT LEARNING ALGORITHM
We chose the PI2 reinforcement learning algorithm (Theodorou et al., 2010) for our experiments, with the particular implementation of the method based on a recently proposed deep reinforcement learning variant (Chebotar et al., 2016). Since our aim is mainly to validate that our learned reward functions capture the goals of the task well enough for learning, we employ a relatively simple linear-Gaussian parameterization of the policy, which corresponds to a sequence of open-loop torque commands with fixed linear feedback to correct for perturbations, as in the work of Chebotar et al. (2016). This policy has the form π(ut|xt) = N (Ktxt + kt,Σt), where Kt is a fixed stabilizing feedback matrix, and kt is a learned control. In this case, the state xt corresponds to the joint angles and angular velocities of a robot, and ut corresponds to the joint torques. Since the reward function is evaluated from camera images, we assume that the image is a (potentially stochastic) consequence of the robot’s state, so that we can evaluate the state reward r(xt) by taking the image
It observed at time t, and computing the corresponding activations at. Overloading the notation, we can write at = f(It(xt)), where f is the network we use for visual features. Then, we have r(xt) = R(f(It(xt))).
The PI2 algorithm is an episodic policy improvement algorithm that uses the reward r(xt) to iteratively improve the policy. The trust-region variant of PI2 that we use Chebotar et al. (2016), which is also similar to the REPS algorithm (Peters et al., 2010), updates the policy at iteration n by sampling from the time-varying linear-Gaussian policy π(ut|xt) to obtain samples {(x(i)t ,u (i) t )}, and updating the controls kt at each time step according to
kt ← ∑ i u (i) t exp βt T∑ t′=t r(x (i) t′ )   / ∑ i exp βt T∑ t′=t r(x (i) t′ )   ,
where the temperature βt is chosen to bound the KL-divergence between the new policy π(ut|xt) and the previous policy π̄(ut|xt), such that DKL(π(ut|xt)‖π̄(ut|xt)) ≤ for a step size epsilon. Further details and a complete derivation are provided in prior work Theodorou et al. (2010); Peters et al. (2010); Chebotar et al. (2016).
The PI2 algorithm is a local policy search method that performs best when provided with demonstrations to bootstrap the policy. In our experiments, we use this method together with our learned reward functions to learn a door opening skill with a real physical robot, as discussed in Section 3.2. Demonstration are provided with kinesthetic teaching, which results in a sequence of reference steps x̂t, and initial controls kt are given by kt = −Ktx̂t, such that the mean of the initial controller is Kt(xt − x̂t), corresponding to a trajectory-following initialization. This initial controller is rarely successful consistently, but the occasional successes it achieves provide a learning signal to the algorithm. The use of demonstrations enables PI2 to be used to quickly and efficiently learn complex robotic manipulation skills.
Although this particular RL algorithm requires demonstrations to begin learning, it can still provide a useful starting point for real-world learning with a real robotic system. As shown by Chebotar et al. (2016), the initial set of demonstrations can be expanded into a generalizable policy by iteratively “growing” the effective region where the policy succeeds. For example, if the robot is provided with a demonstration of opening a door in one position, additional learning can expand the policy to succeed in nearby positions, and the application of a suitable curriculum can grow the region of door poses in which the policy succeeds progressively. However, as with all RL algorithms, this process requires knowledge of the reward function. Using the method described in this paper, we can learn such a reward function from either the initial demonstrations or even from other demonstration videos provided by a human. Armed with this learned reward function, the robot could continue to improve its policy through real-world experience, iteratively increasing its region of competence through lifelong learning.
B ADDITIONAL QUALITATIVE RESULTS

The paper explores a simple approach to learning reward functions for reinforcement learning from visual observations of expert trajectories for cases were only little training data is available. To obtain descriptive rewards even under such challenging conditions the method re-uses a pre-trained neural network as feature extractor (this is similar to a large body of work on task transfer with neural nets in the area of computer vision) and represents the reward function as a weighted distance to features for automatically extracted "key-frames" of the provided expert trajectories.
The paper is well written and explains all involved concepts clearly while also embedding the presented approach in the literature on inverse reinforcement learning (IRL). The resulting algorithm is appealing due to its simplicity and could prove useful for many real world robotic applications. I have three main issues with the paper in its current form, if these can be addressed I believe the paper would be significantly strengthened:
1) Although the recursive splitting approach for extracting the "key-frames" seems reasonable and the feature selection is well motivated I am missing two baselines in the experiments:
   - what happens if the feature selection is disabled and the distance between all features is used ? will this immediately break the procedure ? If not, what is the trade-off here ?
   - an even simpler baseline than what is proposed in the paper would be the following procedure: simply use all frames of the recorded trajectories, calculate the distance to them in feature space and weights them according to their time as in the approach proposed in the paper. How well would that work ?
2) I understand the desire to combine the extracted reward function with a simple RL method but believe the used simple controller could potentially introduce a significant bias in the experiments since it requires initialization from an expert trajectory. As a direct consequence of this initialization the RL procedure is already started close to a good solution and the extracted reward function is potentially only queried in a small region around what was observed in the initial set of images (perhaps with the exception of the human demonstrations). Without an additional experiment it is thus unclear how well the presented approach will work in combination with other RL methods for training the controller.
3) I understand that the low number of available images excludes training a deep neural net directly for the task at hand but one has to wonder how other baselines would do. What happens if one uses a random projection of the images to form a feature vector? How well would a distance measure using raw images (e.g. L2 norm of image differences) or a distance measure based on the first principal components work? It seems that occlusions etc. would exclude them from working well but without empirical evidence it is hard to confirm this.
Minor issues:
- Page 1: "make use of ideas about imitation" reads a bit awkwardly
- Page 3: "We use the Inception network pre-trained ImageNet" -> pre-trained for ImageNet classification
- Page 4: the definition of the transition function for the stochastic case seems broken
- Page 6: "efficient enough to evaluate" a bit strangely written sentence
Additional comments rather than real issues:
- The paper is mainly of empirical nature, little actual learning is performed to obtain the reward function and no theoretical advances are needed. This is not necessarily bad but makes the empirical evaluation all the more important.
- While I liked the clear exposition the approach -- in the end -- boils down to computing quadratic distances to features of pre-extracted "key-frames", it is nice that you make a connection to standard IRL approaches in Section 2.1 but one could argue that this derivation is not strictly necessary.


Deep Residual Networks present a premium in performance in comparison to conventional networks of the same depth and are trainable at extreme depths. It has recently been shown that Residual Networks behave like ensembles of relatively shallow networks. We show that these ensembles are dynamic: while initially the virtual ensemble is mostly at depths lower than half the network’s depth, as training progresses, it becomes deeper and deeper. The main mechanism that controls the dynamic ensemble behavior is the scaling introduced, e.g., by the Batch Normalization technique. We explain this behavior and demonstrate the driving force behind it. As a main tool in our analysis, we employ generalized spin glass models, which we also use in order to study the number of critical points in the optimization of Residual Networks.
1 INTRODUCTION
Residual Networks (He et al., 2015) (ResNets) are neural networks with skip connections. These networks, which are a specific case of Highway Networks (Srivastava et al., 2015), present state of the art results in the most competitive computer vision tasks including image classification and object detection.
The success of residual networks was attributed to the ability to train very deep networks when employing skip connections (He et al., 2016). A complementary view is presented by Veit et al. (2016), who attribute it to the power of ensembles and present an unraveled view of ResNets that depicts ResNets as an ensemble of networks that share weights, with a binomial depth distribution around half depth. They also present experimental evidence that short paths of lengths shorter than half-depth dominate the ResNet gradient during training.
The analysis presented here shows that ResNets are ensembles with a dynamic depth behavior. When starting the training process, the ensemble is dominated by shallow networks, with depths lower than half-depth. As training progresses, the effective depth of the ensemble increases. This increase in depth allows the ResNet to increase its effective capacity as the network becomes more and more accurate.
Our analysis reveals the mechanism for this dynamic behavior and explains the driving force behind it. This mechanism remarkably takes place within the parameters of Batch Normalization (Ioffe & Szegedy, 2015), which is mostly considered as a normalization and a fine-grained whitening mechanism that addresses the problem of internal covariate shift and allows for faster learning rates.
We show that the scaling introduced by batch normalization determines the depth distribution in the virtual ensemble of the ResNet. These scales dynamically grow as training progresses, shifting the effective ensemble distribution to bigger depths.
The main tool we employ in our analysis is spin glass models. Choromanska et al. (2015a) have created a link between conventional networks and such models, which leads to a comprehensive study of the critical points of neural networks based on the spin glass analysis of Auffinger et al. (2013). In our work, we generalize these results and link ResNets to generalized spin glass models. These models allow us to analyze the dynamic behavior presented above. Finally, we apply the results of Auffinger & Arous (2013) in order to study the loss surface of ResNets.
2 A RECAP OF CHOROMANSKA ET AL. (2015A)
We briefly summarize Choromanska et al. (2015a), which connects the loss function of multilayer networks with the hamiltonian of the p spherical spin glass model, and state their main contributions and results. The notations of our paper are summarized in Appendix A and slightly differ from those in Choromanska et al. (2015a).
A simple feed forward fully connected network N , with p layers and a single output unit is considered. Let ni be the number of units in layer i, such that n0 is the dimension of the input, and np = 1. It is further assumed that the ReLU activation functions denoted by R() are used. The output Y of the network given an input vector x ∈ Rd can be expressed as
Y = d∑ i=1 γ∑ j=1 xijAij p∏ k=1 w (k) ij , (1)
where the first summation is over the network inputs x1...xd, and the second is over all paths from input to output. There are γ = ∏p i=1 ni such paths and ∀i, xi1 = xi2 = ...xiγ . The variable Aij ∈ {0, 1} denotes whether the path is active, i.e., whether all of the ReLU units along this path are producing positive activations, and the product ∏p k=1 w (k) ij represents the specific weight configurationw1ij ...w k ij multiplying xi given path j. It is assumed throughout the paper that the input variables are sampled i.i.d from a normal Gaussian distribution.
Definition 1. The mass of the network N is defined as ψ = ∏p i=0 ni.
The variablesAij are modeled as independent Bernoulli random variables with a success probability ρ, i.e., each path is equally likely to be active. Therefore,
EA[Y ] = d∑ i=1 γ∑ j=1 xijρ p∏ k=1 w (k) ij . (2)
The task of binary classification using the networkN with parameters w is considered, using either the hinge loss LhN or the absolute loss LaN :
LhN (w) = EA[max(0, 1− YxY )], LaN (w) = EA[|Yx − Y |] (3)
where Yx is a random variable corresponding to the true label of sample x. In order to equate either loss with the hamiltonian of the p-spherical spin glass model, a few key approximations are made:
A1 Variable independence - The inputs xij are modeled as independent normal Gaussian random variables.
A2 Redundancy in network parameterization - It is assumed the set of all the network weights [w1, w2...wN ] contains only Λ unique weights such that Λ < N .
A3 Uniformity - It is assumed that all unique weights are close to being evenly distributed on the graph of connections defining the network N . Practically, this means that we assume every node is adjacent to an edge with any one of the Λ unique weights.
A4 Spherical constraint - The following is assumed:
1
Λ Λ∑ i=1 w2i = C 2 (4)
for some constant C > 0.
These assumptions are made for the sake of analysis, and do not necessarily hold. The validity of these assumption was posed as an open problem in Choromanska et al. (2015b), where a different degree of plausibility was assigned to each. Specifically, A1, as well as the independence assumption of Aij , were deemed unrealistic, and A2 - A4 as plausible. For example, A1 does not hold since each input xi is associated with many different paths and xi1 = xi2 = ...xiγ . See Choromanska et al. (2015a) for further justification of these approximations.
Under A1–A4, the loss takes the form of a centered Gaussian process on the sphere SΛ−1( √
Λ). Specifically, it is shown to resemble the hamiltonian of the a spherical p-spin glass model given by:
Hp,Λ(w̃) = 1
Λ p−1 2 Λ∑ i1...ip xi1...ip r∏ k=1 w̃ik (5)
with spherical constraint 1
Λ Λ∑ i=1 w̃2i = 1 (6)
where xi1...ip are independent normal Gaussian variables.
In Auffinger et al. (2013), the asymptotic complexity of spherical p spin glass model is analyzed based on random matrix theory. In Choromanska et al. (2015a) these results are used in order to shed light on the optimization process of neural networks. For example, the asymptotic complexity of spherical spin glasses reveals a layered structure of low-index critical points near the global optimum. These findings are then given as a possible explanation to several central phenomena found in neural networks optimization, such as similar performance of large nets, and the improbability of getting stuck in a “bad” local minima.
As part of our work, we follow a similar path. First, a link is formed between residual networks and the hamiltonian of a general multi-interaction spherical spin glass model as given by:
Hp,Λ(w̃) = p∑ r=1 r Λ r−1 2 Λ∑ i1,i2...ir=1 xi1,i2...ir r∏ k=1 w̃ik (7)
where 1... p are positive constants. Then, using Auffinger & Arous (2013), we obtain insights on residual networks. The other part of our work studies the dynamic behavior of residual networks, where we relax the assumptions made for the spin glass model.
3 RESIDUAL NETS AND GENERAL SPIN GLASS MODELS
We begin by establishing a connection between the loss function of deep residual networks and the hamiltonian of the general spherical spin glass model. We consider a simple feed forward fully connected network N , with ReLU activation functions and residual connections. For simplicity of notations without the loss of generality, we assume n1 = ... = np = n. n0 = d as before. In our ResNet model, there exist p− 1 identity connections skipping a single layer each, starting from the first hidden layer. The output of layer l > 1 is given by:
Nl(x) = R(W>l Nl−1(x)) +Nl−1(x) (8) where Wl denotes the weight matrix connecting layer l− 1 with layer l. Notice that the first hidden layer has no parallel skip connection, and so N1(x) = R(W>1 x). Without loss of generality, the scalar output of the network is the sum of the outputs of the output layer p and is expressed as
Y = p∑ r=1 d∑ i=1 γr∑ j=1 x (r) ij A (r) ij r∏ k=1 w (r)(k) ij (9)
where A(r)ij ∈ {0, 1} denotes whether path j of length r is open, and ∀j, j′, r, r′ xrij = xr ′
ij′ . The residual connections in N imply that the output Y is now the sum of products of different lengths, indexed by r. Since our ResNet model attaches a skip connection to every layer except the first, 1 ≤ r ≤ p. See Sec. 6 regarding models with less frequent skip connections. Each path of length r includes r − 1 non-skip connections (those involving the first term in Eq. 8 and not the second, identity term) out of layers l = 2..p. Therefore, γr = ( p−1 r−1 ) nr. We define the following measure on the network: Definition 2. The mass of a depth r subnetwork in N is defined as ψr = dγr.
The properties of redundancy in network parameters and their uniform distribution, as described in Sec. 2, allow us to re-index Eq. 9.
Lemma 1. Assuming assumptions A2 − A4 hold, and nΛ ∈ Z, then the output can be expressed after reindexing as:
Y = p∑ r=1 Λ∑ i1,i2...ir=1 ψr Λr∑ j=1 x (j) i1,i2...ir A (j) i1,i2...ir r∏ k=1 wik . (10)
All proofs can be found in Appendix B.
Making the modeling assumption that the ReLU gates are independent Bernoulli random variables with probability ρ, we obtain that for every path of length r, EA(j)i1,i2...ir = ρ r and
EA[Y ] = p∑ r=1 Λ∑ i1,i2...ir=1 ψr Λr∑ j=1 x (j) i1,i2...ir ρr r∏ k=1 wik . (11)
In order to connect ResNets to generalized spherical spin glass models, we denote the variables:
ξi1,i2...ir = ψr Λr∑ j=1 xji1,i2...ir , x̃i1,i2...ir = ξi1,i2...ir Ex[ξ2i1,i2...ir ] 1 2
(12)
Note that since the input variables x1...xd are sampled from a centered Gaussian distribution (dependent or not), then the set of variables x̃i1,i2...ir are dependent normal Gaussian variables. Lemma 2. Assuming A2−A3 hold, and nΛ ∈ N then ∀r,i1...ir the following holds:
1 d ( ψr Λr )2 ≤ E[ξ2i1,i2...ir ] ≤ ( ψr Λr )2. (13)
We approximate the expected output EA(Y ) with Ŷ by assuming the minimal value in 13 holds such that ∀r,i1...ir E[ξ2i1,i2...ir ] = 1 d ( ψr Λr )
2. This approximation holds exactly when Λ = n, since all weight configurations of a particular length in Eq. 10 will appear the same number of times. When Λ 6= n, the uniformity assumption dictates that each configuration of weights would appear approximately equally regardless of the inputs, and the expectation values would be very close to the lower bound. The following expression for Ŷ is thus obtained:
Ŷ = p∑ r=1 ( ρ Λ )r ψr√ d Λ∑ i1,i2...ir=1 x̃i1,i2...ir r∏ k=1 wik . (14)
The independence assumption A1 was not assumed yet, and 14 holds regardless. Assuming A4 and denoting the scaled weights w̃i = 1Cwi, we can link the distribution of Ŷ to the distribution on x̃:
Ŷ = p∑ r=1 ψr√ d ( ρC Λ )r Λ∑ i1,i2...ir=1 x̃i1,i2...ir r∏ k=1 w̃ik
= z p∑ r=2 r Λ r−1 2 Λ∑ i1,i2...ir=1 x̃i1,i2...ir r∏ k=1 w̃ik (15)
where r = r = 1z ( p−1 r−1 ) (ρnC√ Λ )r and z is a normalization factor such that ∑p r=1 2 r = 1.
The following lemma gives a generalized expression for the binary and hinge losses of the network.
Lemma 3 ( Choromanska et al. (2015a)). Assuming assumptions A2 − A4 hold, then both the losses LhN (x) and LaN (x) can be generalized to a distribution of the form:
LN (x) = C1 + C2Ŷ (16)
where C1, C2 are positive constants that do not affect the optimization process.
The model in Eq. 16 has the form of a spin glass model, except for the dependency between the variables x̃i1,i2...ir . We later use an assumption similar to A1 of independence between these variables in order to link the two binary classification losses and the general spherical spin glass model. However, for the results in this section, this is not necessary.
We denote the important quantities:
β = ρnC√
Λ , r =
1
z ( p− 1 r − 1 ) βr (17)
The series ( r) p r=1 determines the weight of interactions of a specific length in the loss surface. Notice that for constant depth p and large enough β, arg maxr( r) = p. Therefore, for wide networks, where n and, therefore, β are large, interactions of order p dominate the loss surface, and the effect of the residual connections diminishes. Conversely, for constant β and a large enough p (deep networks), we have that arg maxr( r) < p, and can expect interactions of order r < p to dominate the loss. The asymptotic behavior of is captured by the following lemma:
Theorem 1. Assuming β1+β p ∈ N, we have that:
lim p→∞
1 p arg max r ( r) = β 1 + β (18)
As the next theorem shows, the epsilons are concentrated in a narrow band near the maximal value.
Theorem 2. For any α1 < β1+β < α2, and assuming α1p, α2p, β 1+β p ∈ N, it holds that:
lim p→∞ α2p∑ r=α1p 2r = 1 (19)
Thm. 2 implies that for deep residual networks, the contribution of weight products of order far away from the maximum β1+β p is negligible. The loss is, therefor, similar in complexity to that of an ensemble of potentially shallow conventional nets. The next Lemma shows that we can shift the effective depth to any value by simply controlling C.
Lemma 4. For any integer 1 ≤ k ≤ p there exists a global scaling parameter C such that arg maxr( r(C)) = k.
A simple global scaling of the weights is, therefore, enough to change the loss surface, from an ensemble of shallow conventional nets, to an ensemble of deep nets. This is illustrated in Fig. 1(a-c) for various values of β. In a common weight initialization scheme for neural networks,C = 1√
n (Orr
& Müller, 2003; Glorot & Bengio, 2010). With this initialization and Λ = n, β = ρ and the maximal weight is obtained at less than half the network’s depth limp→∞ arg maxr( r) < p 2 . Therefore, at the initialization, the loss function is primarily influenced by interactions of considerably lower order than the depth p, which facilitates easier optimization.
4 DYNAMIC BEHAVIOR OF RESIDUAL NETS
The expression for the output of a residual net in Eq. 15 provides valuable insights into the machinery at work when optimizing such models. Thm. 1 and 2 imply that the loss surface resembles that of an ensemble of shallow nets (although not a real ensemble due to obvious dependencies), with various depths concentrated in a narrow band. As noticed in Veit et al. (2016), viewing ResNets as ensembles of relatively shallow networks helps in explaining some of the apparent advantages of these models, particularly the apparent ease of optimization of extremely deep models, since deep paths barely affect the overall loss of the network. However, this alone does not explain the increase in accuracy of deep residual nets over actual ensembles of standard networks. In order to explain the improved performance of ResNets, we make the following claims:
1. The distribution of the depths of the networks within the ensemble is controlled by the scaling parameter C.
2. During training, C changes and causes a shift of focus from a shallow ensemble to deeper and deeper ensembles, which leads to an additional capacity.
3. In networks that employ batch normalization, C is directly embodied as the scale parameter λ. The starting condition of λ = 1 offers a good starting condition that involves extremely shallow nets.
For the remainder of Sec.4, we relax all assumptions, and assume that at some point in time 1 Λ ∑Λ i=1 w 2 i = C
2, and Λ = N . Using Eq. 9 for the output of the network Ŷ in Lemma. 3, the loss can be expressed:
LN (x,w) = C1 + C2 p∑ r=1 d∑ i=1 γr∑ j=1 x (r) ij A (r) ij r∏ k=1 w (r)(k) ij (20)
where C1, C2 are some constants that do not affect the optimization process. In order to gain additional insight into this dynamic mechanism, we investigate the derivative of the loss with respect to the scale parameter C. Using Eq. 9 for the output, we obtain:
∂LN (x,w) ∂C = C2 C p∑ r=1 r d∑ i=1 γr∑ j=1 x (r) ij A (r) ij r∏ k=1 w (r)(k) ij (21)
Notice that the addition of a multiplier r indicates that the derivative is increasingly influenced by deeper networks.
4.1 BATCH NORMALIZATION
Batch normalization has shown to be a crucial factor in the successful training of deep residual networks. As we will show, batch normalization layers offer an easy starting condition for the network, such that the gradients from early in the training process will originate from extremely shallow paths.
We consider a simple batch normalization procedure, which ignores the additive terms, has the output of each ReLU unit in layer l normalized by a factor σl and then is multiplied by some parameter λl. The output of layer l > 1 is therefore:
Nl(x) = λl σl R(W>l Nl−1(x)) +Nl−1(x) (22)
where σl is the mean of the estimated standard deviations of various elements in the vector R(W>l Nl−1(x)). Furthermore, a typical initialization of batch normalization parameters is to set ∀l, λl = 1. In this case, providing that units in the same layer have equal variance σl, the recursive relation E[Nl+1(x)2j ] = 1 + E[Nl(x)2j ] holds for any unit j in layer l. This, in turn, implies that the output of the ReLU units should have increasing variance σ2l as a function of depth. Multiplying the weight parameters in deep layers with an increasingly small scaling factor 1σl , effectively reduces the influence of deeper paths, so that extremely short paths will dominate the early stages of optimization. We next analyze how the weight scaling, as introduced by batch normalization, provides a driving force for the effective ensemble to become deeper as training progresses.
4.2 THE DRIVING FORCE BEHIND THE SCALE INCREASE
The analysis below focuses on a single residual connection skipping a block of one or more layers. Since it holds for each block individually, it holds also for a residual network of multiple skipped blocks of arbitrary depth.
We consider a simple network of depth p, with a single residual connection skipping p −m layers. We further assume that batch normalization is applied at the output of each ReLU unit as described in Eq. 22. We denote by l1...lm the indices of layers that are not skipped by the residual connection, and λ̂m = ∏m i=1
λli σli
, λ̂p = ∏p i=1 λi σi . Since every path of length m is multiplied by λ̂m, and every
path of length p is multiplied by λ̂p, the expression for the loss can be expressed using Eq. 20 and
ignoring constant terms:
LN (x,w) = λ̂m d∑ i=1 γm∑ j=1 x (m) ij A (m) ij r∏ k=1 w (m)(k) ij + λ̂p d∑ i=1 γp∑ j=1 x (m) ij A (p) ij p∏ k=1 w (p)(k) ij
= Lm(x,w) + Lp(x,w) (23)
We denote by ∇w the derivative operator with respect to the parameters w, and the gradient g = ∇wLN (x,w) = gm + gp evaluated at point w.
Theorem 3. Considering the loss in 23, and assuming ∂LN (x,w)∂λl = 0, then for a small learning rate 0 < µ << 1 the following hold:
1. For any λl∈l1...lm then: ∣∣∣∣λl − µ∂LN (x,w − µg)∂λl ∣∣∣∣ > |λl| (24)
2. For any λl 6∈l1...lm , if ‖gp‖22 + g>p gm > 0 then:∣∣∣∣λl − µ∂LN (x,w − µg)∂λl ∣∣∣∣ > |λl| (25)
Thm. 3 suggests that |λl| will increase for layers l that do not have skip-connections. Conversely, if layer l has a parallel skip connection, then |λl| will increase if ‖gp‖2 > ‖gm‖2, where the later condition implies that shallow paths are nearing a local minima. Notice that an increase in |λl 6∈l1...lm | results in an increase in |λ̃p|, while |λ̃m| remains unchanged, therefore shifting the balance into deeper ensembles.
This steady increase of |λl|, as predicted in our theoretical analysis, is also backed in experimental results, as depicted in Fig. 1(d). Note that the first layer, which cannot be skipped, behaves differently than the other layers. More experiments can be found in Appendix C.
It is worth noting that the mechanism for this dynamic property of residual networks can also be observed without the use of batch normalization, as a steady increase in the L2 norm of the weights, as shown in Fig. 1(e). In order to model this, consider the residual network as discussed above, without batch normalization layers. Recalling, ‖w‖2 = C √ Λ, w̃ = wC , the loss of this network is expressed as:
LN (x,w) = Cm d∑ i=1 γm∑ j=1 x (m) ij A (m) ij r∏ k=1 w̃ (m)(k) ij + C p d∑ i=1 γp∑ j=1 x (m) ij A (p) ij p∏ k=1 w̃ (p)(k) ij
= Lm(x,w) + Lp(x,w) (26)
Theorem 4. Considering the loss in 26, and assuming ∂LN (x,w)∂C = 0, then for a small learning rate 0 < µ << 1 the following hold:
∂LN (x,w − µg) ∂C ≈ −µ 1 C (m‖gm‖22 + p‖gp‖22 + (m+ p)g>p gm) (27)
Thm. 4 indicates that if either ‖gp‖2 or ‖gm‖2 is dominant (for example, near local minimas of the shallow network, or at the start of training), the scaling of the weights C will increase. This expansion will, in turn, emphasize the contribution of deeper paths over shallow paths, and increase the overall capacity of the residual network. This dynamic behavior of the effective depth of residual networks is of key importance in understanding the effectiveness of these models. While optimization starts off rather easily with gradients largely originating from shallow paths, the overall advantage of depth is still maintained by the dynamic increase of the effective depth.
5 THE LOSS SURFACE OF ENSEMBLES
We now present the results of Auffinger & Arous (2013) regarding the asymptotic complexity in the case of limΛ→∞ of the multi-spherical spin glass model given by:
H ,Λ = − ∞∑ r=2 r Λ r−1 2 Λ∑ i1,...ir=1 Jri1...ir w̃i2 ...w̃ir (28)
where Jri1...ir are independent centered standard Gaussian variables, and = ( r)r>2 are positive real numbers such that ∑∞ r=2 r2
r < ∞. A configuration w of the spin spherical spin-glass model is a vector in RΛ satisfying the spherical constraint:
1
Λ Λ∑ i=1 w2i = 1, ∞∑ r=2 2r = 1 (29)
Note that the variance of the process is independent of :
E[H2 ,Λ] = ∞∑ r=2 Λ1−r 2r( Λ∑ i=1 w2i ) r = Λ ∞∑ r=1 2r = Λ (30)
Definition 3. We define the following:
v′ = ∞∑ r=2 2rr, v ′′ = ∞∑ r=2 2rr(r − 1), α2 = v′′ + v′ − v′2 (31)
Note that for the single interaction spherical spin model α2 = 0. The index of a critical point of H ,Λ is defined as the number of negative eigenvalues in the hessian∇2H ,Λ evaluated at the critical point w. Definition 4. For any 0 ≤ k < Λ and u ∈ R, we denote the random number Crtλ,k(u, ) as the number of critical points of the hamiltonian in the set BX = {ΛX|X ∈ (−∞, u)} with index k. That is:
CrtΛ,k(u, ) = ∑
w:∇H ,Λ=0 1 {H ,Λ ∈ Λu}1
{ i(∇2H ,Λ) = k } (32)
Furthermore, define θk(u, ) = limΛ→∞ 1Λ log E[CrtΛ,k(u )]. Corollary 1.1 of Auffinger & Arous (2013) states that for any k > 0:
θk(R, ) = 1
2 log(
v′′
v′ )− v
′′ − v′
v′′ + v′ (33)
Eq. 33 provides the asymptotic mean total number of critical points with non-diverging index k. It is presumed that the SGD algorithm will easily avoid critical points with a high index that have many descent directions, and maneuver towards low index critical points. We, therefore, investigate how the mean total number of low index critical points vary as the ensemble distribution embodied in ( r)r>2 changes its shape by a steady increase in β.
Fig. 1(f) shows that as the ensemble progresses towards deeper networks, the mean amount of low index critical points increases, which might cause the SGD optimizer to get stuck in local minima. This is, however, resolved by the the fact that by the time the ensemble becomes deep enough, the loss function has already reached a point of low energy as shallower ensembles were more dominant earlier in the training. In the following theorem, we assume a finite ensemble such that∑∞ r=p+1 r2
r ≈ 0. Theorem 5. For any k ∈ N, p > 1, we denote the solution to the following constrained optimization problems:
∗ = arg max θk(R, ) s.t p∑ r=2 2r = 1 (34)
It holds that:
∗r =
{ 1, r = p
0, otherwise (35)
Thm. 5 implies that any heterogeneous mixture of spin glasses contains fewer critical points of a finite index, than a mixture in which only p interactions are considered. Therefore, for any distribution of that is attainable during the training of a ResNet of depth p, the number of critical points is lower than the number of critical points for a conventional network of depth p.
6 DISCUSSION
In this work, we use spin glass analysis in order to understand the dynamic behavior ResNets display during training and to study their loss surface. In particular, we use at one point or another the assumptions of redundancy in network parameters, near uniform distribution of network weights, independence between the inputs and the paths and independence between the different copies of the input as described in Choromanska et al. (2015a). The last two assumptions, i.e., the two independence assumptions, are deemed in Choromanska et al. (2015b) as unrealistic, while the remaining are considered plausible.
Our analysis of critical points in ensembles (Sec. 5) requires all of the above assumptions. However, Thm. 1 and 2, as well as Lemma. 4, do not assume the last assumption, i.e., the independence between the different copies of the input. Moreover, the analysis of the dynamic behavior of residual nets (Sec. 4) does not assume any of the above assumptions.
Our results are well aligned with some of the results shown in Larsson et al. (2016), where it is noted empirically that the deepest column trains last. This is reminiscent of our claim that the deeper networks of the ensemble become more prominent as training progresses. The authors of Larsson et al. (2016) hypothesize that this is a result of the shallower columns being stabilized at a certain point of the training process. In our work, we discover the exact driving force that comes into play.
In addition, our work offers an insight into the mechanics of the recently proposed densely connected networks (Huang et al., 2016). Following the analysis we provide in Sec. 3, the additional shortcut paths decrease the initial capacity of the network by offering many more short paths from input to output, thereby contributing to the ease of optimization when training starts. The driving force mechanism described in Sec. 4.2 will then cause the effective capacity of the network to increase.
Note that the analysis presented in Sec. 3 can be generalized to architectures with arbitrary skip connections, including dense nets. This is done directly by including all of the induced sub networks in Eq. 9. The reformulation of Eq. 10 would still holds, given that Ψr is modified accordingly.
7 CONCLUSION
Ensembles are a powerful model for ResNets, which unravels some of the key questions that have surrounded ResNets since their introduction. Here, we show that ResNets display a dynamic ensemble behavior, which explains the ease of training such networks even at very large depths, while still maintaining the advantage of depth. As far as we know, the dynamic behavior of the effective capacity is unlike anything documented in the deep learning literature. Surprisingly, the dynamic mechanism typically takes place within the outer multiplicative factor of the batch normalization module.
A SUMMARY OF NOTATIONS
Table 1 presents the various symbols used throughout this work and their meaning.
B PROOFS
Proof of Lemma 1. There are a total of ψr paths of length r from input to output, and a total of Λr unique r length configurations of weights. The uniformity assumption then implies that each configuration of weights is repeated ψrΛr times. By summing over the unique configurations, and re indexing the input we arrive at Eq. 10.
Proof of Lemma 2. From 12, we have that ξi1,i2...ir is defined as a sum of ψr Λr inputs. Since there are only p distinct inputs, it holds that for each ξi1,i2...ir there exists a sequence α = (αi) p i=1 ∈ N such
that ∑d i=1 αi = ψr Λr , and ξi1,i2...ir = ∑d i=1 αixi. We, therefore, have that E[ξ2i1,i2...ir ] = ‖α‖ 2 2. Note that the minimum value of E[ξ2i1,i2...ir ] is a solution to the following:
min(E[ξ2i1,i2...ir ]) = minα(‖α‖2) s.t ‖α‖1 = ψr Λr , (αi) p i=1 ∈ N, (36)
which achieves its minimal value at ∀i, αi = 1d ψr Λr . Similarly, the maximum value is achieved at αi = ψr Λr δi for some index i.
Proof of Thm. 1. We use the stirling approximation, which states limp→∞ 1p log( ( p αp ) ) = H(α), where H(α) = −αlog(α)− (1−α)log(1−α). Ignoring the constants which do not depend on α,
lim p→∞
1 p log(
( p
αp
) βαp) = H(α) + αlog(β) (37)
which achieves its maximum value at α = α∗.
Proof of Thm. 2. For brevity, we provide a sketch of the proof. It is enough to show that limp→∞ ∑α1p r=1 2 r = 0 for β < 1. Ignoring the constants in the binomial terms, we have:
lim p→∞ α1p∑ r=1 2i = lim p→∞ ∑α1p i=1 ( p r )2 β2r z2 ≤ lim p→∞ α1p ( p α1p )2 β2α1p z2 (38)
Where z2 = ∑p r=1 ( p r )2 β2r, which can be expressed using the Legendre polynomial of order p:
z2 = (1− β2)pPp( 1 + β2
1− β2 ) (39)
In order to compute the limit of Eq. 38, we use the asymptotic of the Legendre polynomial of order p for x > 1, Pp(x) ∼ 1√2πp (x+ √ x2−1)p+ 1 2
(x2−1) 1 4
. For the term in the nominator of Eq. 38 , we use the
Stirling approximation for factorials p! ∼ √
2πp(pe ) p. Substituting both approximations in Eq. 38
and taking the limit completes the proof.
Proof of Lemma 4. For simplicity, we ignore the constants in the binomial coefficient, and assume r = 1 z ( p r ) βr. Notice that for β∗ = ( p p 2 ) , we have that arg maxr( r(β ∗)) = p, arg maxr( r( 1 β∗ )) = 1 and arg maxr( r(1)) = p 2 . From the monotonicity and continuity of β
r, any value 1 ≥ k ≥ p can be attained. The linear dependency β(C) = ρnC√
Λ completes the proof.
Proof of Thm. 3. 1. Notice that by definition, layer l is not skipped by the residual connection, and therefore λl multiplies every path in the network. Therefore,
∂LN (x,w) ∂λl = 1λl (Lm(x,w) + Lp(x,w)). Using taylor series expansion:
∂LN (x,w − µg) ∂λl ≈ ∂LN (x,w) ∂λl − µ∇w ∂LN (x,w) ∂λl g (40)
Substituting∇w ∂LN (x,w)∂λl = 1 λl (gm + gp) in 40 we have:
∂LN (x,w− µgw) ∂λl ≈ 0− µ 1 λl (gm + gp) >(gm + gp) = −µ 1 λl ‖gm + gp‖22 < 0 (41)
And hence:
λl − µ ∂LN (x,w− µgw)
∂λl = λl + µ
2 1
λl ‖gm + gp‖22
= λl(1 + µ 2 1
λ2l ‖gm + gp‖22) (42)
Finally:
|λl(1 + µ2 1
λ2l ‖gm + gp‖22)| = |λl|(1 + µ2
1
λ2l ) ≥ |λl| (43)
2. Since paths of length m skip layer l, we have that∇w ∂LN (x,w)∂λl = 1 λl gp. Therefore:
∂LN (x,w − µg) ∂λl ≈ 0− µ 1 λl (gm + gp) >gp = −µ 1 λl (g>mgp + ‖gp‖22) (44)
The condition ‖gp‖2 > ‖gm‖2 implies that g>mgp + ‖gp‖22 > 0, completing the proof.
Proof of Thm 4. Notice that ∂LN (x,w)∂C = ∂LN (x,w) ∂w ∂w ∂‖w‖2
√ Λ = g>w̃ = 0, and hence the gradient
is orthogonal to the weights. We have that ∂LN (x,w)∂C = 1 C (mLm(x,w) + pLp(x,w)). Using taylor series expansion we have:
∂LN (x,w − µg) ∂C ≈ ∂LN (x,w) ∂C − µ∇w ∂LN (x,w) ∂C g (45)
For the last term we have:
∇w ∂LN (x,w)
∂C g = (mLm(x,w) + pLp(x,w))∇w
√ Λ
‖w‖2 g +
1 C (mgm + pgp) >g
= (mLm(x,w) + pLp(x,w)) w>g
C 3 2
+ 1
C (mgm + pgp)
>g = 1
C (mgm + pgp)
>g, (46)
where the last step stems from the fact thatw>g = 0. Substituting∇w ∂LN (x,w)∂C = 1 C (mgm+pgp) in 45 we have:
∂LN (x,w− µgw) ∂C ≈ 0− µ 1 C (mgm + pgp) >(gm + gp)
= −µ 1 C (m‖gp‖22 + p‖gp‖22 + (m+ p)g>p gm) (47)
Proof of Thm 5. Inserting Eq. 31 into Eq. 33 we have that:
θk(R, ) = 1
2 log(
∑p r=2
2 rr(r − 1)∑p
r=2 2 rr
)− ∑p r=2 2 rr(r − 2)∑p
r=2 2 rr
2 (48)
We denote the matrices V ′ and V ′′ such that V ′ij = rδij and V ′′ ij = r(r − 1)δij . We then have:
θk(R, ) = 1
2 log(
>V ′′ >V ′ )− >(V ′′ − V ′) >(V ′′ + V ′)
(49)
max θk(R, ) ≤ max ( 1
2 log(
>V ′′ >V ′ ))−min ( >(V ′′ − V ′) >(V ′′ + V ′) )
= 1
2 log
( maxi(V ′′ iiV ′−1 ii ) ) −mini ( (V ′′ii − V ′ii)(V ′′ii + V ′ii)−1 ) = 1
2 log(p− 1)− (1− 2 p ) = θk(R, ∗) (50)
C ADDITIONAL EXPERIMENTS
Fig. 1(d) and 1(e) report the experimental results of a straightforward setting, in which the task is to classify a mixture of 10 multivariate Gaussians in 50D. The input is therefore of size 50. The loss employed is the cross entropy loss of ten classes. The network has 10 blocks, each containing 20 hidden neurons, a batch normalization layer, and a skip connection. Training was performed on 10,000 samples, using SGD with minibatches of 50 samples.
Next, we provide additional experiments performed on the public CIFAR-10 and CIFAR-100 data sets (Krizhevsky, 2009). The public ResNet code of https://github.com/facebook/fb. resnet.torch is used for networks of depth 32.
As noted in Sec. 4.2, the dynamic behavior can be present in the Batch Normalization multiplicative coefficient or in the weight matrices themselves. In the following experiments, it seems that
until the learning rate is reduced, the dynamic behavior is manifested in the Batch Normalization multiplicative coefficients and then it moves to the convolution layers themselves. We therefore absorb the BN coefficients into the convolutional layer using the public code of https: //github.com/e-lab/torch-toolbox/tree/master/BN-absorber. Note that the multiplicative coefficient of Batch Normalization is typically refereed to as γ. However, throughout our paper, since we follow the notation of Choromanska et al. (2015a), γ refers to the number of paths. The multiplicative factor of Batch normalization appears as λ in Sec. 4.
Fig. 2 depicts the results. There are two types of plots: Fig. 2(a,c) presents for CIFAR-10 and CIFAR-100 respectively the magnitude of the various convolutional layers for multiple epochs (similar in type to Fig. 1(d) in the paper). Fig. 2(b,d) depict for the two datasets the mean of these norms over all convolutional layers as a function of epoch (similar to Fig. 1(e)).
As can be seen, the dynamic phenomenon we describe is very prominent in the public ResNet implementation when applied to these conventional datasets: the dominance of paths with fewer skip connections increases over time. Moreover, once the learning rate is reduced in epoch 81 the phenomenon we describe speeds up.
In Fig. 3 we present the multiplicative coefficient of the Batch Normalization when not absorbed. As future work, we would like to better understand why these coefficients start to decrease once the learning rate is reduced. As shown above, taking the magnitude of the convolutions into account, the dynamic phenomenon we study becomes even more prominent at this point. The change of location from the multiplicative coefficient of the Batch Normalization layers to the convolutions themselves might indicate that Batch Normalization is no longer required at this point. Indeed, Batch Normalization enables larger training rates and this shift happens exactly when the training rate is reduced. A complete analysis is left for future work.

This paper extend the Spin Glass analysis of Choromanska et al. (2015a) to Res Nets which yield the novel dynamic ensemble results for Res Nets and the connection to Batch Normalization and the analysis of their loss surface of Res Nets.
The paper is well-written with many insightful explanation of results. Although the technical contributions extend the Spin Glass model analysis of the ones by Choromanska et al. (2015a), the updated version could eliminate one of the unrealistic assumptions and the analysis further provides novel dynamic ensemble results and the connection to Batch Normalization that gives more insightful results about the structure of Res Nets.
It is essential to show this dynamic behaviour in a regime without batch normalization to untangle the normalization effect on ensemble feature. Hence authors claim that steady increase in the L_2 norm of the weights will maintain the this feature but setting for Figure 1 is restrictive to empirically support the claim. At least results on CIFAR 10 without batch normalization for showing effect of L_2 norm increase and results that support claims about Theorem 4 would strengthen the paper.
This work provides an initial rigorous framework to analyze better the inherent structure of the current state of art Res Net architectures and its variants which can stimulate potentially more significant results towards careful understanding of current state of art models (Rather than always to attempting to improve the performance of Res Nets by applying intuitive incremental heuristics, it is important to progress on some solid understanding too).