Visual servoing involves choosing actions that move a robot in response to observations from a camera, in order to reach a goal configuration in the world. Standard visual servoing approaches typically rely on manually designed features and analytical dynamics models, which limits their generalization capability and often requires extensive application-specific feature and model engineering. In this work, we study how learned visual features, learned predictive dynamics models, and reinforcement learning can be combined to learn visual servoing mechanisms. We focus on target following, with the goal of designing algorithms that can learn a visual servo using low amounts of data of the target in question, to enable quick adaptation to new targets. Our approach is based on servoing the camera in the space of learned visual features, rather than image pixels or manually-designed keypoints. We demonstrate that standard deep features, in our case taken from a model trained for object classification, can be used together with a bilinear predictive model to learn an effective visual servo that is robust to visual variation, changes in viewing angle and appearance, and occlusions. A key component of our approach is to use a sample-efficient fitted Q-iteration algorithm to learn which features are best suited for the task at hand. We show that we can learn an effective visual servo on a complex synthetic car following benchmark using just 20 training trajectory samples for reinforcement learning. We demonstrate substantial improvement over a conventional approach based on image pixels or hand-designed keypoints, and we show an improvement in sample-efficiency of more than two orders of magnitude over standard model-free deep reinforcement learning algorithms. Videos are available at http://rll.berkeley.edu/visual_servoing.
1 INTRODUCTION
Visual servoing is a classic problem in robotics that requires moving a camera or robot to match a target configuration of visual features or image intensities. Many robot control tasks that combine perception and action can be posed as visual servoing, including navigation (DeSouza & Kak, 2002; Chen et al., 2006), where a robot must follow a desired path; manipulation, where the robot must servo an end-effector or a camera to a target object to grasp or manipulate it (Malis et al., 1999; Corke, 1993; Hashimoto, 1993; Hosoda & Asada, 1994; Kragic & Christensen, 2002); and various other problems, as surveyed in Hutchinson et al. (1996). Most visual servoing methods assume access to good geometric image features (Chaumette & Hutchinson, 2006; Collewet et al., 2008; Caron et al., 2013) and require knowledge of their dynamics, which are typically obtained from domain knowledge about the system. Using such hand-designed features and models prevents exploitation of statistical regularities in the world, and requires manual engineering for each new system.
In this work, we study how learned visual features, learned predictive dynamics models, and reinforcement learning can be combined to learn visual servoing mechanisms. We focus on target following, with the goal of designing algorithms that can learn a visual servo using low amounts of
data of the target in question, so as to be easy and quick to adapt to new targets. Successful target following requires the visual servo to tolerate moderate variation in the appearance of the target, including changes in viewpoint and lighting, as well as occlusions. Learning invariances to all such distractors typically requires a considerable amount of data. However, since a visual servo is typically specific to a particular task, it is desirable to be able to learn the servoing mechanism very quickly, using a minimum amount of data. Prior work has shown that the features learned by large convolutional neural networks on large image datasets, such as ImageNet classification (Deng et al., 2009), tend to be useful for a wide range of other visual tasks (Donahue et al., 2014). We explore whether the usefulness of such features extends to visual servoing.
To answer this question, we propose a visual servoing method that uses pre-trained features, in our case obtained from the VGG network (Simonyan & Zisserman, 2014) trained for ImageNet classification. Besides the visual features, our method uses an estimate of the feature dynamics in visual space by means of a bilinear model. This allows the visual servo to predict how motion of the robot’s camera will affect the perceived feature values. Unfortunately, servoing directly on the high-dimensional features of a pre-trained network is insufficient by itself to impart robustness on the servo: the visual servo must not only be robust to moderate visual variation, but it must also be able to pick out the target of interest (such as a car that the robot is tasked with following) from irrelevant distractor objects. To that end, we propose a sample-efficient fitted Q-iteration procedure that automatically chooses weights for the most relevant visual features. Crucially, the actual servoing mechanism in our approach is extremely simple, and simply seeks to minimize the Euclidean distance between the weighted feature values at the next time step and the target. The form of the servoing policy in our approach leads to an analytic and tractable linear approximator for the Qfunction, which leads to a computationally efficient fitted Q-iteration algorithm. We show that we can learn an effective visual servo on a complex synthetic car following benchmark using just 20 training trajectory samples for reinforcement learning. We demonstrate substantial improvement over a conventional approach based on image pixels or hand-designed keypoints, and we show an improvement in sample-efficiency of more than two orders of magnitude over standard model-free deep reinforcement learning algorithms.
The environment for the synthetic car following benchmark is available online as the package CitySim3D1, and the code to reproduce our method and experiments is also available online2. Supplementary videos of all the test executions are available on the project’s website3.
2 RELATED WORK
Visual servoing is typically (but not always) performed with calibrated cameras and carefully designed visual features. Ideal features for servoing should be stable and discriminative, and much of the work on visual servoing focuses on designing stable and convergent controllers under the assumption that such features are available (Espiau et al., 2002; Mohta et al., 2014; Wilson et al., 1996). Some visual servoing methods do not require camera calibration (Jagersand et al., 1997; Yoshimi & Allen, 1994), and some recent methods operate directly on image intensities (Caron et al., 2013), but generally do not use learning to exploit statistical regularities in the world and improve robustness to distractors.
Learning is a relatively recent addition to the repertoire of visual servoing tools. Several methods have been proposed that apply ideas from reinforcement learning to directly acquire visual servoing controllers (Lampe & Riedmiller, 2013; Sadeghzadeh et al., 2015). However, such methods have not been demonstrated under extensive visual variation, and do not make use of state-of-the-art convolutional neural network visual features. Though more standard deep reinforcement learning methods (Lange et al., 2012; Mnih et al., 2013; Levine et al., 2016; Lillicrap et al., 2015) could in principle be applied to directly learn visual servoing policies, such methods tend to require large numbers of samples to learn task-specific behaviors, making them poorly suited for a flexible visual servoing algorithm that can be quickly repurposed to new tasks (e.g. to following a different object).
1https://github.com/alexlee-gk/citysim3d 2https://github.com/alexlee-gk/visual_dynamics 3http://rll.berkeley.edu/visual_servoing
Instead, we propose an approach that combines learning of predictive models with pre-trained visual features. We use visual features trained for ImageNet (Deng et al., 2009) classification, though any pre-trained features could in principle be applicable for our method, so long as they provide a suitable degree of invariance to visual distractors such as lighting, occlusion, and changes in viewpoint. Using pre-trained features allows us to avoid the need for large amounts of experience, but we must still learn the policy itself. To further accelerate this process, we first acquire a predictive model that allows the visual servo to determine how the visual features will change in response to an action. General video prediction is an active research area, with a number of complex but data-hungry models proposed in recent years (Oh et al., 2015; Watter et al., 2015; Mathieu et al., 2015; Xue et al., 2016; Lotter et al., 2016; Jia et al., 2016; Walker et al., 2016; Vondrick et al., 2016).
However, we observe that convolutional response maps can be interpreted as images and, under mild assumptions, the dynamics of image pixels during camera motion can be well approximated by means of a bilinear model (Censi & Murray, 2015). We therefore train a relatively simple bilinear model for short-term prediction of visual feature dynamics, which we can use inside a very simple visual servo that seeks to minimize the error between the next predicted feature values and a target image.
Unfortunately, simply training predictive models on top of pre-trained features is insufficient to produce an effective visual servo, since it weights the errors of distractor objects the same amount as the object of interest. We address this challenge by using an efficient Q-iteration algorithm to train the weights on the features to maximize the servo’s long-horizon reward. This method draws on ideas from regularized fitted Q-iteration (Gordon, 1995; Ernst et al., 2005; Farahmand et al., 2009) and neural fitted Q-iteration (Riedmiller, 2005) to develop a sample-efficient algorithm that can directly estimate the expected return of the visual servo without the use of any additional function approximator.
3 PROBLEM STATEMENT
Let yt be a featurization of the camera’s observations xt and let y∗ be some given goal feature map. For the purposes of this work, we define visual servoing as the problem of choosing controls ut for a fixed number of discrete time steps t as to minimize the error ‖y∗ − yt‖. We use a relatively simple gradient-based servoing policy that uses one-step feature dynamics, f : {yt,ut} → yt+1. The policy chooses the control that minimizes the distance between the goal feature map and the one-step prediction:
π(xt,x∗) = argmin u ‖y∗ − f(yt,u)‖2 . (1)
Learning this policy amounts to learning the robot dynamics and the distance metric ‖·‖. To learn the robot dynamics, we assume that we have access to a dataset of paired observations and controls xt,ut,xt+1. This data is relatively easy to obtain as it involves collecting a stream of the robot’s observations and controls. We use this dataset to learn a general visual dynamics model that can be used for any task.
To learn the distance metric, we assume that the robot interacts with the world and collects tuples of the form xt,ut, ct,xt+1,x∗. At every time step during learning, the robot observes xt and takes action ut. After the transition, the robot observes xt+1 and receives an immediate cost ct. This cost is task-specific and it quantifies how good that transition was in order to achieve the goal. At the beginning of each trajectory, the robot is given a goal observation x∗, and it is the same throughout the trajectory. We define the goal feature map to be the featurization of the goal observation. We learn the distance metric using reinforcement learning and we model the environment as a Markov Decision Process (MDP). The state of the MDP is the tuple of the current observation and the episode’s target observation, st = (xt,x∗), the action ut is the discrete-time continuous control of the robot, and the cost function maps the states and action (st,ut, st+1) to a scalar cost ct.
4 VISUAL FEATURES DYNAMICS
We learn a multiscale bilinear model to predict the visual features of the next frame given the current image from the robot’s camera and the action of the robot. An overview of the model is shown in Figure 1. The learned dynamics can then be used for visual servoing as described in Section 5.
4.1 VISUAL FEATURES
We consider both pixels and semantic features for the visual representation. We define the function h to relate the image x and its feature y = h (x). Our choice of semantic features are derived from the VGG-16 network (Simonyan & Zisserman, 2014), which is a convolutional neural network trained for large-scale image recognition on the ImageNet dataset (Deng et al., 2009). Since spatial invariance is undesirable for servoing, we remove some of the max-pooling layers and replace the convolutions that followed them with dilated convolutions, as done by Yu & Koltun (2015). The modified VGG network is shown in Figure 2. We use the model weights of the original VGG-16 network, which are publicly available as a Caffe model (Jia et al., 2014). The features that we use are the outputs of some of the intermediate convolutional layers, that have been downsampled to a 32× 32 resolution (if necessary) and standarized with respect to our training set. We use multiple resolutions of these features for servoing. The idea is that the high-resolution representations have detailed local information about the scene, while the low-resolution representations have more global information available through the image-space gradients. The features at level l of the multiscale pyramid are denoted as y(l). The features at each level are obtained from the features below through a downsampling operator d(y(l−1)) = y(l) that cuts the resolution in half.
4.2 BILINEAR DYNAMICS
The features y(l)t are used to predict the corresponding level’s features y (l) t+1 at the next time step, conditioned on the action ut, according to a prediction function f (l)(y (l) t ,ut) = ŷ (l) t+1. We use a bilinear model to represent these dynamics, motivated by prior work (Censi & Murray, 2015). In order to servo at different scales, we learn a bilinear dynamics model at each scale. We consider two variants of the bilinear model in previous work in order to reduce the number of model parameters.
The first variant uses fully connected dynamics as in previous work but models the dynamics of each channel independently. When semantic features are used, this model interprets the feature maps as
being abstract images with spatial information within a channel and different entities or factors of variation across different channels. This could potentially allow the model to handle moving objects, occlusions, and other complex phenomena.
The fully connected bilinear model is quite large, so we propose a bilinear dynamics that enforces sparsity in the parameters. In particular, we constrain the prediction to depend only on the features that are in its local spatial neighborhood, leading to the following locally connected bilinear model:
ŷ (l) t+1,c = y (l) t,c + ∑ j ( W (l) c,j ∗ y (l) t,c +B (l) c,j ) ut,j + ( W (l) c,0 ∗ y (l) t,c +B (l) c,0 ) . (2)
The parameters are the 4-dimensional tensor W (l)c,j and the matrix B (l) c,j for each channel c, scale l, and control coordinate j. The last two terms are biases that allow to model action-independent visual changes, such as moving objects. The ∗ is the locally connected operator, which is like a convolution but with untied filter weights4.
4.3 TRAINING VISUAL FEATURE DYNAMICS MODELS
The loss that we use for training the bilinear dynamics is the sum of the losses of the predicted features at each level, ∑L l=0 `
(l), where the loss for each level l is the squared `-2 norm between the predicted features and the actual features of that level, `(l) = ‖y(l)t+1 − ŷ (l) t+1‖2.
We optimize for the dynamics while keeping the feature representation fixed. This is a supervised learning problem, which we solve with ADAM (Kingma & Ba, 2014). The training set, consisting of triplets xt,ut,xt+1, was obtained by executing a hand-coded policy that moves the robot around the target with some Gaussian noise.
5 LEARNING VISUAL SERVOING WITH REINFORCEMENT LEARNING
We propose to use a multiscale representation of semantic features for servoing. The challenge when introducing multiple scales and multi-channel feature maps for servoing is that the features do not necessarily agree on the optimal action when the goal is unattainable or the robot is far away from the goal. To do well, it’s important to use a good weighing of each of the terms in the objective. Since there are many weights, it would be impractically time-consuming to set them by hand, so we resort to learning. We want the weighted one-step lookahead objective to encourage good longterm behavior, so we want this objective to correspond to the state-action value function Q. So we propose a method for learning the weights based on fitted Q-iteration.
5.1 SERVOING WITH WEIGHTED MULTISCALE FEATURES
Instead of attempting to build an accurate predictive model for multi-step planning, we use the simple greedy servoing method in Equation (1), where we minimize the error between the target and predicted features for all the scales. Typically, only a few objects in the scene are relevant, so the errors of some channels should be penalized more than others. Similarly, features at different scales might need to be weighted differently. Thus, we use a weighting w(l)c ≥ 0 per channel c and scale l:
π(xt,x∗) = argmin u ∑ c L∑ l=0 w (l) c |y(l)·,c | ∥∥∥y(l)∗,c − f (l)c (y(l)t,c,u)∥∥∥2 2 + ∑ j λju 2 j , (3)
where | · | denotes the cardinality operator and the constant 1/|y(l)·,c| normalizes the feature errors by its spatial resolution. We also use a separate weight λj for each control coordinate j. This optimization can be solved efficiently since the dynamics is linear in the controls (see Appendix A).
4 The locally connected operator, with a local neighborhood of nf × nf (analogous to the filter size in convolutions), is defined as:
(W ∗ y)kh,kw = kh+bnf/2c∑
ih=kh−bnf/2c kw+bnf/2c∑ iw=kw−bnf/2c Wkh,kw,ih−kh,iw−kwyih,iw .
5.2 Q-FUNCTION APPROXIMATION FOR THE WEIGHTED SERVOING POLICY
We choose a Q-value function approximator that can represent the servoing objective such that the greedy policy with respect to the Q-values results in the policy of Equation (3). In particular, we use a function approximator that is linear in the weight parameters θ> = [ w> λ> ] :
Qθ,b(st,u) = φ(st,u) >θ + b, φ (st,u) > =
[[ 1
|y(l)·,c| ∥∥∥y(l)∗,c − f (l)c (y(l)t,c,u)∥∥∥2 2 ]> c,l [ u2j ]> j ] .
We denote the state of the MDP as st = (xt,x∗) and add a bias b to the Q-function. The servoing policy is then simply πθ(st) = argminu Qθ,b(st,u). For reinforcement learning, we optimized for the weights θ but kept the feature representation and its dynamics fixed.
5.3 LEARNING THE Q-FUNCTION WITH FITTED Q-ITERATION
Reinforcement learning methods that learn a Q-function do so by minimizing the Bellman error:∥∥∥∥Q (st,ut)− (ct + γminu Q (st+1,u) )∥∥∥∥2
2
. (4)
In fitted Q-iteration, the agent iteratively gathers a dataset {s(i)t ,u(i)t , c(i)t , s(i)t+1}Ni of N samples according to an exploration policy, and then minimizes the Bellman error using this dataset. We use the term sampling iteration to refer to each iteration j of this procedure. At the beginning of each sampling iteration, the current policy with added Gaussian noise is used as the exploration policy.
It is typically hard or unstable to optimize for both Q-functions that appear in the Bellman error of Equation (4), so it is usually optimized by iteratively optimizing the current Q-function while keeping the target Q-function constant. However, we notice that for a given state, the action that minimizes its Q-values is the same for any non-negative scaling α of θ and for any bias b. Thus, to speed up the optimization of the Q-function, we first set α(k− 1 2 ) and b(k− 1 2 ) by jointly solving for α and b of both the current and target Q-function:
min α≥0,b
1
N N∑ i=1 ∥∥∥∥Qαθ(k−1),b (s(i)t ,u(i)t )− (c(i)t + γminu Qαθ(k−1),b (s(i)t+1,u) )∥∥∥∥2 2 + ν ‖θ‖22 . (5)
This is similar to how, in policy evaluation, state values can be computed by solving a linear system. We regularize the parameters with an `-2 penalty, weighted by ν ≥ 0. We use the term FQI iteration to refer to each iteration k of optimizing the Bellman error, and we use the notation (k− 12 ) to denote an intermediate step between iterations (k−1) and (k). The parameters θ can then be updated with θ(k− 1 2 ) = α(k− 1 2 )θ(k−1). Then, we update θ(k) and b(k) by optimizing for θ and b of the current Q-function while keeping the parameters of the target Q-function fixed:
min θ≥0,b
1
N N∑ i=1 ∥∥∥∥Qθ,b (s(i)t ,u(i)t )− (c(i)t + γminu Qθ(k− 12 ),b(k− 12 ) (s(i)t+1,u) )∥∥∥∥2 2 + ν ‖θ‖22 . (6)
A summary of the algorithm used to learn the feature weights is shown in Algorithm 1.
Algorithm 1 FQI with initialization of policy-independent parameters
1: procedure FQI(θ(0), σ2exploration, ν) 2: for s = 1, . . . , S do . sampling iterations 3: Gather dataset {s(i)t ,u(i)t , c(i)t , s(i)t+1}Ni using exploration policy N (πθ(0) , σ2exploration) 4: for k = 1, . . . ,K do . FQI iterations 5: Fit α(k− 1 2 ) and b(k− 1 2 ) using (5) 6: θ(k− 1 2 ) ← α(k− 12 )θ(k−1) 7: Fit θ(k) and b(k) using (6) 8: θ(0) ← θ(K)
6 EXPERIMENTS
We evaluate the performance of the model for visual servoing in a simulated environment. The simulated quadcopter is governed by rigid body dynamics. The robot has 4 degrees of freedom, corresponding to translation along three axis and yaw angle. This simulation is inspired by tasks in which an autonomous quadcopter flies above a city, with the goal of following some target object (e.g., a car).
6.1 LEARNING FEATURE DYNAMICS AND WEIGHTS WITH FQI
The dynamics for each of the features were trained using a dataset of 10000 samples (corresponding to 100 trajectories) with ADAM (Kingma & Ba, 2014). A single dynamics model was learned for each feature representation for all the training cars (Figure 3). This training set was generated by executing a hand-coded policy that navigates the quadcopter around a car for 100 time steps per trajectory, while the car moves around the city.
We used the proposed FQI algorithm to learn the weightings of the features and control regularizer. At every sampling iteration, the current policy was executed with Gaussian noise to gather data from 10 trajectories. All the trajectories in our experiments were up to 100 time steps long. The immediate cost received by the agent encodes the error of the target in image coordinates (details in Appendix B). Then, the parameters were iteratively updated by running K = 10 iterations of FQI. We ran the overall algorithm for only S = 2 sampling iterations and chose the parameters that achieved the best performance on 10 validation trajectories. These validation trajectories were obtained by randomly choosing 10 cars from the set of training cars and randomly sampling initial states, and executing the policy with the parameters of the current iteration. All the experiments share the same set of validation trajectories.
Feature Dynamics Observations from Test Executions Cost
6.2 COMPARISON OF FEATURE REPRESENTATIONS FOR SERVOING
We compare the servoing performance for various feature dynamics models, where the weights are optimized with FQI. We execute the learned policies on 100 test trajectories and report the average cost of the trajectory rollouts on Figure 5. The cost of a single trajectory is the (undiscounted) sum of costs ct. We test the policies with cars that were seen during training as well as with a set of novel cars (Figure 4), to evaluate the generalization of the learned dynamics and optimized policies.
The test trajectories were obtained by randomly sampling 100 cars (with replacement) from one of the two sets of cars, and randomly sampling initial states (which are different from the ones used for validation). For consistency and reproducibility, the same sampled cars and initial states were used across all the test experiments, and the same initial states were used for both sets of cars. These test trajectories were never used during the development of the algorithm or for choosing hyperparameters.
From these results, we notice that policies based on deeper VGG features, up to VGG conv4 3, generally achieve better performance. However, the deepest feature representation, VGG conv5 3, is not as suitable for approximating Q-values. We hypothesize that this feature might be too spatially invariant and it might lack the necessary spatial information to differentiate among different car positions. The policies based on pixel intensities and VGG conv5 3 features perform worse on the novel cars. However, VGG features conv1 2 through conv4 3 achieve some degree of generalization on the novel cars.
We show sample trajectories in Table 1. The policy based on pixel-intensities is susceptible to occlusions and distractor objects that appear in the target image or during executions. This is because distinguishing these occlusions and distractors from the cars cannot be done using just RGB features.
6.3 COMPARISON OF WEIGHTINGS FROM OTHER OPTIMIZATION METHODS
We compare our policy using conv4 3 feature dynamics, with weights optimized by FQI, against policies that use these dynamics but with either no feature weighting or weights optimized by other algorithms.
For the case of no weighting, we use a single feature weight w but optimize the relative weighting of the controls λ with the cross entropy method (CEM) (De Boer et al., 2005). For the other cases, we learn the weights with Trust Region Policy Optimization (TRPO) (Schulman et al., 2015). Since the servoing policy is the minimizer of a quadratic objective (Equation (3)), we represent the policy as a neural network that has a matrix inverse operation at the output. We train this network for 2 and 50 sampling iterations, and use a batch size of 4000 samples per iteration. All of these methods use the same feature representation as ours, the only difference being how the weights w and λ are chosen.
We report the average costs of these methods on the right of Figure 6. In 2 sampling iterations, the policy learned with TRPO does not improve by much, whereas our policy learned with FQI significantly outperforms the other policies. The policy learned with TRPO improves further in 50 iterations; however, the cost incurred by this policy is still about one and a half times the cost of our policy, despite using more than 100 times as many trajectories.
6.4 COMPARISON TO PRIOR METHODS
We also consider other methods that do not use the dynamics-based servoing policy that we propose. We report their average performance on the left of Figure 6.
For one of the prior methods, we train a convolutional neural network (CNN) policy end-to-end with TRPO. The policy is parametrized as a 5-layer CNN, consisting of 2 convolutional and 3 fullyconnected layers, with ReLU activations except for the output layer; the convolutional layers use
16 filters (4 × 4, stride 2) each and the first 2 fully-connected layers use 32 hidden units each. The policy takes in raw pixel-intensities and outputs controls.
This policy achieves a modest performance (although still worse than the policies based on conv4 3 feature dynamics) but it requires significantly more training samples than any of the other learningbased methods. We also trained CNN policies that take in extracted VGG features (without any dynamics) as inputs, but they perform worse (see Table 4 in the Appendix). This suggests that given a policy parametrization that is expressive enough and given a large number of training samples, it is better to directly provide the raw pixel-intensity images to the policy instead of extracted VGG features. This is because VGG features are not optimized for this task and their representation loses some information that is useful for servoing.
The other two prior methods use classical image-based visual servoing (IBVS) (Chaumette & Hutchinson, 2006) with respect to Oriented FAST and Rotated BRIEF (ORB) feature points (Rublee et al., 2011), or feature points extracted from a visual tracker. For the former, the target features consist of only the ORB feature points that belong to the car, and this specifies that the car is relevant for the task. For the tracker-based method, we use the Continuous Convolution Operator Tracker (C-COT) (Danelljan et al., 2016) (the current state-of-the-art visual tracker) to get bounding boxes around the car and use the four corners of the box as the feature points for servoing. We provide the ground truth car’s bounding box of the first frame as an input to the C-COT tracker. For all of the IBVS methods, we provide the ground truth depth values of the feature points, which are used in the algorithm’s interaction matrix5.
The first method performs poorly, in part because ORB features are not discriminative enough for some of the cars, and the target feature points are sometimes matched to feature points that are not on the car. The tracker-based method achieves a relatively good performance. The gap in performance with respect to our method is in part due to the lack of car dynamics information in the IBVS model, whereas our method implicitly incorporates that in the learned feature dynamics. It is also worth noting that the tracker-based policy runs significantly slower than our method. The open-source implementation of the C-COT tracker6 runs at about 1Hz whereas our policy based on conv4 3 features runs at about 16Hz. Most of the computation time of our method is spent computing features from the VGG network, so there is room for speedups if we use a network that is less computationally demanding.
7 DISCUSSION
Manual design of visual features and dynamics models can limit the applicability of visual servoing approaches. We described an approach that combines learned visual features with learning predictive dynamics models and reinforcement learning to learn visual servoing mechanisms. Our experiments demonstrate that standard deep features, in our case taken from a model trained for object classification, can be used together with a bilinear predictive model to learn an effective visual servo that is robust to visual variation, changes in viewing angle and appearance, and occlusions. For control we propose to learn Q-values, building on fitted Q-iteration, which at execution time allows for one-step lookahead calculations that optimize long term objectives. Our method can learn an effective visual servo on a complex synthetic car following benchmark using just 20 training trajectory samples for reinforcement learning. We demonstrate substantial improvement over a conventional approach based on image pixels or hand-designed keypoints, and we show an improvement in sample-efficiency of more than two orders of magnitude over standard model-free deep reinforcement learning algorithms.
ACKNOWLEDGEMENTS
This research was funded in part by the Army Research Office through the MAST program and the Berkeley DeepDrive consortium. Alex Lee was also supported by the NSF GRFP.
5The term interaction matrix, or feature Jacobian, is used in the visual servo literature to denote the Jacobian of the features with respect to the control.
6https://github.com/martin-danelljan/Continuous-ConvOp
A LINEARIZATION OF THE BILINEAR DYNAMICS
The optimization of Equation (3) can be solved efficiently by using a linearization of the dynamics,
f (l)c ( y (l) t,c,u ) = f (l)c ( y (l) t,c, ū ) + J (l) t,c (u − ū) = f (l)c ( y (l) t,c,0 ) + J (l) t,cu, (7)
where J (l)t,c is the Jacobian matrix with partial derivatives ∂f(l)c ∂u (y (l) t,c, ū) and ū is the linearization point. Since the bilinear dynamics are linear with respect to the controls, this linearization is exact and the Jacobian matrix does not depend on ū. Without loss of generality, we set ū = 0.
Furthermore, the bilinear dynamics allows the Jacobian matrix to be computed efficiently by simply doing a forward pass through the model. For the locally bilinear dynamics of Equation (2), the j-th column of the Jacobian matrix is given by
J (l) t,c,j =
∂f (l) c
∂uj (y
(l) t,c,0) =W (l) c,j ∗ y (l) t,c +B (l) c,j . (8)
B SERVOING COST FUNCTION FOR REINFORCEMENT LEARNING
The goal of reinforcement learning is to find a policy that maximizes the expected sum of rewards, or equivalently, a policy that minimizes the expected sum of costs. The cost should be one that quantifies progress towards the goal. We define the cost function in terms of the position of the target object (in the camera’s local frame) after the action has been taken,
c(st,ut, st+1) =  √( pxt+1 pzt+1 )2 + ( pyt+1 pzt+1 )2 + ( 1 pzt+1 − 1pz∗ )2 , if ||pt+1||2 ≥ τ and car in FOV
(T − t+ 1) c(·, ·, st), otherwise, (9)
where T is the maximum trajectory length. The episode terminates early if the camera is too close to the car (less than a distance τ ) or the car’s origin is outside the camera’s field of view (FOV). The car’s position at time t is pt = (p x t ,p y t ,p z t ) and the car’s target position is p∗ = (0, 0,p z ∗), both in the camera’s local frame (z-direction is forward). Our experiments use T = 100 and τ = 4m.
C EXPERIMENT DETAILS
C.1 TASK SETUP
The camera is attached to the vehicle slightly in front of the robot’s origin and facing down at an angle of π/6 rad, similar to a commercial quadcopter drone. The robot has 4 degrees of freedom, corresponding to translation and yaw angle. Pitch and roll are held fixed.
In our simulations, the quadcopter follows a car that drives at 1m s−1 along city roads during training and testing. The quadcopter’s speed is limited to within 10m s−1 for each translational degree of freedom, and its angular speed is limited to within π/2 rad s−1. The simulator runs at 10Hz. For each trajectory, a car is chosen randomly from a set of cars, and placed randomly on one of the roads. The quadcopter is initialized right behind the car, in the desired relative position for following. The image observed at the beginning of the trajectory is used as the goal observation.
C.2 LEARNING FEATURE DYNAMICS
The dynamics of all the features were trained using a dataset of 10000 triplets xt,ut,xt+1. The observations are 128× 128 RGB images and the actions are 4-dimensional vectors of real numbers encoding the linear and angular (yaw) velocities. The actions are normalized to between −1 and 1. The training set was generated from 100 trajectories of a quadcopter following a car around the city with some randomness. Each trajectory was 100 steps long. Only 5 training cars were shown during learning. The generation process of each trajectory is as follows: First, a car is chosen at random from the set of available cars and it is randomly placed on one of the roads. Then, the quadcopter is placed at some random position relative to the car’s horizontal pose, which is the car’s pose that has been rotated so that the vertical axis of it and the world matches. This quadcopter position is uniformly sampled in cylindrical coordinates relative to the car’s horizontal pose, with heights in the interval 12m to 18m, and azimuthal angles in the interval −π/2 rad to π/2 rad (where the origin of the azimuthal angle is the back of the car). The radii and yaw angles are initialized so that the car is in the middle of the image. At every time step, the robot takes an action that moves it towards a target pose, with some additive Gaussian noise (σ = 0.2). The target pose is sampled according to the same procedure as the initial pose, and it is sampled once at the beginning of each trajectory.
We try the fully and locally connected dynamics for pixel intensities to better understand the performance trade-offs when assuming locally connected dynamics. We do not use the latter for the semantic features since they are too high-dimensional for the dynamics model to fit in memory. The dynamics models were trained with ADAM using 10000 iterations, a batch size of 32, a learning rate of 0.001, and momentums of 0.9 and 0.999, and a weight decay of 0.0005.
C.3 LEARNING WEIGHTING OF FEATURE DYNAMICS WITH REINFORCEMENT LEARNING
We use CEM, TRPO and FQI to learn the feature weighting and report the performance of the learned policies in Table 2. We use the cost function described in Appendix B, a discount factor of γ = 0.9, and trajectories of up to 100 steps. All the algorithms used initial weights of w = 1 and λ = 1, and a Gaussian exploration policy with the current policy as the mean and a fixed standard deviation σexploration = 0.2.
For the case of unweighted features, we use CEM to optimize for a single weight w and for the weights λ. For the case of weighted features, we use CEM to optimize for the full space of parameters, but we only do that for the pixel feature dynamics since CEM does not scale for highdimensional problems, which is the case for all the VGG features. Each iteration of CEM performs a certain number of noisy evaluations and selects the top 20% for the elite set. The number of noisy evaluations per iteration was 3 times the number of parameters being optimized. Each noisy evalua-
Feature Dynamics Observations from Test Executions Cost
tion used the average sum of costs of 10 trajectory rollouts as its evaluation metric. The parameters of the last iteration were used for the final policy. The policies with unweighted features dynamics and the policies with pixel features dynamics were trained for 10 and 25 iterations, respectively.
We use TRPO to optimize for the full space of parameters for each of the feature dynamics we consider in this work. We use a Gaussian policy, where the mean is the servoing policy of Equation (3) and the standard deviation is fixed to σexploration = 0.2 (i.e. we do not learn the standard deviation). Since the parameters are constrained to be non-negative, we parametrize the TRPO policies with √ w and √ λ. We use a Gaussian baseline, where the mean is a 5-layer CNN, consisting of 2 convolutional and 3 fully connected layers, and a standard deviation that is initialized to 1. The convolutional layers use 16 filters (4 × 4, stride 2) each, the first 2 fully-connected layers use 32 hidden units each, and all the layers except for the last one use ReLU activations. The input of the baseline network are the features (either pixel intensities or VGG features) corresponding to the feature dynamics being used. The parameters of the last iteration were used for the final policy. The policies are trained with TRPO for 50 iterations, a batch size of 4000 samples per iteration, and a step size of 0.01.
We use our proposed FQI algorithm to optimize for the weights w,λ, and surpass the other methods in terms of performance on test executions, sample efficiency, and overall computation efficiency7. The updates of the inner iteration of our algorithm are computationally efficient; since the data is fixed for a given sampling iteration, we can precompute φ (st,ut) and certain terms of φ (st+1, ·). The parameters that achieved the best performance on 10 validation trajectories were used for the final policy. The policies are trained with FQI for S = 2 sampling iterations, a batch size of 10 trajectories per sampling iteration, K = 10 inner iterations per sampling iteration, and a regularization coefficient of ν = 0.1. We found that regularization of the parameters was important for the algorithm to converge. We show sample trajectories of the resulting policies in Table 3.
The FQI algorithm often achieved most of its performance gain after the first iteration. We ran additional sampling iterations of FQI to see if the policies improved further. For each iteration, we evaluated the performance of the policies on 10 validation trajectories. We did the same for the policies trained with TRPO, and we compare the learning curves of both methods in Figure 7.
7Our policy based on conv4 3 features takes around 650 s to run K = 10 iterations of FQI for a given batch size of 10 training trajectories.
C.4 LEARNING END-TO-END SERVOING POLICIES WITH TRPO
We use TRPO to train end-to-end servoing policies for various observation modalities and report the performance of the learned policies in Table 4. The policies are trained with the set of training cars, and tested on both this set and on the set of novel cars. The observation modalities that we consider are ground truth car positions (relative to the quadcopter), images of pixel intensities from the quadcopter’s camera, and VGG features extracted from those images. Unlike our method and the other experiments, no feature dynamics are explicitly learned for these experiments.
We use a Gaussian policy, where the mean is either a multi-layer perceptron (MLP) or a convolutional neural net (CNN), and the standard deviation is initialized to 1. We also use a Gaussian baseline, which is parametrized just as the corresponding Gaussian policy (but no parameters are shared between the policy and the baseline). For the policy that takes in car positions, the mean is parametrized as a 3-layer MLP, with tanh non-linearities except for the output layer; the first 2 fully connected layers use 32 hidden units each. For the other policies, each of their means is parametrized as a 5-layer CNN, consisting of 2 convolutional and 3 fully-connected layers, with ReLU non-linearities except for the output layer; the convolutional layers use 16 filters (4×4, stride 2) each and the first 2 fully-connected layers use 32 hidden units each.
The CNN policies would often not converge for several randomly initialized parameters. Thus, at the beginning of training, we tried multiple random seeds until we got a policy that achieved a relatively low cost on validation trajectories, and used the best initialization for training. The MLP policy did not have this problem, so we did not have to try multiple random initializations for it. All the policies are trained with a batch size of 4000 samples, 500 iterations, and a step size of 0.01. The parameters of the last iteration were used for the final policy.
C.5 CLASSICAL IMAGE-BASED VISUAL SERVOING
Traditional visual servoing techniques (Feddema & Mitchell, 1989; Weiss et al., 1987) use the image-plane coordinates of a set of points for control. For comparison to our method, we evaluate the servoing performance of feature points derived from bounding boxes and keypoints derived from hand-engineered features, and report the costs of test executions on Table 5.
We use bounding boxes from the C-COT tracker (Danelljan et al., 2016) (the current state-of-the-art visual tracker) and ground truth bounding boxes from the simulator. The latter is defined as the box that tightly fits around the visible portions of the car. We provide the ground truth bounding box of the first frame to the C-COT tracker to indicate that we want to track the car. We use the four corners of the box as the feature points for servoing to take into account the position and scale of the car in image coordinates.
We provide the ground truth depth values of the feature points for the interaction matrices. In classical image-based visual servoing, the control law involves the interaction matrix (also known as feature Jacobian), which is the Jacobian of the points in image space with respect to the camera’s control (see Chaumette & Hutchinson (2006) for details). The analytical feature Jacobian used in IBVS assumes that the target points are static in the world frame. This is not true for a moving car, so we consider a variant where the feature Jacobian incorporates the ground truth dynamics of the car. This amounts to adding a non-constant translation bias to the output of the dynamics function, where the translation is the displacement due to the car’s movement of the 3-dimensional point in the camera’s reference frame. Note that this is still not exactly equivalent to having the car being static since the roads have different slopes but the pitch and roll of the quadcopter is constrained to be fixed.
For the hand-crafted features, we consider SIFT (Lowe, 2004), SURF (Bay et al., 2006) and ORB (Rublee et al., 2011) keypoints. We filter out the keypoints of the first frame that does not belong to the car and use these as the target keypoints. However, we use all the keypoints for the subsequent observations.
The servoing policies based on bounding box features achieve low cost, and even lower ones if ground truth car dynamics is used. However, servoing with respect to hand-crafted feature points is significantly worse than the other methods. This is, in part, because the feature extraction and matching process introduces compounding errors. Similar results were found by Collewet & Marchand (2011), who proposed photometric visual servoing (i.e. servoing with respect to pixel intensities) and showed that it outperforms, by an order of magnitude, classical visual servoing that uses SURF features.
C.6 CLASSICAL POSITION-BASED VISUAL SERVOING
Position-based visual servoing (PBVS) techniques use poses of a target object for control (see Chaumette & Hutchinson (2006) for details). We evaluate the servoing performance of a few variants, and report the costs of test executions on Table 6.
Similar to our IBVS experiments, we consider a variant that uses the car pose of the next time step as a way to incorporate the ground truth car dynamics into the interaction matrix. Since the cost function is invariant to the orientation of the car, we also consider a variant where the policy only minimizes the translational part of the pose error.
These servoing policies, which use ground truth car poses, outperforms all the other policies based on images. In addition, the performance is more than two orders of magnitude better if ground truth car dynamics is used.

1) Summary
This paper proposes to tackle visual servoing (specifically target following) using spatial feature maps from convolutional networks pre-trained on general image classification tasks. The authors combine bilinear models of one-step dynamics of visual feature maps at multiple scales with a reinforcement learning algorithm to learn a servoing policy. This policy is learned by minimizing a regularized weighted average of distances to features predicted by the aforementioned model of visual dynamics.
2) Contributions
+ Controlled experiments in simulation quantifying the usefulness of pre-trained deep features for visual servoing.
+ Clear performance benefits with respect to many sensible baselines, including ones using ground truth bounding boxes.
+ Principled learning of multi-scale visual feature weights with an efficient trust-region fitted Q-iteration algorithm to handle the problem of distractors.
+ Good sample efficiency thanks to the choice of Q-function approximator and the model-based one-step visual feature dynamics.
+ Open source virtual city environment to benchmark visual servoing.
3) Suggestions for improvement
- More complex benchmark:
Although the environment is not just a toy synthetic one, the experiments would benefit greatly from more complex visual conditions (clutter, distractors, appearance and motion variety, environment richness and diversity, etc). At least, the realism and diversity of object appearances could be vastly improved by using a larger number of 3D car models, including more realistic and diverse ones that can be obtained from Google SketchUp for instance, and populating the environment with more distractor cars (in traffic or parked). This is important as the main desired quality of the approach is robustness to visual variations.
- End-to-end and representation learning:
Although the improvements are already significant in the current synthetic experiments, it would be interesting to measure the impact of end-to-end training (i.e. also fine-tuning the convnet), as it is possibly needed for better generalization in more challenging visual conditions. It would also allow to measure the benefit of deep representation learning for visual servoing, which would be relevant to ICLR (there is no representation learning so far, although the method can be straightforwardly adapted as the authors mention briefly).
- Reproducibility:
The formalism and algorithms are clearly explained, but there is a slightly overwhelming mass of practical tricks and implementation details described with varying levels of details throughout the paper and appendix. Grouping, simplifying, or reorganizing the exposition of these implementation details would help, but a better way would probably consist in only summarizing the most important ones in section and link to an open source implementation of the method for completeness.
- Typos:
p.2: "learning is a relative[ly] recent addition"
p.2: "be applied [to] directly learn"
4) Conclusion
In spite of the aforementioned limits of the experiments, this paper is interesting and solid, in part thanks to the excellent reply to the pre-review questions and the subsequent improved revision. This leads me to believe the authors are more than capable of following to a significant extent the aforementioned suggestions for improvement, thus leading to an even better paper.


SIGMA-DELTA QUANTIZED NETWORKS
Deep neural networks can be obscenely wasteful. When processing video, a convolutional network expends a fixed amount of computation for each frame with no regard to the similarity between neighbouring frames. As a result, it ends up repeatedly doing very similar computations. To put an end to such waste, we introduce SigmaDelta networks. With each new input, each layer in this network sends a discretized form of its change in activation to the next layer. Thus the amount of computation that the network does scales with the amount of change in the input and layer activations, rather than the size of the network. We introduce an optimization method for converting any pre-trained deep network into an optimally efficient Sigma-Delta network, and show that our algorithm, if run on the appropriate hardware, could cut at least an order of magnitude from the computational cost of processing video data.
Deep neural networks can be obscenely wasteful. When processing video, a convolutional network expends a fixed amount of computation for each frame with no regard to the similarity between neighbouring frames. As a result, it ends up repeatedly doing very similar computations. To put an end to such waste, we introduce SigmaDelta networks. With each new input, each layer in this network sends a discretized form of its change in activation to the next layer. Thus the amount of computation that the network does scales with the amount of change in the input and layer activations, rather than the size of the network. We introduce an optimization method for converting any pre-trained deep network into an optimally efficient Sigma-Delta network, and show that our algorithm, if run on the appropriate hardware, could cut at least an order of magnitude from the computational cost of processing video data.
1 INTRODUCTION
For most deep-learning architectures, the amount of computation required to process a sample of input data is independent of the contents of that data.
Natural data tends to contain a great deal of spatial and temporal redundancy. Researchers have taken advantage of such redundancy to design encoding schemes, like jpeg and mpeg, which introduce small compromises to image fidelity in exchange for substantial savings in the amount of memory required to store images and videos.
In neuroscience, it seems clear that that some kind of sparse spatio-temporal coding is going on. Koch et al. (2006) estimate that the human retina transmits 8.75Mbps, which is about the same as compressed 1080p video at 30FPS.
Thus it seems natural to think that perhaps we should be doing this in deep learning. In this paper, we propose a neural network where neurons only communicate discretized changes in their activations to one another. The computational cost of running such a network would be proportional to the amount of change in the input. Neurons send signals when the change in their input accumulates past some threshold, at which point they send a discrete “spike” notifying downstream neurons of the change. Such a system has at least two advantages over the conventional way of doing things.
1. When extracting features from temporally redundant data, it is much more efficient to communicate the changes in activation than it is to re-process each frame.
2. When receiving data asynchronously from different sources (e.g. sensors, or nodes in a distributed network) at different rates, it no longer makes sense to have a global network update. We could recompute the network with every new input, reusing the stale inputs from the other sources, but this requires doing a great deal of repeated computation for only small differences in input data. We could keep a history of all inputs and update the network periodically, but then we lose the ability to respond immediately to new inputs. Our approach gets around this ugly tradeoff by allowing for efficient approximate updates of the network given a partial update to the input data. The computational cost of the update is proportional to the effect that the new information has on the network’s state.
2 RELATED WORK
This work originated in the study of spiking neural networks, but treads into the territory of discretizing neural nets. The most closely related work is that of Zambrano and Bohte (2016). In this work, the authors describe an Adaptive Sigma-Delta modulation method, in which neurons communicate analog signals to one another by means of a “spike-encoding” mechanism, where a temporal signal is encoded into a sequence of weighted spikes and then approximately decoded as a sum of temporally-shifted exponential kernels. The authors create a scheme for being parsimonious with spikes by allowing adaptive scaling of thresholds, at the cost of sending spikes with real values attached to them, rather than the classic “all or nothing” spikes. Their work references a slightly earlier work by Yoon (2016) which reframes common neural models as forms of Asynchronous Sigma-Delta modulation. In a concurrent work, Lee et al. (2016) implement backpropagation in a similar system (but without adaptive threshold scaling), and demonstrate the best-yet performance on MNIST for networks trained with spiking models. This work postdates Diehl et al. (2015), which proposes a scheme for normalizing neuron activations so that a spiking neural network can be optimized for fast classification.
Our model contrasts with all of the above in that it is time-agnostic. Although we refer to sending “temporal differences” between neurons, our neurons have no concept of time - their is no “leak” in neuron potential, and our neurons’ behaviour only depends on the order of the inputs. Our work also separates the concepts of nonlinearity and discretization, uses units that communicate differences rather than absolute signal values, and explicitly minimizes an objective function corresponding to computational cost.
Coming from another corner, Courbariaux et al. describe a scheme for binarizing networks with the aim of achieving reductions in the amount of computation and memory required to run neural nets. They introduce a number of tricks for training binarized neural networks - a normally difficult task due to the lack of gradient information. Esser et al. (2016) use a similar binarization scheme to efficiently implement a spiking neural network on the IBM TrueNorth chip. Ardakani et al. (2015) take another approach - to approximate real-valued operations of a neural net with a sequence of stochastic integer operations, and show how these can lead to cheaper computation.
These discretization approaches differ from ours in that they do not aim to take advantage of temporal redundancy in data, but rather aim to find ways of saving computation by learning in a low-precision regime. Ideas from these works could be combined with the ideas presented in this paper.
The idea of sending quantized temporal differences has been applied to make event-based sensors, such as the Dynamic-Vision Sensor (Lichtsteiner et al., 2008), which quantize changes in analog pixelvoltages and send out pixel-change events asynchronously. The model we propose in this paper could be used to efficiently process the outputs of such sensors.
Finally, our previous work, (O’Connor and Welling, 2016) develops a method for doing backpropagation with the same type of time-agnostic spiking neurons we use here. In this paper, we do not aim to train the network from scratch, but instead focus on how we can compute efficiently by sending temporal differences between neurons.
3 THE SIGMA-DELTA NETWORK
In this Section, we describe how we start with a traditional deep neural network and apply two modifications - temporal-difference communication and rounding - to create the Sigma-Delta network. To explain the network, we follow the Figure 1 from top to bottom, starting with a standard deep network and progressing to our Sigma-Delta network. Here, we will think of the forward pass of a neural network as composition of subfunctions: f(x) = (fL ◦ ... ◦ f2 ◦ f1)(x).
3.1 TEMPORAL DIFFERENCE NETWORK
We now define “temporal difference” (∆T ) and “temporal integration” (ΣT ) modules as follows:
Algorithm 1 Temporal Difference (∆T ): 1: Internal: ~xlast ∈ Rd ← ~0 2: Input: ~x ∈ Rd 3: ~y ← ~x− ~xlast 4: ~xlast ← ~x 5: Return: ~y ∈ Rd
Algorithm 2 Temporal Integration (ΣT ): 1: Internal: ~y ∈ Rd ← ~0 2: Input: ~x ∈ Rd 3: ~y ← ~y + ~x 4: Return: ~y ∈ Rd
So that when presented with a sequence of inputs x1, ...xt, ∆T (xt) = xt − xt−1|x0=0, and ΣT (xt) =∑t τ=1 xτ . It should be noted that when we refer to “temporal differences”, we refer not to the change in the signal over time, but in the change between two inputs presented sequentially. The output of our network only depends on the value and order of inputs, not on the temporal spacing between them. This distinction only matters when dealing with asynchronous inputs such as the Dynamic Vision Sensor, (Lichtsteiner et al., 2008), which are not considered in this paper.
Since ΣT (∆T (xt)) = xt, we can insert ΣT ◦∆T pairs into the network without affecting the function. So we can re-express our network function as: f(x) = (fL◦ΣT ◦∆T ◦...◦f2◦ΣT ◦∆T ◦f1◦ΣT ◦∆T )(x). Now suppose our network consists of alternating linear functions w(x), and nonlinear functions h(x), so that f(x) = (hL ◦wL... ◦ h2 ◦w2 ◦ h1 ◦w1)(x). As before, we can harmlessly insert our ΣT ◦∆T pairs into the network. But this time, note that for a linear function w(x), the operations (ΣT , w,∆T ) all commute with one another. That is:
∆T (w(ΣT (x))) = w(∆T (ΣT (x))) = w(x) (1)
Therefore we can replace all instances of ∆T ◦ w ◦ ΣT with w, yielding f(x) = (hL ◦ ΣT ◦ wL ◦ ... ◦ ∆T ◦h2 ◦ΣT ◦w2 ◦∆T ◦h1 ◦ΣT ◦w1 ◦∆T )(x), which corresponds to the network shown in Figure 1 B. For now this is completely pointless, since we do not change the network function at all, but it will come in handy in the next section, where we discretize the output of the ∆T modules.
3.2 DISCRETIZATING THE DELTAS
When dealing with data that is naturally spatiotemporally redundant, like most video, we expect the output of the ∆T modules to be a vector with mostly low values, with some peaks corresponding to temporal transitions at certain input positions. We expect the data to have this property not only at the input layer, but even more so at higher layers, which encode higher level features (edges, object parts, class labels), which we would expect to vary more slowly over time than pixel values. If we discretize this “peaky” vector, we end up with a sparse vector of integers, which can then be used to cheaply communicate the approximate change in state of a layer to its downstream layer(s).
A sensible approach is to apply rounding before the temporal-difference operation - i.e. round the activation values and then send the temporal differences of these rounded values. It is then easy to show that the network’s function will remain identical to that of the rounding network:
ΣT (w(∆T (round(x)))) = w(ΣT (∆T (round(x)))) = w(round(x)) (2)
It’s worth noting that this is equivalent to doing discrete-time Sigma-Delta modulation to quantize the temporal differences - this connection is explained in Appendix A.
It follows from this result that our Sigma-Delta network depicted in Figure 1 D computes an identical function to that of the rounding network in Figure 1 C. In other words, the output yt of the Sigma-Delta network is solely dependent on the parameters of the network and the current input xt, and not on any of the previous inputs x1..xt−1. The amount of computation required for the update, however, depends on xt−1. Specifically, if xt is similar to xt−1, the Sigma-Delta network should require less computation to perform an update than the Rounding Network.
3.3 SPARSE DOT PRODUCT
Most of the computation in Deep Neural networks is consumed doing matrix multiplications and convolutions. The architecture we propose saves computation by translating the input to these operations into an integer array with a small L1 norm.
With sparse, low-magnitude integer input, we can compute the vector-matrix dot product efficiently by decomposing it into a sequence of vector additions. We can see this by decomposing the vector ~x ∈ Idin into a set of indices 〈(in, sn) : i ∈ [1..len(~x)], s ∈ ±1, n = [1..N ]〉, such that: ~x = ∑N n=1 sn~ein , where ein is a one-hot vector with element in hot, and N = |~x|L1 is the total L1 magnitude of the vector. We can then compute the dot-product as a series of additions, as shown in Equation 3.
u = ~x ·W : W ∈ Rdin×dout
= ( N∑ n=1 sn~ein ) ·W = N∑ n=1 ~snein ·W = N∑ n=1 sn ·Win,· (3)
Computing the dot product this way takes N · dout additions. A normal dense dot-product, by comparison, takes din · dout multiplications and (din − 1) · dout additions. This is where the energy savings come in. Horowitz (2014) estimates that on current 45nm silicon process, a 32-bit floating-point multiplication costs 3.7pJ, vs 0.9pJ for floating-point addition. With integer or fixed-point arithmetic, the difference is even more pronounced, with 3.1pJ for multiplication vs 0.1pJ for addition. This of course ignores the larger cost of processing instructions and moving memory, but gives us an idea of how these operations might compare on optimized hardware. So provided we can approximate the forward pass of a network to a satisfactory degree of precision without doing many more operations than the original network, we can potentially compute much more efficiently.
3.4 PUTTING IT ALL TOGETHER
Figure 1 visually summarizes the four types of network we have described. Inserting the temporal sum and difference modules discussed in Section 3.1 leads to the Temporal Difference Network, which
is functionally identical to the Original Network. Discretizing the output of the temporal difference modules, as discussed in Section 3.2, leads to the Sigma-Delta network. The Sigma-Delta Network is functionally equivalent to the Rounding network, except that it requires less computation per forward pass if it is fed with temporally redundant data.
4 OPTIMIZING AN EXISTING NETWORK
In this work, we do not aim to train a quantized networks from scratch, as we did in O’Connor and Welling (2016). Rather, we will take existing pretrained networks and optimize them as Sigma-Delta networks. In in our situation, we have two competing objectives: Error (with respect to a non-quantized forward pass), and Computation: the number of additions performed in a forward pass.
4.1 RESCALING OUR NEURONS
We can control the trade-off between these objectives by changing the scale of our discretization. We can thus extend our rounding function by adding a scale k ∈ R+:
round(~x, k) ≡ round(~x · k)/k (4)
This scale can either be layerwise or unitwise (in which case we have a vector of scales per layer). Higher k values will lead to higher precision, but also more computation, for the reason mentioned in Section 3.2. Note that the final division-by-k is equivalent to scaling the following weight matrix by 1 k ,. So in practice, our network functions become:
fround(x) = ( hL ◦
wL kL ◦ round ◦ ·kL ◦ ... ◦ h1 ◦ w1 k1 ◦ round ◦ ·k1
) (x) (5)
fΣ∆(x) = ( hL ◦ ΣT ◦
wL kL ◦ round ◦ ·kL ◦∆T ◦ ... ◦ h1 ◦ ΣT ◦ w1 k1 ◦ round ◦ ·k1 ◦∆T
) (x)
(6)
For the Rounding Network and the Sigma-Delta Network, respectively. By adjusting these scales kl, we can affect the tradeoff between computation and error. Note that if we use ReLU activation functions, parameters kl can simply be baked into the parameters of the network (see Appendix C.)
4.2 THE ART OF COMPROMISE
In this section, we aim to find the optimal trade-offs between Error and Computation for the Rounding Network (Network C in Figure 1). We define our loss as follows:
Lerror = D(fround(x), ftrue(x)) (7)
Lcomp = L−1∑ l=1 |sl|L1dl+1 (8)
Ltotal = Lerror + λLcomp (9)
Where D(a, b) is some scalar distance function (We use KL-divergence for softmax output layers and L2-norm otherwise), fround(x) is the output of the Rounding Network, ftrue(x) is the output of the Original Network. Lcomp is the computational loss, defined as the total number of additions required in a forward pass. Each layer performs |sl|L1dl+1 additions, where sl is the discrete output of the l’th layer, dl+1 is the dimensionality of the (l + 1)’th layer. Finally λ is the tradeoff parameter balancing the importance of the two losses.
We aim to use this loss function to optimize our layer-scales, kl to find an optimal tradeoff between accuracy and computation, given the tradeoff parameter λ.
4.3 DIFFERENTIATING THE UNDIFFERENTIABLE
We run into an obvious problem: y = round(k · x) is not differentiable with respect to our scale, k or our input, x. We get around this by using a similar method to Courbariaux et al., who in turn borrowed it from a lecture by Hinton (2012). That is, on the backward pass, when computing the gradient with respect to the error ∂Lerror∂kl , we simply pass the gradient through the rounding function in layers [l + 1, .., .L], i.e. we say ∂∂xround(x) ≈ 1.
When computing the gradient with respect to the computational cost, ∂Lcomp∂kl , we again just pass the gradient through all rounding operations in the backward pass for layers [l + 1, .., .L]. We found instabilities in training when using the computational loss of higher layers: Lcomp,l′ : l′ ∈ [l+1, ..., L], to update the scale of layer l. Since we don’t expect this term to have much effect anyway, we choose to only use the gradient of the computational cost in layer l when updating scale kl, i.e., we approximate: ∂Lcomp ∂kl ≈ ∂Lcomp,l∂kl .
Our scale parameters also must remain in the positive range, and stay well away from zero, where they can cause instability due to the division-by-k (see Equation 5). To handle this, we parametrize our scales in log-space, as κl = log(kl). Our scale-parameter update rule becomes:
∆κl = −η ( ∂Lerror ∂κl ∣∣∣∣ pass:[l+1..L] + λ ∂ ∂κl |~sl|L1dl+1 ∣∣∣∣ pass:l ) (10)
Where ~sl is the rounded signal from layer l, dl+1 is the “fan-out” (equivalent to the dimension of layer l + 1 in a fully-connected network), and pass : [l + 1..L] indicates that, on the backward pass, we simply pass the gradient through the rounding functions on layers [l + 1..L].
5 EXPERIMENTS
5.1 TOY PROBLEM: A RANDOM NETWORK
We start with a very simple toy problem to verify our method. We initialize a 2-layer (100-100-100) ReLU network with random weights using the initialization scheme proposed in Glorot and Bengio (2010), then scaled the weights by ( 1 2 , 8, 1 4 ) . The weight-rescaling does not affect the function of the network but makes it very ill-adapted for discretization (the first layer will be represented too coarsely, causing error; the second too finely, causing wasted computation). We create random input data, and use it to optimize the layer scales according to Equation 10. We verify, by comparing to a large collection of randomly drawn rescalings, that by tuning lambda we land on different places of the Pareto frontier balancing error and computation. Figure 2 shows that this is indeed the case. In this experiment, error and computation are evaluated just on the Rounding network - we test the Sigma-Delta network in the next experiment, which includes temporal data.
5.2 TEMPORAL-MNIST
In order to evaluate our network’s ability to save computation on temporal data, we create a dataset that we call “Temporal-MNIST”. This is just a reshuffling of the standard MNIST dataset so that similar frames tend to be nearby, giving the impression of a temporal sequence (see Appendix D for details). The columns of Figure 3 show eight snippets from the Temporal-MNIST dataset.
We started our experiment with a conventional ReLU network with layer sizes [784-200-200-10] pretrained on MNIST to a test-accuracy of 97.9%. We then apply the same scale-optimization procedure for the Rounding Network used in the previous experiment to find the optimal rescalings under a range of values for λ. This time, we test the learned scale parameters on both the Rounding Network and the Sigma-Delta network. We do not attempt to directly optimize the scales with respect to the amount of computation in the Sigma-Delta network - we assume that the result should be similar to that for the rounding network, but verifying this is the topic of future work.
The results of this experiment can be seen in Figure 4. We see that our discretized networks (Rounding and Sigma-Delta) converge to the error of the original network with fewer computations than are required for a forward pass of the original neural network. Note that the errors of the rounding and
Sigma-Delta networks are identical. This is a consequence of their equivalence, described in Section 3.2. Note also that the errors for all networks are identical between the MNIST and Temporal-MNIST datasets, since for all networks, the prediction function is independent of the order in which inputs are processed. We see that as expected, our Sigma-Delta network does fewer computations than the rounding network on the Temporal-MNIST dataset for the same error, because the update-mechanism of this network takes advantage of the temporal redundancy in the data.
5.3 A DEEP CONVOLUTIONAL NETWORK ON VIDEO
Our final experiment is a preliminary exploration into how Sigma Delta networks could perform on natural video data. We start with “VGG 19” - a 19 layer convolutional network, trained to recognise the 1000 ImageNet categories. The network was trained and made public by Simonyan and Zisserman (2014). We take selected videos from the ILSVRC 2015 dataset (Russakovsky et al., 2015), and apply the rescaling method from Section 4.1 to adjust the scales on a per-layer basis. We initially had some difficulty in optimizing the scale parameters of network to a stable point. The network would either fail to reduce computation when it could afford to, or reduce it to the point where the network’s function was so corrupted that error gradients would be meaningless, causing computation loss to win out and activations to drop to zero. A simple solution was to replace the rounding operation in training with addition of uniform random noise ∼ U(− 12 , 1 2 ). This seemed to prevent the network from pushing itself into a regime where all activations become zero. More work is need to understand why the addition of noise is necessary here. Figure 5 shows some preliminary results, which indicate that for video data we can get about 4-10x savings in the amount of computation required, in exchange for a modest loss in computational accuracy.
6 DISCUSSION
We have introduced Sigma-Delta Networks, which give us a new way compute the forward pass of a deep neural network. In Sigma-Delta Networks, neurons communicate not by telling other neurons about their current level of activation, but about their change in activation. By discretizing these changes, we end up with very sparse communication between layers. The more similar two consecutive inputs (xt, xt+1) are, the less computation is required to update the network. We show that, while the Sigma-Delta Network’s internal state at time-step t depends on past inputs x1..xt−1, the output yt only depends on the current input xt. We show that there is a tradeoff between the accuracy of this network (with respect to the function of a traditional deep net with the same parameters), and the amount of computation required. Finally, we propose a method to jointly optimize error and computation, given a tradeoff parameter λ that indicates how much accuracy we are willing to sacrifice in exchange for fewer computations. We demonstrate that this method substantially reduces the number of computations required to run a deep network on natural, temporally redundant data. However, we observe in our final experiment (Figure 5, bottom) that our assumption that higher-level features would be more temporally stable - and thus require less computation in our Sigma-Delta net - was not true. We suspect that if we were to train the network from scratch on temporal data, we may learn more temporally stable “slow” features, but this is a topic of future work.
A huge amount of data (eg. video, audio) comes in the form of temporal sequences, and there is an increasingly obvious need to be able to process this data efficiently. There is much to be gained by only doing processing when necessary, based on the contents of the data, and we provide one method for doing that. Further work is needed to determine whether this method would be of use on modern computing hardware, namely GPUs. The problem is that these devices are designed for large, fixed-size array operations, and tend not to be good at taking advantage of sparsity in the data, which requires many random memory accesses to parameters. Fortunately, other devices such as the the IBM TrueNorth (Cassidy et al., 2013) are being designed which keep memory close to processing, and such handle sparse data (and random memory access) much more efficiently.
This work opens up an interesting door. In asynchronous, distributed neural networks, a node may receive input from many different nodes asynchronously. Recomputing the function of the network every time a new input signals arrives may be prohibitively expensive. Our scheme deals with this by making the computational cost of an update proportional to the amount of change in the input. The next obvious step is to extend this approach to communicating changes in gradients, which may be helpful in setting up distributed, asynchronous schemes for training neural networks.
Code for our experiments can be found at: https://github.com/petered/sigma-delta/
ACKNOWLEDGMENTS
This work was supported by Qualcomm, who we’d also like to thank for discussing their past work in the field with us. We’d also like to thank fellow lab members, especially Changyong Oh and Matthias Reisser, for fruitful discussions contributing to this work.
A DELTA-HERDING PROOF
Algorithm 3 Herding
1: Internal: ~φ ∈ Rd ← ~0 2: Input: ~xt ∈ Rd 3: ~φ← ~φ+ ~xt 4: ~s← round(~φ) 5: ~φ← ~φ− ~s 6: Return: ~s ∈ Id
Algorithm 4 Delta-Herding 1: Internal: ~slast ∈ Id ← ~0 2: Input: ~xt ∈ Rd 3: ~s← round(xt) 4: ∆~s← ~s− ~slast 5: ~slast ← ~s 6: Return: ∆~s ∈ Id
In previous work (O’Connor and Welling, 2016), we used a quantization scheme which we refer to as herding for brevity and because of its relation to the deterministic sampling scheme in (Welling, 2009), but could otherwise be called Discrete-Time Bidirectional Sigma-Delta Modulation. The procedure is described in Algorithm 3. The input is summed into a potential φ over time until crossing a quantization threshold (in this case the ± 12 at which the round function changes value), and then resets. Here we prove that Algorithm 4 is equivalent to applying Algorithm 3 to the output of a temporal difference modules. i.e. herd(∆t(xt)) = ∆T (round(xt))∀t. First start by observing the following equivalence:
b = round(a)⇔ |a− b| < 1 2 : b ∈ I (11)
We can apply this to the update rule in Algorithm 3:
st = round(φt−1 + xt) ∈ I φt = (φt−1 + xt)− st
(12)
⇒ |φt| < 1
2 (13)
Now, if we unroll the two Equations 12 over time, with initial condition φ0 = 0, we see that.
φt = t∑ τ=1 xτ − t∑ τ=1 sτ : φt ∈ R, xτ ∈ R, sτ ∈ I (14)
Using Equations 13 and 11, respectively, we can say:
|φ| = ∣∣∣∣∣ t∑
τ=1
xt − t∑
τ=1
sτ ∣∣∣∣∣ < 12 (15) ⇒
t∑ τ=1 sτ = round
( t∑
τ=1
xτ
) (16)
Which can be rearranged to solve for st.
st = round
( t∑
τ=0
xτ
) − round ( t−1∑ τ=0 xτ ) (17)
Now if we receive inputs from a ∆T unit: xτ = uτ − uτ−1 with initial condition u0 = 0, then:
~st = round
( t∑
τ=0
(uτ − uτ−1) ) − round ( t−1∑ τ=0 (uτ − uτ−1) ) (18)
= round (ut)− round (ut−1) (19) = ∆T (round(ut)) (20)
Leaving us with the Delta-Herding algorithm (Algorithm 4).
Therefore, if we have a linear function w(x), and make use of Equation 1, then we can see that the following is true:
ΣT (w(herd(∆T (x)))) = ΣT (w(∆T (round(x)))) = w(ΣT (∆T (round(x)))) = w(round(x)) (21)
B CALCULATING FLOPS
When computing the number of operations required for a forward pass, we only account for the matrixproducts/convolutions (which form the bulk of computation in done by a neural network), and not hidden layer activations.
We compute the number of operations required for a forward pass of a fully connected network as follows:
For the non-discretized network, the number of flops for a single forward pass of a single data point through the network, the flop count is:
nF lopsdense = L−1∑ l=0 (dl · dl+1 + (dl − 1) · dl+1 + dl+1) = 2 L−1∑ l=0 dl · dl+1 (22)
Where dl is the dimensionality of layer l (with l = 0 indicating the input layer). The first term counts the number of multiplications, the second the number of additions for reducing the dot-product, and the third the addition of the bias.
It can be argued that this is an unfair way to count the number of computations done by the nondiscretized network because of the sparsity of the input layer (due to the zero-background of datasets like MNIST) and the hidden layers (due to ReLU units). Thus we also compute the number of operations for the non-discretized network when factoring in sparsity. The equation is:
nF lopssparse = L−1∑ l=0 ( Nl∑ i=0 ([al]i 6= 0) · dl+1 + ( Nl∑ i=0 ([al]i 6= 0)− 1 ) · dl+1 + dl+1 )
= 2 L−1∑ l=0 Nl∑ i=0 ([al]i 6= 0) · dl+1
(23)
Where al are the layer activations Nl is the number of units in layer l and ([al]i 6= 0) is 1 if unit i in layer l has nonzero activation and 0 otherwise.
For the rounding networks, we count the total absolute value of the discrete activations.
nF lopsRound = L−1∑ l=0 ( Nl∑ i=0 |[sl]i| · dl+1 + dl+1 ) (24)
Where sl is the discrete activations of layer l. This corresponds to the number of operations that would be required for doing a dot product with the “sequential addition” method described in Section 3.2.
Finally, the Sigma-Delta network required slightly fewer flops, because the bias only need to be added once (at the beginning), so its cost is amortized.
nF lopsΣ∆ = L−1∑ l=0 Nl∑ i=0 |[sl]i| · dl+1 (25)
C BAKING THE SCALES INTO THE PARAMETERS
In Section 4.1, we mention that we can “bake the scales into the parameters” for ReLU networks. Here we explain that statement.
Suppose you have a function
f(x) = k2 · h ( x · w
k1
)
If our nonlinearity h is homogeneous (i.e. k ·h(x) = h(k ·x)), as is the case for relu(x) = max(0, x), we can collapse the scales k into the parameters:
f(x) = k2 · relu(x · w/k1 + b) (26) = relu (x · w · k2/k1 + k2 · b) (27)
So that after training scales, for a given network, we can simply incorporate them into the parameters, as: w′ = w · k2/k1, and b′ = k2 · b.
D TEMPORAL MNIST
The Temporal MNIST dataset is a version of MNIST that is reshuffled so that similar frames end up being nearby. We generate this by iterating through the dataset, keeping a fixed-size buffer of candidates for the next frame. On every iteration, we compare all the candidates to the current frame, and select the closest one. The place that this winning candidate occupied in the buffer is then filled by a new sample from the dataset, and the winning candidate becomes the current frame. The process is repeated until we’ve sorted though all frames in the dataset. Code for generating the dataset can be found at: https://github.com/petered/sigma-delta/blob/master/sigma_ delta/temporal_mnist.py
E MNIST RESULTS TABLE
F HIGH-LEVEL FEATURE STABILITY
We had initially expected that, when a convolutional network is tasked with processing subsequent frames of video, high-level features would change much more slowly than the pixels and low-level features. This would give a computational advantage to our Sigma-Delta networks, whose computational cost scales with the amount of change in the feature representations. To our surprise, this appeared not to be the case. See the final plot of Figure 5. To verify that this was a property of the original convolutional network (and not somehow related our discretization scheme), we take the same snippet of video used for Figure 5 and measure the inter-frame differences. Figure 6 shows the results of this small experiment, and confirms that our initial belief - that inter-frame differences should become smaller and smaller at higher layers, was not quite correct.

This is an interesting paper about quantized networks that work on temporal difference inputs.  The basic idea is that when a network has only to process differences then this is computational much more efficient specifically with natural video data since large parts of an image would be fairly constant so that the network only has to process the informative sections of the image (video stream). This is of course how the human visual system works, and it is hence of interest even beyond the core machine learning community.
As an aside, there is a strong community interested in event-based vision such as the group of Tobi Delbrück, and it might be interesting to connect to this community. This might even provide a reference for your comments on page 1.
I guess the biggest novel contribution is that a rounding network can be replaced by a sigma-delta network, but that the order of discretization and summation doe make some difference in the actual processing load. I think I followed the steps and
Most of my questions have already been answers in the pre-review period. My only question remaining is on page 3, “It should be noted that when we refer to “temporal differences”, we refer not to the change in the signal over time, but in the change between two inputs presented sequentially. The output of our network only depends on the value and order of inputs, not on the temporal spacing between them.”
This does not make sense to me. As I understand you just take the difference between two frames regardless if you call this temporal or not it is a change in one frame. So this statement rather confuses me and maybe should be dropped unless I do miss something here, in which case some more explanation would be necessary.
Figure 1 should be made bigger.
An improvement of the paper that I could think about is a better discussion of the relevance of the findings. Yes, you do show that your sigma-delta network save some operation compared to threshold, but is this difference essential for a specific task, or does your solution has relevance for neuroscience?