summaries: The authors propose an algorithm for transferring policies across domains differing in their transition model. The idea is to "regularize" the target policy being learned to generate a trajectory distribution similar to the one induced by the source optimal policy in the respective MDP. The method is proved effective in standard simulated robotic tasks (Mujoco).

summaries: This paper addresses the actively studied problem of efficiently transferring policies across domains in reinforcement learning. Authors propose a framework to transfer policies between tasks in domains with significantly different state transition. The proposed algorithm is based on a policy adaptation mechanism, with the idea that provided that a source optimal policy of a task is available, that policy is adapted to derive the optimal policy of the target task at a low sample complexity.

summaries: This paper tackles the problem of transferring a policy from source to target MDP, which differ in the state transition function. The idea is to add an additional cost that is the KL divergence between the trajectory likelihood under target policy (being learned) and target dynamics and the trajectory likelihood under the source policy (assumed optimal and deterministic) and source dynamics. The intuition is that the target policy will learn to match the state distribution of the optimal source policy. Results on MuJoCo locomotion robots with varying physics show that the proposed method performs better on target than warm-started RL or learning from scratch.

abstractText: Efficient and robust policy transfer remains a key challenge in reinforcement learning. Policy transfer through warm initialization, imitation, or interacting over a large set of agents with randomized instances, have been commonly applied to solve a variety of Reinforcement Learning (RL) tasks. However, this is far from how behavior transfer happens in the biological world: Humans and animals are able to quickly adapt the learned behaviors between similar tasks and learn new skills when presented with new situations. Here we seek to answer the question: Will learning to combine adaptation reward with environmental reward lead to a more efficient transfer of policies between domains? We introduce a principled mechanism that can “Adapt-to-Learn”, that is adapt the source policy to learn to solve a target task with significant transition differences and uncertainties. We show through theory and experiments that our method leads to a significantly reduced sample complexity of transferring the policies between the tasks.

authors:
Haitham B Ammar
Karl Tuyls
Matthew E Taylor
Kurt Driessens
Gerhard Weiss

title: Reinforcement learning transfer via sparse coding

venue: In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume

authors:
Haitham Bou Ammar
Eric Eaton
Paul Ruvolo
Matthew E Taylor

title: Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment

venue: In Proc. of AAAI,

authors:
Karl J Åström
Björn Wittenmark

title: Adaptive control

venue: Courier Corporation,

authors:
Karl J Åström
Richard E Klein
Anders Lennartsson

title: Bicycle dynamics and control

venue: IEEE Control Systems Magazine,

authors:
Bikramjit Banerjee
Peter Stone

title: General game learning using knowledge transfer

venue: In IJCAI, pp

authors:
Andrew G Barto
Richard S Sutton
Charles W Anderson

title: Neuronlike adaptive elements that can solve difficult learning control problems

venue: IEEE transactions on systems, man, and cybernetics,

authors:
Girish Chowdhary
Tongbin Wu
Mark Cutler
Jonathan P How

title: Rapid transfer of controllers between uavs using learning-based adaptive control

venue: In Robotics and Automation (ICRA),

authors:
Paul Christiano
Zain Shah
Igor Mordatch
Jonas Schneider
Trevor Blackwell
Joshua Tobin
Pieter Abbeel
Wojciech Zaremba

title: Transfer from simulation to real world through learning deep inverse dynamics model

venue: arXiv preprint arXiv:1610.03518,

authors:
Shreyansh Daftry
J Andrew Bagnell
Martial Hebert

title: Learning transferable policies for monocular reactive mav control

venue: In International Symposium on Experimental Robotics,

authors:
Yan Duan
Marcin Andrychowicz
Bradly Stadie
OpenAI Jonathan Ho
Jonas Schneider
Ilya Sutskever
Pieter Abbeel
Wojciech Zaremba

title: One-shot imitation learning

venue: In Advances in neural information processing systems,

authors:
Mitch J Fryling
Cristin Johnston
Linda J Hayes

title: Understanding observational learning: An interbehavioral approach

venue: The Analysis of Verbal Behavior,

authors:
Nicolas Heess
Srinivasan Sriram
Jay Lemmon
Josh Merel
Greg Wayne
Yuval Tassa
Tom Erez
Ziyu Wang
Ali Eslami
Martin Riedmiller

title: Emergence of locomotion behaviours in rich environments

venue: arXiv preprint arXiv:1707.02286,

title: Pac reinforcement learning with an imperfect model

authors:
Girish Joshi
Girish Chowdhary

title: Cross-domain transfer in reinforcement learning using target apprentice

venue: IEEE International Conference on Robotics and Automation (ICRA),

authors:
John W Krakauer
Pietro Mazzoni

title: Human sensorimotor learning: adaptation, skill, and beyond

venue: Current opinion in neurobiology,

authors:
Sergey Levine
Vladlen Koltun

title: Guided policy search

venue: In International Conference on Machine Learning, pp

authors:
Sergey Levine
Nolan Wagener
Pieter Abbeel

title: Learning contact-rich manipulation skills with guided policy search

authors:
Sergey Levine
Peter Pastor
Alex Krizhevsky
Deirdre Quillen

title: Learning hand-eye coordination for robotic grasping with large-scale data collection

authors:
Libin Liu
Jessica Hodgins

title: Learning to schedule control fragments for physics-based characters using deep q-learning

venue: ACM Transactions on Graphics (TOG),

authors:
Anusha Nagabandi
Gregory Kahn
Ronald S Fearing
Sergey Levine

title: Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning

authors:
Xue Bin Peng
Glen Berseth
Michiel Van de Panne

title: Terrain-adaptive locomotion skills using deep reinforcement learning

authors:
Xue Bin Peng
Marcin Andrychowicz
Wojciech Zaremba
Pieter Abbeel

title: Sim-to-real transfer of robotic control with dynamics randomization

venue: arXiv preprint arXiv:1710.06537,

authors:
Xue Bin Peng
Glen Berseth
KangKang Yin
Michiel Van De Panne

title: Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning

authors:
Jan Peters
Stefan Schaal

title: Policy gradient methods for robotics

venue: In Intelligent Robots and Systems,

authors:
Jette Randlov
Preben Alstrom

title: Learning to drive a bicycle using reinforcement learning and shaping

venue: In Proceedings of the Fifteenth International Conference on Machine Learning,

authors:
Stéphane Ross
Geoffrey Gordon
Drew Bagnell

title: A reduction of imitation learning and structured prediction to no-regret online learning

venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,

authors:
John Schulman
Sergey Levine
Pieter Abbeel
Michael Jordan
Philipp Moritz

title: Trust region policy optimization

venue: In International Conference on Machine Learning,

authors:
John Schulman
Filip Wolski
Prafulla Dhariwal
Alec Radford
Oleg Klimov

title: Proximal policy optimization algorithms

venue: arXiv preprint arXiv:1707.06347,

authors:
Richard S Sutton
David A McAllester
Satinder P Singh
Yishay Mansour

title: Policy gradient methods for reinforcement learning with function approximation

authors:
Matthew E Taylor
Peter Stone

title: Transfer learning for reinforcement learning domains: A survey

venue: Journal of Machine Learning Research,

authors:
Matthew E Taylor
Peter Stone
Yaxin Liu

title: Value functions for rl-based behavior transfer: A comparative study

venue: In Proceedings of the National Conference on Artificial Intelligence,

authors:
Emanuel Todorov
Tom Erez
Yuval Tassa

title: Mujoco: A physics engine for model-based control

venue: In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems,

authors:
Chang Wang
Sridhar Mahadevan

title: Manifold alignment without correspondence

venue: In IJCAI,

authors:
Markus Wulfmeier
Ingmar Posner
Pieter Abbeel

title: Mutual alignment transfer learning

venue: arXiv preprint arXiv:1707.07907,

authors:
Mengyuan Yan
Iuri Frosio
Stephen Tyree
Jan Kautz

title: Sim-to-real transfer of accurate grasping

authors:
Henry Zhu
Abhishek Gupta
Aravind Rajeswaran
Sergey Levine
Vikash Kumar

title: eye-in-hand observations and continuous control

venue: arXiv preprint arXiv:1712.03303,

heading: 1 INTRODUCTION

text:
Lack of principled mechanisms to quickly and efficiently transfer policies learned between domains has become the major bottleneck in Reinforcement Learning (RL). This inability to transfer or adapt policies is one major reason why RL has still not proliferated physical application like robotics. Since RL agents cannot quickly transfer policies, the agent is forced to learn every task from scratch, which is both time and sample expensive. Warm-start, a method in which weights from one neural network are transferred to another, has been reasonably successful for supervised learning. However, this method can often lead to mixed and even negative results in RL (Joshi & Chowdhary, 2018; Taylor & Stone, 2009).
Our main contribution is an algorithm to transfer policies between tasks with significant differences in state transitions via a policy adaptation mechanism. Unlike the majority of existing work in transfer learning for RL, our approach does not merely use the transferred policy to warm start (initialize the parameter of the target network with learned source network) policy learning in the target domain. Neither does it rely on a multitude of simulations across randomly generated source domains. Instead, we combine supervised reference trajectory tracking and unsupervised reinforcement learning to adapt the source policy to the target domain directly. We show through theory and experiments that our method enjoys significantly reduced sample complexity in solving the task.
Adapt-to-Learn is inspired by the fact that combined adaptation of behaviors and learning through experience is a primary mechanism of learning in biological creatures (Krakauer & Mazzoni, 2011; Fryling et al., 2011).Inspired by this ability of biological creatures, we seek to answer the question: Will learning to combine intrinsic adaptation reward with environment reward lead to more efficient transfer of policies between domains? Imitation Learning (IL) (Duan et al., 2017; Zhu et al., 2018) seems to play a crucial part in biological learning, and as such has been widely studied in RL. However, the key is, when presented with a new situation, animals do not just imitate, but quickly adapt existing behaviors, and improve them through further experience. In particular, an animal learning to walk on a different terrain does not just imitate its existing gait, but adapts it to the new environment.The theory behind such adaptation in reference tracking control problems has been typically restricted to deterministic dynamical systems with well-defined reference trajectories (Åström & Wittenmark, 2013; Chowdhary et al., 2013). This ability to adapt and incorporate further learning through optimization on the environment reward is one key difference between our method and ex-
isting imitation learning and Guided Policy Search (GPS) methods (Levine & Koltun, 2013). Unlike IL and GPS, our method transfers policies between task with significant differences in the transition models. Moreover, by mixing environment reward with intrinsic adaptation rewards, we ensure that the agent quickly adapts and also learns to acquire skills beyond what the source policy can teach. We posit that the presented method can be the foundation of a broader class of RL algorithms that can choose seamlessly between learning through RL to supervised adaptive imitation. Our empirical results show that approach is capable of robustly transferring policies between tasks, even in the presence of nonlinear and time-varying differences in the dynamic model of the systems. In particular, we show that it suffices to execute adapted greedy policies to ensure −optimal behavior in the target domain.
Related work: D-RL has recently enabled agents to learn policies for complex robotic tasks in simulation (Peng et al., 2016; 2017b; Liu & Hodgins, 2017; Heess et al., 2017). However, D-RL has been plagued by the curse of sample complexity. Therefore, the capabilities demonstrated in the simulated environment are hard to replicate in the real world. This learning inefficiency of RL has led to significant work in the field of TL (Taylor & Stone, 2009). A significant body of literature on transfer in RL is focused on initialized RL in the target domain using learned source policy; known as jump-start/warm-start methods (Taylor et al., 2005; Ammar et al., 2012; 2015). Some examples of these transfer architectures include transfer between similar tasks (Banerjee & Stone, 2007), transfer from human demonstrations (Peters & Schaal, 2006) and transfer from simulation to real (Peng et al., 2017a; Ross et al., 2011; Yan et al., 2017). Efforts have also been made in exploring accelerated learning directly on real robots, through Guided Policy Search (GPS) (Levine et al., 2015) and parallelizing the training across multiple agents using meta-learning (Levine et al., 2016; Nagabandi et al., 2018; Zhu et al., 2018). Sim-to-Real transfers have been widely adopted in the recent works and can be viewed as a subset of same domain transfer problems. Daftry et al. (Daftry et al., 2016) demonstrated the policy transfer for control of aerial vehicles across different vehicle models and environments. Christiano et al. (Christiano et al., 2016) transferred policies from simulation to real using an inverse dynamics model estimated interacting with the real robot. Through learning over an adversarial loss, the agents are trained to achieve robust policies across various environments (Wulfmeier et al., 2017). However, these and other reported architectures do not necessarily lead to improved sample efficiency, handle relatively minor changes in the transition model, and are even known to cause negative transfer. In contrast, our approach directly adapts the source policies to target with significant transition model difference while interacting with the environment. It enjoys empirically and rigorously proven sample efficiency guarantees of order O(nH), depending polynomially on the horizon length “H”.

heading: 2 PRELIMINARIES

text:
Consider a finite horizon MDP defined as a tupleM = (S,A,P,R, ρ0, γ), where S denote set of continuous states;A is a set of continuous bounded actions, P : S ×A×S → R+ is state transition probability distribution of reaching s′ upon taking action a in s, ρ0 : S → R+ is the distribution over initial states s0 and R : S × A → R+ is deterministic reward function and H be the finite horizon of the problem.
Let π(a|s) : S ×A → [0, 1] be stochastic policy over continuous state and action space. The action from policy is a draw from this distribution ai ∼ π(ai|si). The agent’s goal is to find a policy π? which maximize the total return. The total return starting from states s0 ∼ ρ0 under a policy π is
ηπ(s0) = Es0,a0,... ( H∑ t=0 r(st, at) ) (1)
where, s0 ∼ ρ0, at ∼ π(at|st) and st+1 ∼ p(st+1|st, at). We formalize the underlying problem of policy transfer by considering a source and target MDP as follows,MS = (S,A,P,R, ρ0, γ)S ,MT = (S,A,P,R, ρ0, γ)T , each with its own state, action space and transition model respectively. We will mainly focus on the problem of same domain transfer in this paper, where the state and action space are analogous S(S) = S(T ) = S ∈ Rm and A(T ) = A(S) = A ∈ Rk, but the source and target state transition models differ significantly due to unmodeled dynamics or external environment interactions. Furthermore, an example of how the method can extend to cross-domain transfer using manifold alignment (Joshi & Chowdhary,
2018; Wang & Mahadevan, 2009) is in the supplementary. Let π∗ be a parameterized optimal stochastic policy for source MDP MS . The policy π∗ can be obtained using any available RL methods (Sutton et al., 2000; Schulman et al., 2015; 2017). In this work, we have used the Proximal Policy Optimization(PPO) (Schulman et al., 2017) algorithm to generate the source optimal policy. We will use the following definition of the state value function V π defined under any policy π
V π(st) = Eat,st+1,at+1,... ( H∑ i=0 r(si+t, ai+t) ) The associated optimal value function fo source MDPMS is V π∗ for all s ∈ S. Let T : Rn → Rn be the Bellman update operator defined as
(T f)(s) = max a∈A
[ r(s, a) + γ E
s′∼P (s,a) f(s′)
] (2)
Optimal value function V π ∗ satisfies the Bellman Equation-(2) such that V π ∗ (s) = T V π∗(s),∀ s.

heading: 3 ALGORITHMIC CONTRIBUTION: TRANSFER THROUGH ADAPTATION

heading: 3.1 ADAPTATION AS A MECHANISM FOR POLICY TRANSFER IN RL

text:
In this paper, we approach the problem of transfer learning for RL through adaptation of previously learned policies to related tasks. Our approach to adaptation is by enabling the agent to learn the best mixture of imitation and learning from environment reward.Our method differs from RL transfer methods that rely on jump-starts or direct imitation (Zhu et al., 2018; Duan et al., 2017) in a key way: We do not aim to emulate the source optimal policy itself in the target domain, but the source transitions under optimal policy. To make this point, we demonstrate in our empirical results that the source policy, when used directly, does not produce sensible behaviors of the agent in the target domain when transition models are significantly different.On the other hand, our method uses the source transitions projected onto the target task as reference exploration trajectories, helping to adapt and optimize the source policy to the target domain efficiently.
In canonical reinforcement learning, the goal is to learn the optimal policy π∗ which maximizes the future cumulative reward when the reward function and the transition function of the MDP are unknown. In statistical RL, this leads to the (in)famous explore-exploit tradeoff.Yet, this learnthrough-experience approach has demonstrated significant potential to enable general solutions to control of complex physical systems for which first principles-based models are not easily obtainable. However, one of the key challenges in RL has been the high sample complexity of obtaining a reasonable policy. In other words, RL, and specifically Deep RL, require an absurdly huge amount of interactions with the environment (experience) to find reasonable policies. This high learning inefficiency is in apparent contrast with the efficient learning demonstrated by humans and many biological creatures. For examples, octopuses are known to learn to solve complex problems such as opening jars and squeezing through puzzles with just a few trials. This ability to quickly transfer and adapt learned policies between related tasks is likely a key to general intelligence that seems to be missing from RL. Our goal in the rest of this paper is to show that an algorithm that can judiciously combine adaptation to changes and learning new skills is capable of avoiding brute force random exploration to a large extent and be significantly less sample expensive.

heading: 3.2 ADAPT-TO-LEARN: POLICY TRANSFER

text: We begin by mathematically describing adaptation for policy transfer in RL and state all the necessary assumptions in Section 3.2.1. We then develop the Adapt-to-Learn algorithm in Section 3.2.2.

heading: 3.2.1 ADAPTATION

text:
The proposed Adapt-to-Learn (ATL) algorithm objective is to accumulate higher total returns through maximizing the likelihood of the target policies πθ, which emulate the optimal source transition behavior in the target MDP. We achieve this objective by minimizing the KL divergence between point-wise local trajectories realized using the target policy and source optimal policy in the target
domain (Refer Figure-3 in Appendix-C). This adaptation objective can be formalized as minimizing the KL-divergence between source and target transition trajectories:
η KL (πθ, π ∗) = E
st,at∼τ
( pπθ (τ) log ( pπθ (τ)
qπ∗(τ)
)) , (3)
where τ is the trajectory in the target domain under the policy πθ defined as
τ = (s0, a0, s1, a1, . . . ),
and probability of the trajectory pπθ (τ) under policy πθ and target transition p T (.|st, at) can be written as
pπθ (τ) = ρ(s0) H∏ t=0 πθ(at|st)pT (st+1|st, at). (4)
Similarly qπ∗(τ) can be defined as probability of trajectory deviations at every state st ∈ τ , when the source optimal policy π∗ is used in place of target policy πθ, and the states evolve according to the source transitions model pS(.|st, at) (Refer Figure-3 in Appendix-C):
qπ∗(τ) = ρ(s0) H∏ t=0 π∗(a′t|st)pS(s′t+1|st, a′t). (5)
Assumption-1: We assume that optimal source policy is available, such that the source policy variance can be assumed to be zero. That is, the action probabilities π∗(a′t|st) = 1, ∀st. This is a reasonable assumption, since optimal source policy is available it can be treated as deterministic policy and actions can be chosen greedily. We need this assumption only for deriving the expression for intrinsic reward and theoretical analysis. In the empirical evaluation of the algorithm, we treat π∗(.) as a stochastic policy.
The transition probabilities in the KL divergence in (3) term are treated as transition likelihoods and the transitioned state st+1 as a random variable. Since s′t+1 is the optimal state reached starting in st under source transition model using π∗(a′t|st), we try to emulate this behavior in the target domain and hence evaluate the transition likelihoods of the target trajectory at {s′i}Hi=1. Using the definition of the probabilities of the trajectories under πθ and π∗ Equation-(4) & (5) the log term in the KL divergence of the trajectory (3) is simplified as follows
log
( pπθ (τ)
qπ∗(τ)
) = log ( ρ(s0)π(a0|s0)pT (s′1|s0, a0)π(a1|s1)pT (s′2|s1, a1) . . . ρ(s0)π∗(a′0|s0)pS(s′1|s0, a′0)π∗(a′1|s1)pS(s′2|s1, a′1) . . . ) . (6)
Using (6) and with assumption-1 i.e. π∗(a′t|st) = 1∀st the KL term can simplified as follows
η KL (πθ, π ∗) = E
st,at∼τ
( pπθ (τ) H∑ t=0 log ( πθ(at|st)pT (s′t+1|st, at) pS(s′t+1|st, a′t) ))
η KL (πθ, π ∗) = E
st,at∼τ
( pπθ (τ) H∑ t=0 ζt ) (7)
where ζt = log ( πθ(at|st)pT (s′t+1|st,at)
pS(s′t+1|st,a′t)
) .
If calculating the above expectation is feasible, it is possible to minimize the KL divergence and move the policy parameters in the direction of emulating source transition behavior in the target domain. However, this is not generally the case since the true expectation is intractable. Therefore a common practice is to use an empirical estimate of the expectation to do approximate planning. We minimize the trajectory KL divergence by handling the term ζt as the intrinsic adaptation reward, which captures the local deviation in trajectory in the form of the shaped reward function.
Assumption-2: We do not assume to know the true transition distribution for source and the target, but only have access to simulator model of the source. However, we assume both source and target transition models follow a Gaussian distribution centered at the next propagated state and fixed variance “σ”.
Assumption-2 is not very restrictive since we empirically show that for any deterministic model, a bootstrapped Gaussian transition assumption is sufficient for ATL agent to learn the task. Using the above assumption, we can approximate the KL term as
ζt = log ( πθ(at|st)e−(st+1−s ′ t+1) 2/2σ2 ) . (8)
The individual terms in the expectation ζt represent the distance between two transition likelihoods of landing in the next state s′t+1 starting in st and under actions at, a ′ t. The target agent is encouraged to take actions that lead to states which are close in expectation to a reference state provided by an optimal baseline policy operating on the source model. By doing so, we are providing a possible direction of search for the higher environmental rewards “rt”.
We can, therefore, solve the following optimization problem to generate adaptive policy updates:
π∗Tθ = arg min πθ∈Π (η KL ) . (9)
The distance between two transition models ζt is used as an intrinsic reward to calculate the total trajectory intrinsic return, and further, any policy update algorithm can be used (Sutton et al., 2000; Schulman et al., 2017; 2015) to update the policy in direction optimizing this objective. However, it is to be noted that though we use policy gradient kind of update for adaptive policy, the optimization is more akin to supervised learning. For every state-action pair, the KL-distance is the true metric to be minimized, and unlike RL, we do not engage in optimal value search in the adaptation part of learning. Hence the algorithm is more sample efficient compared to any RL policy search methods.

heading: 3.2.2 ADAPTATION AND LEARNING

text:
We achieve Adaption and Learning simultaneously by augmenting environment reward rt with intrinsic reward ζt. By doing so, we achieve transferred policies which try to both optimize optimal reference tracking and also maximize the cumulative future environmental reward to acquire skills beyond what source can teach. This trade-off between learning by exploration and learning by adaptation can be realized as follows:
η KL = E st,at
( pπθ (τ) H∑ t=0 ((1− β)rt − βζt) ) . (10)
Where the term β is the mixing coefficient. We make a heuristic choice for an appropriate β to start with and annealed over episodes for optimal mixing of adaptation and learning. For consistency of the reward mixing, the rewards rt, ζt are normalized to form the total reward r′t = (1− β)rt + βζt.
Algorithm 1 Adaptive-to-Learn Policy Transfer in RL
Require: π∗(.), pS . Inputs: Source Policy, source simulator 1: Initialize sT0 ∈ ρ0. . Draw initial state from the given distribution in target task 2: for i = 1 ≤ K do 3: while si 6= terminal do 4: a′i ∼ π∗(si) . Generate the optimal action using Source policy 5: ai ∼ πθ(si) . Generate the action using the π̂θ 6: si+1 ∼ pT (si, ai) . Propagate the target task model at state si and action ai 7: s′i+1 ∼ pS(si, a′i) . Propagate the source task model at state si and action a′i 8: ζt = πθ(ai|si)e−(si+1−s ′ i+1)
2/2σ2 . Calculate the KL divergence intrinsic reward term 9: Zi = ({si, ri, ζi, ai, a′i}) . Incrementally store the trajectory for policy update
10: PZn(ηKL) = 1 n ∑n i=1 (∑H t=0∇θ log πθ(ai,t|si,t) ∑∞ t=0 r ′ i,t ) . Empirical loss to optimize
11: π̂θ = argmaxπ∈Π PZn(ηKL) . Minimize the loss to obtain a adaptive policy

heading: 3.3 SAMPLE-BASED ESTIMATION OF THE GRADIENT

text:
The previous section proposed an optimization method to find the adaptive policy using KLdivergence as an intrinsic reward, enforcing the target transition model to mimic the source transitions. This section describes how this objective can be approximated using a Monte Carlo simulation. The adaptive policy update methods work by computing an estimator of the gradient of the
return and plugging it into a stochastic gradient ascent algorithm.
π̂∗Tθ = arg max πθ∈Π PZn(ηKL) (11)
θ ← θ + αĝ. Where α is the learning rate and ĝ is the empirical estimate of the gradient (Refer Appendix-A) of the total return η
KL . The gradient estimate over i.i.d data from the collected trajectories is computed
as follows:
ĝ = PZn(∇θηKL) = Êst,at∼τ ( H∑ t=0 ∇θ log πθ(at|st) ∞∑ t=0 r′t ) ,
PZn(∇θηKL) = 1
n n∑ i=1 ( H∑ t=0 ∇θ log πθ(ai,t|si,t) ∞∑ t=0 r′i,t ) , (12)
where PZn is empirical distribution over the data (Zn : {si, ai, a′i}ni ).

heading: 4 THEORETICAL BOUNDS ON SAMPLE COMPLEXITY

text:
Although there is some empirical evidence that transfer can improve performance in subsequent reinforcement-learning tasks, there are not many theoretical guarantees. Since many of the existing transfer algorithms approach the problem of transfer as a method of providing good initialization to target task RL, we can expect the sample complexity of those algorithms to still be a function of the cardinality of state-action pairs |N | = |S| × |A|. On the other hand, in supervised learning setting, the theoretical guarantees of most algorithm have no dependency on size (or dimensionality) of the input domain (which is analogous to |N | in RL domains). Having formulated a policy transfer algorithm using labeled reference trajectories derived from optimal source policy in ATL, we construct supervised learning like PAC property of the proposed method. For the sample complexity analysis we consider only the adaptation part of learning i.e. β = 1 in Equation-(16). This is because in ATL, the adaptive learning is akin to supervised learning, since the source reference trajectories provide the target states given every (st, at) pair.
Suppose we are given the learning problem specified with training set Zn = (Z1, . . . Zn) where each Zi = ({si, ai, a′i})Hi=0 are independently drawn according to some distribution P . Given the data Zn we can compute the empirical return PZn(ηKL) for every πθ ∈ Π, and we will show that the following holds: ‖PZn(ηKL)− P (ηKL)‖ ≤ (13) with probability at least 1− δ, for some very small δ s.t 0 ≤ δ ≤ 1. We can claim that the empirical return for all πθ is a sufficiently accurate estimate of the true return function. Thus a reasonable learning strategy is to find a πθ ∈ Π that would minimize empirical estimate of the objective (10).
Theorem 4.1 If the induced class of the policy πθ:LΠ has uniform convergence property in empirical mean; then the empirical risk minimization is PAC. s.t
Pn(P (η KL,π̂∗ )− P (ηKL,π∗ ) ≥ ) ≤ δ (14)
and number of trajectory samples required can be lower bounded as
n( , δ) ≥ 2H 2C2
2 log ( 2|Π| δ ) . (15)
For the proof of the above theorem refer the Appendix-B.

heading: 4.1 -OPTIMALITY RESULT UNDER ADAPTIVE TRANSFER-LEARNING

text:
Consider MDP M∗ and M̂ which differ in their transition models. For the sake of analysis, let M∗ be the MDP with ideal transition model, such that target follows source transition p∗ precisely. Let p̂ be the transition model achieved (tracked) using the estimated adapted policy learned over data interacting with the target model and the associated MDP be denoted as M̂ . We analyze the -optimality of return under adapted source optimal policy through proposed policy transfer.
Note for the sake of analysis we are deviating from finite horizon assumption and using discout factor γ. To be consistent with previous section we can assume the horizon length H = logγ
( v
2Vmax
) ,
where v is truncation error in total infinite return.
Definition 4.2 Given the value function V ∗ = V π ∗
and model M1 and M2, which only differ in the corresponding transition models p1 and p2. Lets define ∀s, a ∈ S ×A
dV ∗
M1,M2 = sup s,a∈S×A ∣∣∣∣ E s′∼P1(s,a) [V ∗(s′)]− E s′∼P2(s,a) [V ∗(s′)] ∣∣∣∣ Lemma 4.3 Given M∗, M̂ and value function V π ∗
M∗ , V π∗ M̂ the following bound holds∥∥∥V π∗M∗ − V π∗M̂ ∥∥∥∞ ≤ γ (1−γ)2
where maxs,a ‖p̂(.|s, a)− p∗(.|s, a)‖ ≤ and p̂ and p∗ are transition of MDP M̂,M∗ respectively. The proof of this lemma is based on the simulation lemma (see Appendix-C). Similar results for RL with imperfect models were reported by (Jiang, 2018).

heading: 5 POLICY TRANSFER IN SIMULATED ROBOTIC LOCOMOTION TASKS

text:
To evaluate Adapt-to-Learn Policy Transfer in reinforcement learning, we design our experiments using sets of tasks based on the continuous control environments in MuJoCo simulator (Todorov et al., 2012). Our experimental results demonstrate that ATL can adapt to significant changes in transition dynamics. Therefore, we perturb the parameters of the simulated target models for the policy transfer experiments (see Table-1 for original and perturbed parameters of the target mode). To create a challenging training environment, we changed the parameters of the model such that the optimal source policy alone without any learning cannot produce any stable results (see source policy performance in Figure-1). We compare our results against two baselines: (a) Initialized Reinforcement learning (initialized PPO) (Jumpstart-RL (Wang & Mahadevan, 2009)) (b) standalone reinforcement policy learning (PPO) (Schulman et al., 2017).
We experiment with the ATL algorithm on Hopper, Walker2d, and HalfCheetah Environments. The states of the robots are their generalized positions and velocities, and the actions are joint torques. High dimensionality, non-smooth dynamics due to contacts and being under-actuated systems make these tasks very challenging. We use deep neural networks to represent the source and target policy, the details of which are in the Table-2 Appendix-D. The following models are included in our evaluation:
Slippery Hopper: is defined through 11-dimensional state space and 3-dimension action space, with reward function defined as r(t) = (st+1 − st)/dt − 10−3‖a‖2, and a bonus of +1 for being in a non-terminal state. The simulation is terminated upon reaching 1000 steps or hopper toppling. The target model differs in the floor friction and foot joint damping.
Slippery Fat-Walker2d: is defined through 17-dimensional state space and 6-dimension action space, with reward function and termination condition defined same as Hopper. The target model differs in the model density and floor friction.
Fat HalfCheetah: is defined through 17-dimensional state space and 6-dimension action space, with reward function defined as r(t) = (st+1 − st)/dt − 0.1‖a‖2. The simulation is terminated upon reaching 1000 steps. The target model differs in the floor friction coefficient, gravity, and mass.
To establish a standard baseline, we also included the classic cart-pole and Inverted pendulum balancing tasks, based on the formulation (Barto et al., 1983). We also demonstrate the cross-domain transfer capabilities using a model-based variant of the proposed algorithm. The results of policy transfer for Cart-Pole to Inverted Pendulum and Inverted Pendulum to Bicycle transfers are provided in the Appendix-D. Learning curves showing the total reward averaged across three runs of each algorithm are provided in Figure-1. Adapt-to Learn policy transfer solved all the three tasks, yielding quicker learning compared to other baseline methods. These results provide empirical evidence of our hypothesis. Using trajectory KL divergence as intrinsic adaptation reward to adapt source policy to the target, we achieve a more robust and sample efficient policy transfer between two tasks,
compared to using a warm-start or standalone RL method. Note that the target domain perturbations introduced are significant enough such that source policy alone without any adaptation in the target domain produced no meaningful results (Figure-1). This notion of adaptation in the face of uncertainty is a key advancement over traditional policy transfer, meta-learning, or adversarial RL methods aiming to improve performance by learning a policy over a set of lightly perturbed tasks.

heading: 6 CONCLUSION

text:
We introduced a new transfer learning technique for RL: Adapt-to-Learn, that utilizes adaptation of the source policy to target tasks. We demonstrated on nonlinear and continuous robotic locomotion tasks that learning to adapt source policy to the target domain leads to a significant reduction in sample complexity over the prevalent jump-start based approaches. We further also proved theoretical guarantees on the reduced sample complexity of our proposed architecture. There are many exciting directions for future work. A network of policies that can generalize across multiple tasks could be learned based on each new adapted policies. How to train this end-to-end is an important question for meta-learning. The ability of Adapt-to-Learn to handle significant perturbations to the
transition model indicates that it should naturally extend to sim-to-real transfer. Indeed we argue that such adaptation is necessary for real-world robotics, as has been established previously in classical domains like flight control. Another exciting direction is to extend the work to other combinatorial domains (e.g., multiplayer games). We expect, therefore follow on work will find other exciting ways of exploiting such adaptation in RL and machine learning.

heading: A TOTAL RETURN GRADIENT WITH RESPECT TO POLICY PARAMETERS

text:
The total return which we aim to maximize in adapting the source policy to target is the mixture of environmental rewards and Intrinsic KL divergence reward as follows,
η KL (πθ, π ∗) = E
st,at∼τ
( pπθ (τ) H∑ t=0 r′t ) (16)
he adaptive policy update methods work by computing an estimator of the gradient of the return and plugging it into a stochastic gradient ascent algorithm.
π̂∗Tθ = arg max πθ∈Π PZn(ηKL) (17)
θ = θ + αĝ
where α is the learning rate and ĝ is the empirical estimate of the gradient of the total discounted return η
KL . The derivation of the gradient term is exactly same as for policy gradient methods.
Taking the derivative of the total return term
∇θ(ηKL) = E st,at∼τ
( ∇θpπθ (τ) H∑ t=0 r′t ) .
Consider the term ∇θpπθ (τ) and evaluate the derivative. Lets consider the term log pπθ (τ). Using the definition of pπθ (τ) and we can write the log term as
log pπθ (τ) = log(ρ0πθ(a0|s0)pT (s1|s0, a0) . . .)
log pπθ (τ) = log(ρ0) + H∑ t=0 log πθ(at|st) + H∑ t=0 pT (st+1|st, at).
Taking the derivative wrto θ we can write,
∇θ log pπθ (τ) = H∑ t=0 ∇θ log πθ(at|st)
∇θpπθ (τ) pπθ (τ) = H∑ t=0 ∇θ log πθ(at|st).
Therefore we can write
∇θpπθ (τ) = pπθ (τ) H∑ t=0 ∇θ log πθ(at|st)
Substituting the above expression we can write the total gradient as,
∇θ(ηKL) = Epπθ ( H∑ t=0 ∇θ log πθ(at|st) H∑ t=0 r′t ) .

heading: B PROOF OF THEOREM-4.1

text:
Theorem B.1 If the induced class LΠ has uniform convergence in empirical mean property then empirical risk minimization is PAC.
For notation simplicity we drop the superscript T (for Target domain) and subscript θ (policy parameters) in further analysis. Unless stated we will using following simplifications π̂∗ = π̂∗Tθ and π∗ = π∗Tθ .
Proof Fix , δ > 0 we will show that for sufficiently large n ≥ n( , δ) Pn(P (η
KL,π̂∗ )− P (ηKL,π∗ ) ≥ ) ≤ δ (18)
Let π∗ ∈ Π be the minimizer of true return P (η KL
), further adding and subtracting the terms PZn(ηKL,π̂∗ ) and PZn(ηKL,π∗ ) we can write
P (η KL,π̂∗ )− P (ηKL,π∗ ) =
P (η KL,π̂∗ )− PZn(ηKL,π̂∗ ) + PZn(ηKL,π̂∗ )− PZn(ηKL,π∗ ) + PZn(ηKL,π∗ )− P (ηKL,π∗ )
(19)
We can divide the above expression in three terms
1. P (η KL,π̂∗ )− PZn(ηKL,π̂∗ )
2. PZn(ηKL,π̂∗ )− PZn(ηKL,π∗ )
3. PZn(ηKL,π∗ )− P (ηKL,π∗ )
Lets consider the term PZn(ηKL,π̂∗ )−PZn(ηKL,π∗ ) in the above expression is always negative semidefinite, since π̂∗ is a maximizer wrto PZn(ηKL), hence PZn(ηKL,π̂∗ ) ≤ PZn(ηKL,π∗ ) always, i.e
PZn(ηKL,π̂∗ )− PZn(ηKL,π∗ ) ≤ 0 Next the 1st term can be bounded as
P (η KL,π̂∗ )− PZn(ηKL,π̂∗ ) ≤ sup π∈Π [PZn(ηKL)− P (ηKL)] ≤ sup π∈Π ‖PZn(ηKL)− P (ηKL)‖
Similarly upper bound can be written for the 3rd term Therefore we can upper bound the above expression as
P (η KL,π̂∗ )− P (ηKL,π∗ ) ≤ 2 sup π∈Π ‖PZn(ηKL)− P (ηKL)‖ (20)
From Equation-(18) we have
sup π∈Π ‖PZn(ηKL)− P (ηKL)‖ ≥ /2 (21)
Using McDiarmids inequality and union bound, we can state the probability of this event as
Pn(‖PZn(ηKL)− P (ηKL)‖ ≥ /2) ≤ 2|Π|e − n 2 2C2H2 (22)
whereC is bound on state space S andH is horizon length. The finite difference boundC is obtained assuming transition pT , pS follow a Gaussian distributions and the i.i.d trajectories collected are of maximumH length. Equating the RHS of the expression to δ and solving for n we get
n( , δ) ≥ 2H 2C2
2 log ( 2|Π| δ ) (23)
for n ≥ n( , δ) the probability of receiving a bad sample is less than δ. The total number of observed transition is of order O(nH). Refer supplementary document for details of the proof.

heading: C PROOF OF LEMMA-4.3

text:
Lemma C.1 Given M∗, M̂ and value function V π ∗ M∗ , V π∗ M̂ the following bound holds∥∥∥V π∗M∗ − V π∗M̂ ∥∥∥∞ ≤ γ (1−γ)2 .
Here maxs,a ‖p̂(.|s, a)− p∗(.|s, a)‖ ≤ and p̂ and p∗ are transition of MDP M̂,M∗ respectively. Note we are deviating from finite horizon assumption and using γ. To be consistent with previous section we can assume the horizon length H = logγ ( v
2Vmax
) . Where v is truncation error in total
infinite return.
Proof For any s ∈ S
|V π ∗
M̂ (s)− V π
∗
M∗(s)|∞ = |r(s, a) + γ 〈 p̂(s′|s, a), V π ∗ M̂ (s′) 〉 − r(s, a)− γ 〈 p∗(s′|s, a), V π ∗ M∗(s ′) 〉 |∞
Add and subtract the term γ 〈 p∗(s′|s, a), V π∗ M̂ (s′) 〉
= |γ 〈 p̂(s′|s, a), V π ∗ M̂ (s′) 〉 − γ 〈 p∗(s′|s, a), V π ∗ M̂ (s′) 〉 + γ 〈 p∗(s′|s, a), V π ∗ M̂ (s′) 〉 − γ 〈 p∗(s′|s, a), V π ∗ M∗(s ′) 〉 |∞
≤ γ| 〈 p̂(s′|s, a), V π ∗ M̂ (s′) 〉 − 〈 p∗(s′|s, a), V π ∗ M̂ (s′) 〉 |+ γ| 〈 p∗(s′|s, a), V π ∗ M̂ (s′) 〉 − γ 〈 p∗(s′|s, a), V π ∗ M∗(s ′) 〉 |∞ ≤ γ|p̂(s′|s, a)− p∗(s′|s, a)|∞|V π ∗ M̂ (s′)|∞ + γ|V π ∗ M̂ (s)− V π ∗ M∗(s)|∞
Using the definition of in above expression, we can write
|V π ∗
M̂ (s)− V π
∗ M∗(s)|∞ ≤ γ |V π ∗ M̂ (s′)|∞ + γ|V π ∗ M̂ (s)− V π ∗ M∗(s)|∞
Therefore
|V π ∗
M̂ (s)− V π
∗ M∗(s)|∞ ≤ γ |V π∗ M̂ (s′)|∞
1− γ
Now we solve for expression |V π∗ M̂ (s′)|∞. We know that this term is bounded as
|V π ∗ M̂ (s′)|∞ ≤ Rmax 1− γ
where Rmax = 1, therefore we can write the complete expression as
|V π ∗
M̂ (s)− V π
∗ M∗(s)|∞ ≤ γ
(1− γ)2

heading: D ADDITIONAL RESULTS AND DETAILS OF EXPERIMENTAL DOMAIN

text:
To establish a standard baseline, we also include the results for classic cart-pole and Inverted pendulum balancing tasks, based on the formulation Barto et al. (1983). We also demonstrate the crossdomain transfer capabilities using a variant of the proposed algorithm Joshi & Chowdhary (2018) and the results for Cart-Pole to Inverted Pendulum and Inverted Pendulum to Bicycle transfers are provided in the following sections.
D.1 INVERTED PENDULUM (IP) TO TIME-VARYING IP
We demonstrate our approach for a continuous state domain, Inverted Pendulum (IP) swing-up and balance Figure-4. The source task is the conventional IP domain. The target task differs from the source task in the transition model. The target task is a non-stationary inverted pendulum, where the length and mass of the pendulum are continuously time varying with function Li = L0 +0.5cos(πi50 ) and Mi = M0 + 0.5cos(πi50 ), where L0 = 1, M0 = 1 and i = 1 . . . N . The state variables describing the system are angle and angular velocity {θ, θ̇} ∈ [−π, π]. The RL objective is to swing-up and balance the pendulum upright such that θ = 0, θ̇ = 0. The reward function is selected as R(θ, θ̇) = −10|θ|2− 5|θ̇|2, which yields maximum value at upright position and minimum at the down-most position. The continuous action space is bounded by T ∈ [−1, 1]. Note that the domain is tricky, since full throttle action is assumed to not generate enough torque to be able to swing the pendulum to the upright position, hence, the agent must learn to swing the pendulum back and forth and leverages angular momentum to go to the upright position.
D.2 ROBUSTNESS TO NEGATIVE TRANSFER
We demonstrate that the proposed transfer is robust to negative transfers, that is, cases in transfer learning where naive initialization of the target using source policy could be detrimental. We demonstrate this through an inverted pendulum upright balance task with the sign of control flipped. That is, we use an inverted pendulum model as both source and target systems, but the target is the inverse of the source model with the sign of the control action flipped. It should be noted that dealing with a change of sign in control has been considered a highly challenging problem in adaptive control Chowdhary et al. (2013). We demonstrate that ATL is immune to negative transfers. Figure
5a and 5b demonstrate convergence to average maximum reward with lesser training samples for proposed ATL method compared to Initialized-RL(UMA-TL) and standalone-RL methods. It is to be observed that UMA-TL method suffers from a negative transfer phenomenon. The agent under UMA-TL converges to a much lower average reward by getting stuck in local minima and never achieves the upright balance of the pendulum.
Also, the samples observed by UMA-TL in learning the task is much higher compared to no transfer (RL) and proposed ATL methods. If the source and target are not sufficiently related or the features of source task do not correspond to the target, the transfer may not improve or even decrease the performance in target task leading to negative transfer. We show that the UMA-TL suffers from a negative transfer in these results, whereas the performance of presented ATL is much superior compared to UMA-TL and RL(learning from scratch).

heading: E CROSS DOMAIN TRANSFER

text:
E.1 CART-POLE TO BICYCLE:
Bicycle balancing is a challenging physical problem, especially when the bicycle velocity is below the critical velocity Vc = 4m/s to 5m/s. We set the bicycle velocity to be V < Vc i.e., V = 2.778m/s such that the bicycle becomes unstable, and active control is required to maintain stability. The simulation itself is very high fidelity and realistic, which was designed for studying the physics of the bicycle.
The states of the bicycle task are angle and angular velocity of the handlebar and the bike from vertical (θ, θ̇, ω, ω̇) respectively. For the given state the agent is in, it chooses a continuous action of applying torque to the handlebar, T ∈ [−2Nm, 2Nm] trying to keep the bike upright. The details of bicycle dynamics are beyond the scope of this paper; interested readers are referred to Randlov & Alstrom (1998); Åström et al. (2005), and references therein.
We use the Cart-Pole as source task for learning to balance a bicycle. The bicycle balance problem in principle is similar to that of cart-pole, since the objective is to keep the unstable system upright. The objective of balance is achieved in both the systems by moving in the direction of fall, which is termed as a non-minimum phase behavior in the controls system literature. The control in the cart pole affects the angle of the pole, by moving the cart such that it is always under the pole. In the bicycle, the control is to move the handlebar in the direction of fall. However, balancing the bike is not straightforward, to turn the bike under itself, one must first steer in the other direction before turning in the direction of fall; this is called counter-steering Åström et al. (2005). We observe that both cart pole and bicycle have this commonality in their dynamical behaviors, as both the system have a non-minimum phase that is the presence of unstable zero. This similarity qualifies the cartpole system as an appropriate choice of source model for bicycle balance task.
Cart pole is characterized by state vector [x, ẋ, θ, θ̇], i.e., position, the velocity of cart and angle, angular velocity of the pendulum. The action space is the force applied to the cart F ∈ [−1N, 1N ]. For the mapping between the state space of bicycle and cart pole model, we use Unsupervised Manifold Alignment(UMA) to obtain this mapping Wang & Mahadevan (2009). We do not report
the training time to learn the intertask mapping since it is common to both ATL and UMA-TL methods. Figure 7b and 8 shows the quality of transfer for ATL through faster convergence to average maximum reward with lesser training samples compared to UMA-TL and RL methods.
E.2 MOUNTAIN CAR (MC) TO INVERTED PENDULUM (IP)
We also demonstrate the cross-domain transfer between mountain car to an inverted pendulum. The source and target task are characterized by different state and action space. The source task MC is a benchmark RL problem of driving an underpowered car up a hill. The dynamics of MC are described by two continuous state variables (x, ẋ) where x ∈ [−1.2, 0.6] and ẋ ∈ [−0.07, 0.07] and one continuous action F ∈ [−1, 1]. The reward function is proportional to the negative of the squared distance of the car from goal position. The target task is conventional IP with state (θ, θ̇) ∈ (−π, π) and action T ∈ [−1, 1]. We present the performance of transfer methods based on sample efficiency in learning the target task and speed of convergence to the maximum average reward. Similar to the bicycle domain transfer Figure 9a and 9b shows the quality of transfer for ATL through faster convergence to average maximum reward with lesser training samples compared to UMA-TL and RL methods.


summaries: This paper proposed scale-equivariant steerable convolutional neural networks that is able to preserve both the translation and scaling symmetry of the data in the representation. To achieve this, the authors developed the scale-convolution blocks in the network, and generalized other common blocks, such as pooling and nonlinearity, to remain scale-equivariant. Extensive experiments have been conducted to show that the proposed scale-equivariant network

summaries: The paper describes a method for integrating scale equivariance into convolutional networks using steerable filters.  After developing the theory using continuous scale and translation space, a discretized implementation using a fixed set of steerable basis elements is described.  Experiments are performed measuring the error from true equivariance, varying number of layers, image scale and scales in scale interactions.  The method is evaluated using MNIST-scale and STL-10, with convincing results on MNIST-scale and bit less convincing but still good results on STL-10.

summaries: This paper proposes a framework (SESN) for learning deep networks that possess scale equivariance in addition to translation invariance. The formulation is based on group convolution on the scale-translation group. Filters are represented as the coefficients of a set of continuous basis functions, which are sampled (once) at a discrete set of scales. The theoretical formulatioin is clear and interesting. The approach is evaluated in terms of image classification accuracy. The set of baselines is quite exhaustive, including recent papers and papers that are not widely-known.

abstractText: The effectiveness of Convolutional Neural Networks (CNNs) has been substantially attributed to their built-in property of translation equivariance. However, CNNs do not have embedded mechanisms to handle other types of transformations. In this work, we pay attention to scale changes, which regularly appear in various tasks due to the changing distances between the objects and the camera. First, we introduce the general theory for building scale-equivariant convolutional networks with steerable filters. We develop scale-convolution and generalize other common blocks to be scale-equivariant. We demonstrate the computational efficiency and numerical stability of the proposed method. We compare the proposed models to the previously developed methods for scale equivariance and local scale invariance. We demonstrate state-of-the-art results on the MNIST-scale dataset and on the STL-10 dataset in the supervised learning setting.

authors:
Etienne Barnard
David Casasent

title: Invariance and neural nets

venue: IEEE Transactions on neural networks,

authors:
Adam Coates
Andrew Ng
Honglak Lee

title: An analysis of single-layer networks in unsupervised feature learning

authors:
Taco Cohen
Max Welling

title: Group equivariant convolutional networks

venue: In International conference on machine learning,

authors:
Taco Cohen
Mario Geiger
Jonas Köhler
Max Welling

title: Convolutional networks for spherical signals

venue: arXiv preprint arXiv:1709.04893,

authors:
Taco S Cohen
Max Welling

title: Steerable cnns

venue: arXiv preprint arXiv:1612.08498,

authors:
Taco S Cohen
Maurice Weiler
Berkay Kicanaoglu
Max Welling

title: Gauge equivariant convolutional networks and the icosahedral cnn

authors:
Terrance DeVries
Graham W Taylor

title: Improved regularization of convolutional neural networks with cutout

venue: arXiv preprint arXiv:1708.04552,

authors:
Carlos Esteves
Christine Allen-Blanchette
Ameesh Makadia
Kostas Daniilidis

title: Learning so (3) equivariant representations with spherical cnns

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

title: A course in abstract harmonic analysis

authors:
William T. Freeman
Edward H Adelson

title: The design and use of steerable filters

venue: IEEE Transactions on Pattern Analysis & Machine Intelligence,

authors:
Rohan Ghosh
Anupam K Gupta

title: Scale steerable filters for locally scale-invariant convolutional neural networks

venue: arXiv preprint arXiv:1906.03861,

authors:
Joao F Henriques
Andrea Vedaldi

title: Warped convolutions: Efficient invariance to spatial transformations

venue: In Proceedings of the 34th International Conference on Machine Learning-Volume

title: Basic theory of pattern observation

venue: Technical Group on Automata and Automatic Control,

authors:
Jörn-Henrik Jacobsen
Bert De Brabandere
Arnold WM Smeulders

title: Dynamic steerable blocks in deep residual networks

venue: arXiv preprint arXiv:1706.00598,

authors:
Angjoo Kanazawa
Abhishek Sharma
David Jacobs

title: Locally scale-invariant convolutional neural networks

venue: arXiv preprint arXiv:1412.5104,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

title: The structure of images

venue: Biological cybernetics,

title: N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials

venue: arXiv preprint arXiv:1803.01588,

authors:
Risi Kondor
Shubhendu Trivedi

title: On the generalization of equivariance and convolution in neural networks to the action of compact groups

venue: arXiv preprint arXiv:1802.03690,

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

title: Scale-space theory in computer vision, volume 256

venue: Springer Science & Business Media,

authors:
Diego Marcos
Benjamin Kellenberger
Sylvain Lobry
Devis Tuia

title: Scale equivariance in cnns with vector fields

venue: arXiv preprint arXiv:1807.11783,

authors:
Pietro Perona
Jitendra Malik

title: Scale-space and edge detection using anisotropic diffusion

venue: IEEE Transactions on pattern analysis and machine intelligence,

authors:
Kai Sheng Tai
Peter Bailis
Gregory Valiant

title: Equivariant transformer networks

venue: arXiv preprint arXiv:1901.11399,

authors:
Nathaniel Thomas
Tess Smidt
Steven Kearnes
Lusann Yang
Li Li
Kai Kohlhoff
Patrick Riley

title: Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds

venue: arXiv preprint arXiv:1802.08219,

authors:
Matej Ulicny
Vladimir A Krylov
Rozenn Dahyot

title: Harmonic networks with limited training samples

venue: arXiv preprint arXiv:1905.00135,

authors:
Maurice Weiler
Gabriele Cesa

title: General e (2)-equivariant steerable cnns

venue: In Advances in Neural Information Processing Systems,

authors:
Maurice Weiler
Mario Geiger
Max Welling
Wouter Boomsma
Taco Cohen

title: 3d steerable cnns: Learning rotationally equivariant features in volumetric data

authors:
Maurice Weiler
Fred A Hamprecht
Martin Storath

title: Learning steerable filters for rotation equivariant cnns

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

title: Scale-space filtering

venue: In Readings in Computer Vision, pp. 329–332

authors:
Daniel Worrall
Gabriel Brostow

title: Cubenet: Equivariance to 3d rotation and translation

authors:
Daniel E Worrall
Max Welling

title: Deep scale-spaces: Equivariance over scale

venue: arXiv preprint arXiv:1905.11697,

authors:
Daniel E Worrall
Stephan J Garbin
Daniyar Turmukhambetov
Gabriel J Brostow

title: Harmonic networks: Deep translation and rotation equivariance

authors:
Yichong Xu
Tianjun Xiao
Jiaxing Zhang
Kuiyuan Yang
Zheng Zhang

title: Scale-invariant convolutional neural networks

venue: arXiv preprint arXiv:1411.6369,

authors:
Sergey Zagoruyko
Nikos Komodakis

title: Wide residual networks

venue: arXiv preprint arXiv:1605.07146,

heading: 1 INTRODUCTION

text:
Scale transformations occur in many image and video analysis tasks. They are a natural consequence of the variable distances among objects, or between objects and the camera. Such transformations result in significant changes in the input space which are often difficult for models to handle appropriately without careful consideration. At a high level, there are two modeling paradigms which allow a model to deal with scale changes: models can be endowed with an internal notion of scale and transform their predictions accordingly, or instead, models can be designed to be specifically invariant to scale changes. In image classification, when scale changes are commonly a factor of 2, it is often sufficient to make class prediction independent of scale. However, in tasks such as image segmentation, visual tracking, or object detection, scale changes can reach factors of 10 or more. In these cases, it is intuitive that the ideal prediction should scale proportionally to the input. For example, the segmentation map of a nearby pedestrian should be easily converted to that of a distant person simply by downscaling.
Convolutional Neural Networks (CNNs) demonstrate state-of-the-art performance in a wide range of tasks. Yet, despite their built-in translation equivariance, they do not have a particular mechanism for dealing with scale changes. One way to make CNNs account for scale is to train them with data augmentation Barnard & Casasent (1991). This is, however, suitable only for global transformations. As an alternative, Henriques & Vedaldi (2017) and Tai et al. (2019) use the canonical coordinates of scale transformations to reduce scaling to well-studied translations. While these approaches do allow for scale equivariance, they consequently break translation equivariance.
Several attempts have thus been made to extend CNNs to both scale and translation symmetry simultaneously. Some works use input or filter resizing to account for scaling in deep layers Xu et al. (2014); Kanazawa et al. (2014). Such methods are suboptimal due to the time complexity of tensor resizing and the need for interpolation. In Ghosh & Gupta (2019) the authors pre-calculate filters defined on several scales to build scale-invariant networks, while ignoring the important case of scale equivariance. In contrast, Worrall & Welling (2019) employ the theory of semigroup equivariant networks with scale-space as an example; however, this method is only suitable for integer downscale factors and therefore limited. ∗Correspondence to i.sosnovik@uva.nl †Work was done when the author had an internship at the UvA-Bosch Delta Lab Source code is available at http://github.com/isosnovik/sesn
In this paper we develop a theory of scale-equivariant networks. We demonstrate the concept of steerable filter parametrization which allows for scaling without the need for tensor resizing. Then we derive scale-equivariant convolution and demonstrate a fast algorithm for its implementation. Furthermore, we experiment to determine to what degree the mathematical properties actually hold true. Finally, we conduct a set of experiments comparing our model with other methods for scale equivariance and local scale invariance.
The proposed model has the following advantages compared to other scale-equivariant models:
1. It is equivariant to scale transformations with arbitrary discrete scale factors and is not limited to either integer scales or scales tailored by the image pixel grid.
2. It does not rely on any image resampling techniques during training, and therefore, produces deep scale-equivariant representations free of any interpolation artifacts.
3. The algorithm is based on the combination of tensor expansion and 2-dimensional convolution, and demonstrates the same computation time as the general CNN with a comparable filter bank.

heading: 2 PRELIMINARIES

text:
Before we move into scale-equivariant mappings, we discuss some aspects of equivariance, scaling transformations, symmetry groups, and the functions defined on them. For simplicity, in this section, we consider only 1-dimensional functions. The generalization to higher-dimensional cases is straightforward.
Equivariance Let us consider some mapping g. It is equivariant under Lθ if and only if there exists L′θ such that g ◦ Lθ = L′θ ◦ g. In case L′θ is the identity mapping, the function g is invariant. In this paper we consider scaling transformations. In order to guarantee the equivariance of the predictions to such transformations, and to improve the performance of the model, we seek to incorporate this property directly inside CNNs.
Scaling Given a function f : R→ R, a scale transformation is defined as follows:
Ls[f ](x) = f(s −1x), ∀s > 0 (1)
We refer to cases with s > 1 as upscale and to cases with s < 1 as downscale. If we convolve the downscaled function with an arbitrary filter ψ and perform a simple change of variables inside the integral, we get the following property:
[Ls[f ] ? ψ](x) = ∫ R Ls[f ](x ′)ψ(x′ − x)dx′ = ∫ R f(s−1x′)ψ(x′ − x)dx′
= s ∫ R f(s−1x′)ψ(s(s−1x′ − s−1x))d(s−1x′) = sLs[f ? Ls−1 [ψ]](x) (2)
In other words, convolution of the downscaled function with a filter can be expressed through a convolution of the function with the correspondingly upscaled filter where downscaling is performed afterwards. Equation 2 shows us that the standard convolution is not scale-equivariant.
Steerable Filters In order to make computations simpler, we reparametrize ψσ(x) = σ−1ψ(σ−1x), which has the following property:
Ls−1 [ψσ](x) = ψσ(sx) = s −1ψs−1σ(x) (3)
It gives a shorter version of Equation 2:
Ls[f ] ? ψσ = Ls[f ? ψs−1σ] (4)
We will refer to such a parameterization of filters as Steerable Filters because the scaling of these filters is the transformation of its parameters. Note that we may construct steerable filters from any
function. This has the important consequence that it does not restrict our approach. Rather it will make the analysis easier for discrete data. Moreover, note that any linear combination of steerable filters is still steerable.
Scale-Translation Group All possible scales form the scaling group S. Here we consider the discrete scale group, i.e. scales of the form . . . a−1, a−1, 1, a, a2, . . . with base a as a parameter of our method. Analysis of this group by itself breaks the translation equivariance of CNNs. Thus we seek to incorporate scale and translation symmetries into CNNs, and, therefore consider the ScaleTranslation Group H . It is a semidirect product of the scaling group S and the group of translations T ∼= R. In other words: H = {(s, t)|s ∈ S, t ∈ T}. For multiplication of group elements, we have (s2, t2) ·(s1, t1) = (s2s1, s2t1 + t2) and for the inverse (s2, t2)−1 ·(s1, t1) = (s−12 s1, s −1 2 (t1− t2)). Additionally, for the corresponding scaling and translation transformations, we have Lst = LsLt 6= LtLs, which means that the order of the operations matters.
From now on, we will work with functions defined on groups, i.e. mappings H → R. Note, that simple function f : R → R may be considered as a function on H with constant value along the S axis. Therefore, Equation 4 holds true for functions on H as well. One thing we should keep in mind is that when we apply Ls to functions on H and R we use different notations. For example Ls[f ](x ′) = f(s−1x′) and Ls[f ](s′, t′) = f((s, 0)−1(s′, t′)) = f(s−1s′, s−1t′)
Group-Equivariant Convolution Given group G and two functions f and ψ defined on it, Gequivariant convolution is given by
[f ?G ψ](g) = ∫ G f(g′)Lg[ψ](g ′)dµ(g′) = ∫ G f(g′)ψ(g−1g′)dµ(g′) (5)
Here µ(g′) is the Haar measure also known as invariant measure Folland (2016). For T ∼= R we have dµ(g′) = dg′. For discrete groups, the Haar measure is the counting measure, and integration becomes a discrete sum. This formula tells us that the output of the convolution evaluated at point g is the inner product between the function f and the transformed filter Lg[ψ].

heading: 3 SCALE-EQUIVARIANT MAPPINGS

text:
Now we define the main building blocks of scale-equivariant models.
Scale Convolution In order to derive scale convolution, we start from group equivariant convolution with G = H . We first use the property of semidirect product of groups which splits the integral, then choose the appropriate Haar measures, and finally use the properties of steerable filters. Given the function f(s, t) and a steerable filter ψσ(s, t) defined on H , a scale convolution is given by:
[f ?H ψσ](s, t) = ∫ S ∫ T f(s′, t′)Lst[ψσ](s ′, t′)dµ(s′)dµ(t′)
= ∑ s′ ∫ T f(s′, t′)ψsσ(s −1s′, t′ − t)dt′ = ∑ s′ [f(s′, ·) ? ψsσ(s−1s′, ·)](t) (6)
And for the case of Cin input and Cout output channels we have:
[f ?H ψσ]m(s, t) = Cin∑ n=1 ∑ s′ [fn(s ′, ·) ? ψn,m,sσ(s−1s′, ·)](t), m = 1 . . . Cout (7)
The proof of the equivariance of this convolution to transformations fromH is given in Appendix A.
Kondor & Trivedi (2018) prove that a feed-forward neural network is equivariant to transformations from G if and only if it is constructed from G-equivariant convolutional layers. Thus Equation 7 shows the most general form of scale-equivariant layers which allows for building scale-equivariant convolutional networks with such choice of S. We will refer to models using scale-equivariant layers with steerable filters as Scale-Equivariant Steerable Networks, or shortly SESN1
Nonlinearities In order to guarantee the equivariance of the network to scale transformations, we use scale equivariant nonlinearities. We are free to use simple point-wise nonlinearities. Indeed,
1pronounced ‘season’
point-wise nonlinearities ν, like ReLU, commute with scaling transformations:
[ν ◦ Ls[f ]](s′, x′) = ν(Ls[f ](s′, x′)) = ν(f(s−1s′, s−1x′)) = ν[f ](s−1s′, s−1x′) = [Ls ◦ ν[f ]](s′, x′)
(8)
Pooling Until now we did not discuss how to convert an equivariant mapping to invariant one. One way to do this is to calculate the invariant measure of the signal. In case of translation, such a measure could be the maximum value for example.
First, we propose the maximum scale projection defined as f(s, x) → maxs f(s, x). This transformation projects the function f from H to T . Therefore, the representation stays equivariant to scaling, but loses all information about the scale itself.
Second, we are free to use spatial max-pooling with a moving window or global max pooling. Transformation f(s, x)→ maxx f(s, x) projects the function f fromH to S. The obtained representation is invariant to scaling in spatial domain, however, it stores the information about scale.
Finally, we can combine both of these pooling mechanisms in any order. The obtained transformation produces a scale invariant function. It is useful to utilize this transformation closer to the end of the network, when the deep representation must be invariant to nuisance input variations, but already has very rich semantic meaning.

heading: 4 IMPLEMENTATION

text:
In this paragraph we discuss an efficient implementation of Scale-Equivariant Steerable Networks. We illustrate all algorithms in Figure 1. For simplicity we assume that zero padding is applied when it is needed for both the spatial axes and the scale axis.
Filter Basis A direct implementation of Equation 7 is impossible due to several limitations. First, the infinite number of scales in S calls for a discrete approximation. We truncate the scale group and limit ourselves to NS scales and use discrete translations instead of continuous ones. Training of SESN involves searching for the optimal filter in functional space which is a problem by itself. Rather than solving it directly, we choose a complete basis of Nb steerable functions Ψ = {ψs−1σ,i}Nbi=1 and represent convolutional filter as a linear combination of basis functions with trainable parameters w = {wi}Nbi=1. In other words, we do the following substitution in Equation 7: ψσ → κ = ∑ i wiΨi
In our experiments we use a basis of 2D Hermite polynomials with 2D Gaussian envelope, as it demonstrates good results. The basis is pre-calculated for all scales and fixed. For filters of size V × V , the basis is stored as an array of shape [Nb, S, V, V ]. See Appendix C for more details. Conv T →H If the input signal is just a function on T with spatial size U × U , stored as an array of shape [Cin, U, U ], then Equation 7 can be simplified. The summation over S degenerates, and the final result can be written in the following form:
convTH(f, w,Ψ) = squeeze(conv2d(f, expand(w ×Ψ))) (9)
Here w is an array of shape [Cout, Cin, Nb]. We compute filter w×Ψ of shape [Cout, Cin, S, V, V ] and expand it to shape [Cout, CinS, V, V ]. Then we use standard 2D convolution to produce the output with CoutS channels and squeeze it to shape [Cout, S, U, U ]. Note that the output can be viewed as a stack of feature maps, where all the features in each spatial position are vectors of S components instead of being scalars as in standard CNNs.
Conv H →H The function on H has a scale axis and therefore there are two options for choosing weights of the convolutional filter. The filter may have just one scale and, therefore, does not capture the correlations between different scales of the input function; or, it may have a non-unitary extent KS in the scale axis and capture the correlation between KS neighboring scales. We refer to the second case as interscale interaction.
It the first case w has shape [Cout, Cin, Nb] and Equation 7 degenerates in the same way as before
convHH(f, w,Ψ) = squeeze(conv2d(expand(f), expand(w ×Ψ))) (10)
We expand f to an array of shape [CinS,U, U ] and expand w×Ψ to have shape [CoutS,CinS, V, V ]. The result of the convolution is then squeezed in the same way as before.
In the case of interscale interaction, w has shape [Cout, Cin,KS , Nb]. We iterate over all scales in interaction, shift f for each scale, choose a corresponding part of w, and apply convHH to them. We sum the obtained KS results afterwards.

heading: 5 RELATED WORK

text:
Various works on group-equivariant convolutional networks have been published recently. These works have considered roto-translation groups in 2D Cohen & Welling (2016a); Hoogeboom et al. (2018); Worrall et al. (2017); Weiler & Cesa (2019) and 3D Worrall & Brostow (2018); Kondor (2018); Thomas et al. (2018) and rotation equivariant networks in 3D Cohen et al. (2017); Esteves et al. (2018); Cohen et al. (2019). In Freeman & Adelson (1991) authors describe the algorithm for designing steerable filters for rotations. Rotation steerable filters are used in Cohen & Welling (2016b); Weiler et al. (2018a;b) for building equivariant networks. In Jacobsen et al. (2017) the authors build convolutional blocks locally equivariant to arbitrary k-parameter Lie group by using a steerable basis. And in Murugan et al. the authors discuss the approach for learning steerable filters from data. To date, the majority of papers on group equivariant networks have considered rotations in 2D and 3D, but have not payed attention to scale symmetry. As we have argued above, it is a fundamentally different case.
Many papers and even conferences have been dedicated to image scale-space — a concept where the image is analyzed together with all its downscaled versions. Initially introduced in Iijima (1959) and later developed by Witkin (1987); Perona & Malik (1990); Lindeberg (2013) scale space relies on the scale symmetry of images. The differential structure of the image Koenderink (1984) allows one to make a connection between image formation mechanisms and the space of solutions of the 2- dimensional heat equation, which significantly improved the image analysis models in the pre-deep learning era.
One of the first works on scale equivariance and local scale invariance in the framework of CNNs was proposed by Xu et al. (2014) named SiCNN. The authors describe the model with siamese CNNs, where the filters of each instance are rescaled using interpolation techniques. This is the simplest case of equivariance where no interaction between different scales is done in intermediate layers. In SI-ConvNet by Kanazawa et al. (2014) the original network is modified such that, in each layer, the input is first rescaled, then convolved and rescaled back to the original size. Finally, the response with the maximum values is chosen between the scales. Thus, the model is locally scale-invariant. In
Marcos et al. (2018), in the SEVF model, the input of the layers is rescaled and convolved multiple times to form vector features instead of scalar ones. The length of the vector in each position is the maximum magnitude of the convolution, while the direction of the angle encodes the scale of the image which gave this response. These scale-equivariant networks rely on image rescaling which is quite slow. Worrall & Welling (2019) (DSS) generalize the concept of scale-space to deep networks. They use filter dilation to analyze the images on different scales. While this approach is as fast as the standard CNN, it is restricted only to integer downscale factors 2, 4, 8 . . . . In Ghosh & Gupta (2019), while discussing SS-CNN the authors use scale-steerable filters to deal with scale changes. The paper does not discuss equivariance, which is an important aspect for scale.
We summarize the information about these models in Table 1. In contrast to other scale-equivariant models, SESN uses steerable filters which allows for fast scale-convolution with no limitation of flexibility. With the framework of Scale-Equivariant Convolutional Networks we are free to build both equivariant and invariant models of different kinds.

heading: 6 EXPERIMENTS

text:
In this section we conduct the experiments and compare various methods for working with scale variations in input data. Alongside SESN, we test local scale invariant SI-ConvNet and SS-CNN, scale equivariant SiCNN, SEVF and DSS. For SEVF, DSS and SS-CNN we use the code provided by authors, while for others we reimplement the main buildings blocks.
We provide additional experimental results on time performance of all these methods in Appendix B. Due to the algorithm proposed in Section 4 SESN allows for training several times faster than other methods which rely on image rescaling.

heading: 6.1 EQUIVARIANCE ERROR

text:
We have presented scale-convolution which is equivariant to scale transformation and translation for continuous signals. While translation equivariance holds true even for discretized signals and filters, scale equivariance may not be exact. Therefore, before starting any experiments, we check to which degree the predicted properties of scale-convolution hold true. We do so by measuring the difference ∆ = ‖[LsΦ(f) − ΦLs(f)‖22/‖LsΦ(f)‖22, where Φ is scale-convolution with randomly initialized weights.
In case of perfect equivariance the difference is equal to zero. We calculate the error on randomly sampled images from the STL-10 dataset Coates et al. (2011). The results are represented in Figure 2. The networks on the left and on the middle plots do not have interscale interactions. The networks on the middle and on the right plots consist of just one layer. We use NS = 5, 13, 5 scales for the networks on the left, the middle, and the right plots respectively. While discretization introduces some error, it stays very low, and is not much higher than 6% for the networks with 50 layers. The difference, however, increases if the input image is downscaled more than 16 times. Therefore, we are free to use deep networks. However, we should pay extra attention to extreme cases where scale changes are of very big magnitude. These are quite rare but still appear in practice. Finally, we see that using SESN with interscale interaction introduces extra equivariance error due to the truncation of S. We will build the networks with either no scale interaction or interaction of 2 scales.

heading: 6.2 MNIST-SCALE

text:
Following Kanazawa et al. (2014); Marcos et al. (2018); Ghosh & Gupta (2019) we conduct experiments on the MNIST-scale dataset. We rescale the images of the MNIST dataset LeCun et al. (1998) to 0.3 − 1.0 of the original size and pad them with zeros to retain the initial resolution. The scaling factors are sampled uniformly and independently for each image. The obtained dataset is then split into 10,000 for training, 2,000 for evaluation and 50,000 for testing. We generate 6 different realizations and fix them for all experiments.
As a baseline model we use the model described in Ghosh & Gupta (2019), which currently holds the state-of-the-art result on this dataset. It consists of 3 convolutional and 2 fully-connected layers. Each layer has filters of size 7 × 7. We keep the number of trainable parameters almost the same for all tested methods. This is achieved by varying the number of channels. For scale equivariant models we add scale projection at the end of the convolutional block.
For SiCNN, DSS, SEVF and our model, we additionally train counterparts where after each convolution, an extra projection layer is inserted. Projection layers transform vector features in each spatial position of each channel into scalar ones. All of the layers have now scalar inputs instead of
vector inputs. Therefore, we denote these models with “Scalar”. The original models are denoted as “Vector”. The exact type of projection depends on the way the vector features are constructed. For SiCNN, DSS, and SESN, we use maximum pooling along the scale dimension, while for SEVF, it is a calculation of the L2-norm of the vector.
All models are trained with the Adam optimizer Kingma & Ba (2014) for 60 epochs with a batch size of 128. Initial learning rate is set to 0.01 and divided by 10 after 20 and 40 epochs. We conduct the experiments with 4 different settings. Following the idea discussed in Ghosh & Gupta (2019), in addition to the standard setting we train the networks with input images upscaled to 56 × 56 using bilinear interpolation. This results in all image transformations performed by the network becoming more stable, which produces less interpolation artifacts. For both input sizes we conduct the experiments without data augmentation and with scaling augmentation, which results in 4 setups in total. We run the experiments on 6 different realizations of MNIST-scale and report mean ± std calculated over these runs.
The obtained results are summarized in Table 2. The reported errors may differ a bit from the ones in the original paper because of the variations in generated datasets and slightly different training procedure. Nevertheless, we try to keep our configuration as close as possible to Ghosh & Gupta (2019) which currently demonstrated the best classification accuracy on MNIST-scale. For example, SS-CNN reports error of 1.91 ± 0.04 in Ghosh & Gupta (2019) while it has 1.84 ± 0.10 in our experiments.
SESN significantly outperforms other methods in all 4 regimes. “Scalar” versions of it already outperform all previous methods, and “Vector” versions make the gain even more significant. The global architectures of all models are the same for all rows, which indicates that the way scale convolution is done plays an important role.
6.3 STL-10
In order to evaluate the role of scale equivariance in natural image classification, we conduct the experiments on STL-10 dataset Coates et al. (2011). This dataset consists of 8,000 training and 5,000 testing labeled images. Additionally, it includes 100,000 unlabeled images. The images have a resolution of 96×96 pixels and RGB channels. Labeled images belong to 10 classes such as bird, horse or car. We use only the labeled subset to demonstrate the performance of the models in the low data regime.
The dataset is normalized by subtracting the per-channel mean and dividing by the per-channel standard deviation. During training, we augment the dataset by applying 12 pixel zero padding and randomly cropping the images to size 96 × 96. Additionally, random horizontal flips with probability 50% and Cutout DeVries & Taylor (2017) with 1 hole of 32 pixels are used.
As a baseline we choose WideResNet Zagoruyko & Komodakis (2016) with 16 layers and a widening factor of 8. We set dropout probability to 0.3 in all blocks. We train SESN-A with just vector features. For SESN-B we use maximum scalar projection several times in the intermediate layers, and for SESN-C we use interscale interaction.
All models are trained for 1000 epochs with a batch size of 128. We use SGD optimizer with Nesterov momentum of 0.9 and weight decay of 5 · 10−4. The initial learning rate is set to 0.1 and divided by 5 after 300, 400, 600 and 800 epochs.
The results are summarized in Table 3. We found SEVF training unstable and therefore do not include it in the table. Pure scale-invariant SI-ConvNet and SS-CNN demonstrate significantly worse results than the baseline. We note the importance of equivariance for deep networks. We also find that SESN-C performs significantly worse than SESN-A and SESN-B due to high equivariance
error caused by interscale interaction. SESN-B significantly improves the results of both WRN and DSS due to the projection between scales. The maximum scale projection makes the weights of the next layer to have a maximum receptive field in the space of scales. This is an easy yet effective method for capturing the correlations between different scales. This experiment shows that scale-equivariance is a very useful inductive bias for natural image classification with deep neural networks.
To the best of our knowledge, the proposed method achieves a new state-of-the-art result on the STL-10 dataset in the supervised learning setting. The previous lowest error is demonstrated in Ulicny et al. (2019). The authors propose Harm WRN — a network where the convolutional kernels are represented as a linear combination of Discrete Cosine Transform filters.

heading: 7 DISCUSSION

text:
In this paper, we have presented the theory of Scale-Equivariant Steerable Networks. We started from the scaling transformation and its application to continuous functions. We have obtained the exact formula for scale-equivariant mappings and demonstrated how it can be implemented for discretized signals. We have demonstrated that this approach outperforms other methods for scale-equivariant and local scale-invariant CNNs. It demonstrated new state-of-the-art results on MNIST-scale and on the STL-10 dataset in the supervised learning setting.
We suppose that the most exciting possible application of SESN is in computer vision for autonomous vehicles. Rapidly changing distances between the objects cause significant scale variations which makes this well suited for our work. We especially highlight the direction of siamese visual tracking where the equivariance to principle transformations plays an important role.

heading: ACKNOWLEDGMENTS

text: We thank Daniel Worrall for insightful discussion, Thomas Andy Keller, Victor Garcia, Artem Moskalev and Konrad Groh for valuable comments and feedback.

heading: A PROOF OF EQUIVARIANCE

text:
Let us first show that scale-convolution defined in Equation 6 is equivariant to translations.
[Lt̂[f ] ?H ψσ](s, t) = ∑ s′ [Lt̂[f ](s ′, ·) ? ψsσ(s−1s′, ·)](t)
= ∑ s′ Lt̂[f(s ′, ·) ? ψsσ(s−1s′, ·)](t)
= Lt̂ {∑ s′ [f(s′, ·) ? ψsσ(s−1s′, ·)] } (t)
= Lt̂[f ?H ψσ](s, t)
(11)
Now we show that scale convolution is equivariant to scale transformations: [Lŝ[f ] ?H ψσ](s, t) = ∑ s′ [Lŝ[f ](s ′, ·) ? ψsσ(s−1s′, ·)](t)
= ∑ s′ Lŝ[f(ŝ −1s′, ·) ? ψŝ−1sσ(s−1s′, ·)](t)
= ∑ s′′ [f(s′′, ·) ? ψŝ−1sσ(ŝs−1s′′, ·)](ŝ−1t) = [f ?H ψσ](ŝ −1s, ŝ−1t)
= Lŝ[f ?H ψσ](s, t)
(12)
Finally, we can use the property of semidirect product of groups
Lŝt̂[f ] ?H ψσ = LŝLt̂[f ] ?H ψσ = Lŝ[Lt̂[f ] ?H ψσ] = LŝLt̂[f ?H ψσ] = Lŝt̂[f ?H ψσ] (13)

heading: B TIME PERFORMANCE

text:
We report the average time per epoch of different methods for scale equivariance and local scale invariance in Table 4. Experimental setups from Section 6.2 are used. We used 1 Nvidia GeForce GTX 1080Ti GPU for training the models.
The methods relying on image rescaling techniques during training (SiCNN, SI-ConvNet, SEVF) demonstrate significantly worse time performance that the ones, using either steerable filters or filter dilation. Additionally, we see that our method outperforms SS-CNN by a wide margin. Despite the similar filter sizes and comparable number of parameters between SS-CNN and SESN Scalar, the second one demonstrates significantly better results due to the algorithm proposed in Section 4. Finally, DSS performs slightly faster in some cases than our method as each convolution involves less FLOPs. Dilated filters are sparse, while steerable filters are dense.

heading: C BASIS

text:
Assuming that the center of the filter is point (0, 0) in coordinates (x, y), we use the filters of the following form:
ψσ(x, y) = A 1
σ2 Hn (x σ ) Hm ( y σ ) exp [ − x 2 + y2 2σ2 ] (14)
HereA is a constant independent on σ, Hn — Hermite polynomial of the n-th order. We iterate over increasing pairs of n,m to generate the required number of functions.

heading: D MODEL CONFIGURATION

text:
D.1 MNIST-SCALE
D.2 STL-10

title: SCALE-EQUIVARIANT STEERABLE NETWORKS