summaries: This paper provides a theoretical analysis of the inner workings of multi-task learning methods, based on a random matrix analysis applied to Gaussian mixture data model. The analysis is based on MTL LS-SVM with data from a Gaussian mixture model, where the bias of MTL LS-SVM is shown and a simple method is proposed to correct it. Experiments are conducted on a synthetic dataset and image classification task, where superior performance is shown in addition to the theoretical guarantees.

summaries: The paper provides interesting theoretical insights in multi-task learning using common and specific parameters modeling framework and based on least-squares SVM. Especially, it is theoretically established that the standard MTL LS-SVM is biased. Thereon a method derived from the analysis is proposed to correct the bias and allows to achieve enhanced performances. Empirical evaluations highlight the effectiveness of the method.

summaries: The paper considers the multitask least-square SVM problem. Such a problem consists of k SVM tasks, each being a binary classification problem. The normal vector of the separating hyperplane in each task is “close” to each other, reflecting the commonality of the tasks. For an input data point, the problem asks to predict the classification of the input data point for a given task. This problem has a standard optimization formulation.

abstractText: This article provides theoretical insights into the inner workings of multi-task and transfer learning methods, by studying the tractable least-square support vector machine multi-task learning (LS-SVM MTL) method, in the limit of large (p) and numerous (n) data. By a random matrix analysis applied to a Gaussian mixture data model, the performance of MTL LS-SVM is shown to converge, as n, p→∞, to a deterministic limit involving simple (small-dimensional) statistics of the data. We prove (i) that the standard MTL LS-SVM algorithm is in general strongly biased and may dramatically fail (to the point that individual single-task LS-SVMs may outperform the MTL approach, even for quite resembling tasks): our analysis provides a simple method to correct these biases, and that we reveal (ii) the sufficient statistics at play in the method, which can be efficiently estimated, even for quite small datasets. The latter result is exploited to automatically optimize the hyperparameters without resorting to any cross-validation procedure. Experiments on popular datasets demonstrate that our improved MTL LS-SVM method is computationally-efficient and outperforms sometimes much more elaborate state-of-the-art multi-task and transfer learning techniques.

authors:
Arvind Agarwal
Samuel Gerber
Hal Daume

title: Learning multiple tasks using manifold regularization

venue: In Advances in neural information processing systems,

authors:
Murray Aitkin
Nicholas Longford

title: Statistical modelling issues in school effectiveness studies

venue: Journal of the Royal Statistical Society: Series A (General),

authors:
Greg M Allenby
Peter E Rossi

title: Marketing models of consumer heterogeneity

venue: Journal of econometrics,

authors:
Andreas Argyriou
Theodoros Evgeniou
Massimiliano Pontil

title: Multi-task feature learning

title: A bayesian/information theoretic model of learning to learn via multiple task sampling

venue: Machine learning,

title: A model of inductive bias learning

venue: Journal of artificial intelligence research,

authors:
Shai Ben-David
Reba Schuller

title: Exploiting task relatedness for multiple task learning

venue: In Learning Theory and Kernel Machines,

title: Bayesian multitask learning with latent hierarchies

venue: arXiv preprint arXiv:0907.0783,

authors:
Zeyu Deng
Abla Kammoun
Christos Thrampoulidis

title: A model of double descent for highdimensional binary linear classification

venue: arXiv preprint arXiv:1911.05822,

title: On information plus noise kernel random matrices

venue: The Annals of Statistics,

authors:
Theodoros Evgeniou
Massimiliano Pontil

title: Regularized multi–task learning

venue: In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Pinghua Gong
Jieping Ye
Chang-shui Zhang

title: Multi-stage multi-task feature learning. In Advances in neural information processing

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Gregory Griffin
Alex Holub
Pietro Perona

title: Caltech-256 object category

venue: dataset. technical report,

authors:
Samitha Herath
Mehrtash Harandi
Fatih Porikli

title: Learning an invariant hilbert space for domain adaptation

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Judy Hoffman
Erik Rodner
Jeff Donahue
Trevor Darrell
Kate Saenko

title: Efficient learning of domain-invariant image representations

venue: arXiv preprint arXiv:1301.3224,

authors:
Yao-Hung Hubert Tsai
Yi-Ren Yeh
Yu-Chiang Frank Wang

title: Learning cross-domain landmarks for heterogeneous domain adaptation

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Zhenyu Liao
Romain Couillet

title: A large dimensional analysis of least squares support vector machines

venue: IEEE Transactions on Signal Processing,

authors:
Xiaoyi Mai
Romain Couillet

title: A random matrix analysis and improvement of semi-supervised learning for large dimensional data

venue: The Journal of Machine Learning Research,

authors:
Xiaoyi Mai
Zhenyu Liao
Romain Couillet

title: A large scale analysis of logistic regression: Asymptotic performance and new insights

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Andreas Maurer
Massi Pontil
Bernardino Romera-Paredes

title: Sparse coding for multitask and transfer learning

venue: In International conference on machine learning,

authors:
Guillaume Obozinski
Ben Taskar
Michael Jordan

title: Multi-task feature selection

venue: Statistics Department, UC Berkeley, Tech. Rep,

authors:
Sinno Jialin Pan
Ivor W Tsang
James T Kwok
Qiang Yang

title: Domain adaptation via transfer component analysis

venue: IEEE Transactions on Neural Networks,

authors:
Shibin Parameswaran
Kilian Q Weinberger

title: Large margin multi-task metric learning

authors:
Kate Saenko
Brian Kulis
Mario Fritz
Trevor Darrell

title: Adapting visual category models to new domains

venue: In European conference on computer vision,

authors:
Jie Wang
Jieping Ye

title: Safe screening for multi-task feature learning with multiple data matrices

venue: arXiv preprint arXiv:1505.04073,

authors:
Shuo Xu
Xin An
Xiaodong Qiao
Lijun Zhu
Lin Li

title: Multi-output least-squares support vector regression machines

venue: Pattern Recognition Letters, 34:1078–1084,

authors:
Wenlu Zhang
Rongjian Li
Tao Zeng
Qian Sun
Sudhir Kumar
Jieping Ye
Shuiwang Ji

title: Deep model based transfer and multi-task learning for biological image analysis

venue: IEEE transactions on Big Data,

heading: 1 INTRODUCTION

text:
The advent of elaborate learning machines capable to surpass human performances on dedicated tasks has reopened past challenges in machine learning. Transfer learning, and multitask learning (MTL) in general, by which known tasks are used to help a machine learn other related tasks, is one of them. The particularly interesting aspects of multi-task learning lie in the possibility (i) to exploit the resemblance between the datasets associated to each task so the tasks “help each other” and (ii) to train a machine on a specific target dataset comprised of few labelled data by exploiting much larger labelled datasets, however composed of different data. Practical applications are numerous, ranging from the prediction of student test results for a collection of schools (Aitkin & Longford, 1986), to survival of patients in different clinics, to the value of many possibly related financial indicators (Allenby & Rossi, 1998), to the preference modelling of individuals in a marketing context, etc.
Since MTL seeks to improve the performance of a task with the help of related tasks, a central issue to (i) understand the functioning of MTL, (ii) adequately adapt its hyperparameters and eventually (iii) improve its performances consists in characterizing how MTL relates tasks to one another and in identifying which features are “transferred”. The article aims to decipher these fundamental aspects for sufficiently general data models.
Several data models may be accounted for to enforce relatedness between tasks. A common assumption is that the data lie close to each other in a geometrical sense (Evgeniou & Pontil, 2004), live in a low dimensional manifold (Agarwal et al., 2010), or share a common prior (Daumé III, 2009). We follow here the latter assumption in assuming that, for each task, the data arise from a 2-class Gaussian mixture.1
Methodologically, in its simplest approach, MTL algorithms can be obtained from a mere extension of support vector machines (SVM), accounting for more than one task. That is, instead of finding the hyperplane (through its normal vector ω) best separating the two classes of a unique dataset, (Evgeniou & Pontil, 2004) proposes to produce best separating hyperplanes (or normal vectors) ω1, . . . , ωk for each pair of data classes of k tasks, with the additional constraint that the normal vectors take the form ωi = ω0 + vi for some common vector w0 and dedicated vectors vi. The amplitude of the vectors vi is controlled (through an additional hyperparameter) to enforce or relax task relatedness. We study this approach here. Yet, to obtain explicit and thus more insightful results, we specifically resort to a least-square SVM (as proposed e.g., in (Xu et al., 2013)) rather than a margin-based SVM. This only marginally alters the overall behavior of the MTL algorithm and has no impact on the main insights drawn in the article. Moreover, by a now well-established universality argument in large dimensional statistics, (Mai et al., 2019) show that quadratic (leastsquare) cost functions are asymptotically optimal and uniformly outperform alternative costs (such as margin-based methods or logistic approaches), even in a classification setting. This argument further motivates the choice of considering first and foremost the LS-SVM version of MTL-SVM.
Technically, the article exploits the powerful random matrix theory to study the performance of the MTL least-square SVM algorithm (MTL LS-SVM) for data arising from a Gaussian mixture model, assuming the total number n and dimension p of the data are both large, i.e., as n, p → ∞ with p/n→ c ∈ (0,∞). As such, our work follows after the recent wave of interest into the asymptotics of machine learning algorithms, such as studied lately in e.g., (Liao & Couillet, 2019; Deng et al., 2019; Mai & Couillet, 2018; El Karoui et al., 2010). Our analysis reveals the following major conclusions:
• we exhibit the sufficient statistics, which concretely enable task comparison in the MTL LS-SVM algorithm; we show that, even when data are of large dimensions (p 1), these statistics remain small dimensional (they only scale with the number k of tasks); • while it is conventional to manually set labels associated to each dataset within {−1, 1},
we prove that this choice is largely suboptimal and may even cause MTL to severely fail (causing “negative transfer”); we instead provide the optimal values for the labels of each dataset, which depend on the sought-for objective: these optimal values are furthermore easily estimated from very few training data (i.e., no cross-validation is needed); • for unknown new data x, the MTL LS-SVM algorithm allocates a class based on the
comparison of a score g(x) to a threshold ζ, usually set to zero. We prove that, depending on the statistics and number of elements of the training dataset, a bias is naturally induced that makes ζ = 0 a largely suboptimal choice in general. We provide a correction for this bias, which again can be estimated from the training data alone; • we demonstrate on popular real datasets that our proposed optimized MTL LS-SVM is both
resilient to real data and also manages, despite its not being a best-in-class MTL algorithm, to rival and sometimes largely outperform competing state-of-the-art algorithms.
These conclusions thus allow for an optimal use of MTL LS-SVM with performance-maximizing hyperparameters and strong theoretical guarantees. As such, the present article offers through MTL LS-SVM a viable fully-controlled (even better performing) alternative to state-of-the-art MTL.
Reproducibility. Matlab and Julia codes for reproducing the results of the article are available in the supplementary materials.
Notation. e[n]m ∈ Rn is the canonical vector of Rn with [e[n]m ]i = δmi. Moreover, e[2k]ij = e [2k] 2(i−1)+j . Similarly, E[n]ij ∈ Rn×n is the matrix with [E [n] ij ]ab = δiaδjb. The notations A ⊗ B and A B
1In the supplementary material, we extend this setting to a much broader and more realistic scope, and justify in passing the relevance of a Gaussian mixture modelling to address multi-task learning with real data.
for matrices or vectors A,B are respectively the Kronecker and Hadamard products. Dx is the diagonal matrix containing on its diagonal the elements of the vector x and Ai· is the i-th row of A. The notation Å is used when a centering operation is performed on the matrix or vector A. Uppercase calligraphic letters (A,K,Γ,M, V ,...) are used for deterministic small dimensional matrices. Finally, 1m and Im are respectively the vector of all one’s of dimension m and the identity matrix of dimension m×m. The index pair i, j generally refers to Class j in Task i.

heading: 2 RELATED WORKS

text:
Let us first point out the difference between MTL and transfer learning: while MTL makes no distinction between tasks and aims to improve the performance of all tasks, transfer learning aims to maximize the performance of a target task with the help of all source tasks. Yet, both methods mostly sharing the same learning process, in this section, we mainly focus on the MTL literature, which is divided into parameter-based versus feature-based MTL.
In the parameter-based MTL approach, the tasks are assumed to share some parameters (e.g., the hyperplanes best separating each class) or their hyperparameters have a common prior distribution. Existing learning methods (SVM, logistic regression, etc.) can then be appropriately modified to incorporate these relatedness assumptions. In this context, (Evgeniou & Pontil, 2004; Xu et al., 2013; Parameswaran & Weinberger, 2010) respectively adapt the SVM, LS-SVM, and Large Margin Nearest Neighbor (LMNN) algorithms to the MTL paradigm. The present article borrows ideas from Evgeniou & Pontil (2004); Xu et al. (2013).
In the feature-based MTL approach, the tasks data are instead assumed to share a low-dimensional common representation. In this context, most of the works aim to determine a mapping of the ambient data space into a low-dimensional subspace (through sparse coding, deep neural networks, principal component analysis, etc.) in which the tasks have high similarity (Argyriou et al., 2007; Maurer et al., 2013; Zhang et al., 2016; Pan et al., 2010); other works simply use a feature selection method by merely extracting a subset of the original feature space (Obozinski et al., 2006; Wang & Ye, 2015; Gong et al., 2012). We must insist that, in the present work, our ultimate objective is to study and improve “data-generic” MTL mechanisms under no structural assumption on the data; this approach is quite unlike recent works exploiting convolutive techniques in deep neural nets or low dimensional feature-based methods to perform transfer or multi-task learning mostly for computer vision.
From a theoretical standpoint though, few works have provided a proper understanding of the various MTL algorithms. To our knowledge, the only such results arise from elementary learning theory (Rademacher complexity, VC dimension, covering number, stability) and only provide loose performance bounds (Baxter, 2000; Ben-David & Schuller, 2003; Baxter, 1997). As such, the present work fills a long-standing gap in the MTL research.

heading: 3 THE MULTI-TASK LEARNING SETTING

text:
Let X ∈ Rp×n be a collection of n independent data vectors of dimension p. The data are divided into k subsets attached to individual “tasks”. Specifically, letting X = [X1, . . . , Xk], Task i is a binary classification problem from the training samples Xi = [X (1) i , X (2) i ] ∈ Rp×ni with X (j) i = [x (j) i1 , . . . , x (j) inij ] ∈ Rp×nij the nij vectors of class j ∈ {1, 2} for Task i. In particular, n = ∑k i=1 ni and ni = ni1 + ni2 for each i ∈ {1, . . . , k}. To each xil ∈ Rp of the training set is attached a corresponding “label” (or score) yil ∈ R. We denote yi = [yi1, . . . , yini ]
T ∈ Rni the vector of all labels for Task i, and y = [yT1 , . . . , yTk ]T ∈ Rn the vector of all labels. These labels are generally chosen to be ±1 but, for reasons that will become clear in the course of the article, we voluntarily do not enforce binary labels here.
Before detailing the multitask classification scheme, a preliminary task-wise centering operation is performed on the data, i.e., we consider in the following the datasets
X̊i = Xi ( Ini − 1
ni 1ni1 T ni
) , ∀i ∈ {1, . . . , k}.
As such, we systematically work on the labeled datasets (X̊1, y1), . . . , (X̊k, yk). Remark 1 in the supplementary material motivates this choice, which avoids extra biases produced by the algorithm.

heading: 3.1 THE OPTIMIZATION FRAMEWORK

text:
The multitask learning least square support vector machine (MTL LS-SVM) aims to predict, for input vectors x ∈ Rp not belonging to the training set, their associated score y upon which a decision on the class allocation of x is taken, for a given target task. To this end, based on the labeled sets (X̊1, y1), . . . , (X̊k, yk), MTL LS-SVM determines the normal vectorsW = [ω1, ω2, . . . , ωk] ∈ Rp×k and intercepts b = [b1, b2, . . . , bk]T ∈ Rk defining k separating hyperplanes for the corresponding k binary classification tasks. In order to account for task relatedness, each ωi assumes the form ωi = ω0 + vi for some common ω0 ∈ Rp and task-dedicated vi ∈ Rp. Formally, writing V = [v1, . . . , vk] ∈ Rp×k (so that W = ω01>k + V ) and following the work of (Evgeniou & Pontil, 2004; Xu et al., 2013), the optimization function is given by
min (ω0,V,b)∈
Rp×Rp×k×Rk
1
2λ ‖ω0‖2 +
1
2 k∑ i=1 ‖vi‖2 γi + 1 2 k∑ i=1 ‖ξi‖2, ξi = yi − (X̊Ti ωi + bi1ni), 1 ≤ i ≤ k.
In this expression, the parameter λ enforces more task relatedness while the parameters γ1, . . . , γk enforce better classification of the data in their respective classes.
Being a quadratic optimization problem under linear equality constraints, ω0, V, b are obtained explicitly (see details in Section 1 of the supplementary material). The solution is best described through the expression of the hyperplanes ω1, . . . , ωk ∈ Rp which take the form:
ωi = ( e [k]T i ⊗ Ip ) AZα,
and b = (PTQP )−1PTQy, where α = Q(y−Pb) = Q 12 (In−Q 1 2P (PTQP )−1PTQ 1 2 )Q 1 2 y ∈ Rn is the Lagrangian dual and
Q =
( 1
kp ZTAZ + In
)−1 ∈ Rn×n, Z = k∑ i=1 E [k] ii ⊗ X̊i ∈ R pk×n
A = ( Dγ + λ11T ) ⊗ Ip ∈ Rkp×kp, P = k∑ i=1 E [k] ii ⊗ 1ni ∈ R n×k
with γ = [γ1, . . . , γk]T and Dγ = diag(γ).
MTL LS-SVM differs from a single-task joint LS-SVM for all data in that the data X̊1, . . . , X̊k are not treated simultaneously but through k distinct filters: this explains why Z ∈ Rkp×n is not the mere concatenation [X̊1, . . . , X̊k] but a block-diagonal structure isolating each X̊i. As such, the X̊i’s-relating matrix A plays an important role in the MTL learning process.
With this formulation for the solution (W, b), the prediction of the class of any new data point x ∈ Rp for the target Task i is then obtained from the classification score
gi(x) = 1
kp
( e
[k] i ⊗ x̊ )T AZα+ bi (1)
where x̊ = x− 1niXi1ni is a centered version of x with respect to the training dataset for Task i.

heading: 3.2 LARGE DIMENSIONAL STATISTICAL MODELLING

text:
The first objective of the article is to quantify the MTL performance, and thus of the (a priori intricate) statistics of gi(x), under a sufficiently simple but telling Gaussian mixture model for training and test data. Assumption 1 (Distribution of X and x). The columns of [X,x] are independent Gaussian random variables. Specifically, the nij samples x (j) i1 , . . . , x (j) inij
of class j for Task i are independentN (µij , Ip) vectors, and we let ∆µi ≡ µi1 − µi2. As for x, it follows an independent N (µx, Ip) vector.
In the supplementary material, Assumption 1 is relaxed to [X,x] arising from a generative model of the type x(j)il = hij(z (j) il ) for z (j) il ∼ N (0, Ip) and hij : Rp → Rp a 1-Lipschitz function. This model encompasses extremely realistic data models, including data arising from generative networks (e.g., GANs (Goodfellow et al., 2014)) and is shown in the supplementary material to be universal in the sense that, as n, p→∞, the asymptotic performances of MTL LS-SVM only depend on the statistical means and covariances of the x(j)il : the performances under complex mixtures are thus proved to coincide with those under an elementary Gaussian mixture. This generalized study however comes at the expense of more complex definitions and formulas, which impedes readability; hence the simpler isotropic Gaussian mixture model here.
Our central technical approach for the performance evaluation of the MTL LS-SVM algorithm consists in placing ourselves under the large p, n regime of random matrix theory. Assumption 2 (Growth Rate). As n → ∞, n/p → c0 > 0 and, for 1 ≤ i ≤ k, 1 ≤ j ≤ 2, nij n → cij > 0. We let ci = ci1 + ci2, c = [c1, . . . , ck] T ∈ Rk.
With these notations and assumptions, we are in position to present our main theoretical results.

heading: 4 THE MULTI-TASK LEARNING ANALYSIS

heading: 4.1 TECHNICAL STRATEGY AND NOTATIONS

text:
To evaluate the statistics of gi(x) (equation 1), we resort to finding so-called deterministic equivalents for the matrices Q, AZQ, etc., which appear at the core of the formulation of gi(x). Those are provided in Lemma 1 of the supplementary material. Our strategy then consists in “decoupling” the effect of the data statistics from those of the MTL hyperparameters λ, γ1, . . . , γk. Specifically, we extract two fundamental quantities for our analysis: a data-related matrixM ∈ R2k×2k and a hyperparameter matrix A ∈ Rk×k: M = k∑
i,j=1
∆µTi ∆µj
( E
[k] ij ⊗ cic T j ) , ci =  ci2ci √ ci1ci − ci1ci √ ci2 ci  , A = (Ik +D− 12∆̃ (Dγ + λ1k1Tk)−1D− 12∆̃ )−1 where ∆̃ = [∆̃1, . . . , ∆̃k]T are the unique positive solutions to the implicit system ∆̃i = cic0 −Aii (this is implicit because A is a function of the ∆̃j’s). In passing, it will appear convenient to use the shortcut notation ∆̄ = [∆̄11, . . . , ∆̄k2]T ∈ R2k where ∆̄ij = cijci c0∆̃i.
We will see thatM plays the role, in the limit of large p, n, of a sufficient statistic for the performance of the MTL LS-SVM algorithm only involving (i) the data statistics µ11, . . . , µk2 and (ii) the (limiting) relative number c11/c1, . . . , ck2/ck of elements per class in each task. As for A, it captures the information about the impact of the hyperparameters λ, γ1, . . . , γk and of the dimension ratios c1, . . . , ck and c0. These two matrices will be combined in the core matrix Γ ∈ R2k×2k of the upcoming MTL LS-SVM performance analysis, defined as Γ = ( I2k + ( A⊗ 121T2 ) M
)−1 where we recall that ‘ ’ is the Hadamard (element-wise) matrix product.
We raised in the introduction of Section 3 that we purposely relax the binary “labels” yij associated to each datum xij in each task to become “scores” yij ∈ R. This will have fundamental consequences to the MTL performance. Yet, since xi1, . . . , xini are i.i.d. data vectors, we impose equal scores yi1 = . . . = yini within each class. As such, we may reduce the complete score vector y ∈ Rn
under the form y = [ ỹ111Tn11 , . . . , ỹk21 T nk2 ]T for ỹ = [ỹ11, . . . , ỹk2]T ∈ R2k. From Remark 1 of the supplementary material, the performances of MTL are insensitive to a constant shift in the scores yi1 and yi2 of every given Task i: as such, the recentered version ˚̃y = [̊ỹ11, . . . ,˚̃yk2]T of ỹ, where ˚̃yij = ỹij − (ni1ni ỹi1 + ni2 ni ỹi2) will be central in the upcoming results.

heading: 4.2 MAIN RESULTS

text:
Theorem 1 (Asymptotics of gi(x)). Under Assumptions 1–2, for x ∼ N (µx, Ip) with µx = µij , gi(x)−Gij → 0, Gij ∼ N (mij , σ2i )
in distribution where, letting m = [m11, . . . ,mk2]T,
m = ỹ −D− 1 2
∆̄ ΓD
1 2
∆̄ ˚̃y, σ2i =
1
∆̃i ˚̃yTD
1 2 ∆̄ Γ ( DKi. c ⊗12 + Vi ) ΓD 1 2 ∆̄ ˚̃y
with Vi = 1c0 (AD c0Ki·c +e[k]i A⊗ 121 T 2 ) M and K = [A A](Ik −D c0kc [A A]) −1.
As anticipated, the statistics of the classification scores gi(x) mainly depend on the data statistics µi′j′ and on the hyperparameters λ and γ1, . . . , γk through the matrix Γ (and more marginally through Vi and K for the variances). Since gi(x) has a Gaussian limit centered about mij , the (asymptotic) standard decision for x to be allocated to Class 1 (x→ C1) or Class 2 (x→ C2) for Task i is obtained by the “averaged-mean” test
gi(x) C1 ≷ C2 1 2 (mi1 +mi2) (2)
the classification error rate i1 ≡ P (x→ C2|x ∈ C1) of which is then i1 ≡ P ( gi(x) ≥
mi1 +mi2 2 ∣∣∣x ∈ C1) = Q(mi1 −mi2 2σi ) + o(1) (3)
with mij , σi as in Theorem 1 and Q(t) = ∫∞ t e− u2 2 du.
Further comment on Γ is due before moving to practical consequences of Theorem 1. From the expression (A⊗ 121T2 )−1 M, we observe that: (i) if λ 1, then A is diagonal dominant and thus “filters out” in the Hadamard product all off-diagonal entries ofM, i.e., all cross-terms ∆µTi ∆µj for i 6= j, therefore refusing to exploit the correlation between tasks; (ii) if instead λ is not small, A may be developed (using the Sherman-Morrison matrix inverse formulas) as the sum of a diagonal matrix, which again filters out the ∆µTi ∆µj for i 6= j, and of a rank-one matrix which instead performs a weighted sum (through the γi’s and the ∆̃i’s) of the entries ofM. Specifically, letting γ−1 = (γ−11 , . . . , γ −1 k ) T in the expression of A, we have ( Dγ + λ1k1Tk )−1 = D−1γ − λγ−1γ−1T 1+λ 1k ∑k i=1 γ −1 i . As such, disregarding the regularization effect of the ∆̃i’s, the off-diagonal ∆µTi ∆µj entry ofM is weighted with coefficient (γiγj)−1: the impact of the γi’s is thus strongly associated to the relevance of the correlation between tasks.
A fundamental aspect of Theorem 1 is that it concludes that the performances of the large dimensional (n, p 1) classification problem at hand merely boils down to 2k-dimensional statistics, as all objects defined in the theorem statement are at most of size 2k. More importantly from a practical perspective, these “sufficient statistics” are easily amenable to fast and efficient estimation: it only requires a few training samples to estimate all quantities involved in the theorem. This, as a corollary, lets one envision the possibility of efficient transfer learning methods based on very scarce data samples as discussed in Remark 1.
Estimating mij and σi not only allows one to anticipate theoretical performances but also enables the actual estimation of the decision threshold 12 (mi1 +mi2) in equation 2 and opens the possibility to largely optimize MTL LS-SVM through an (asymptotically) optimal choice of the training scores ỹ. Indeed, the asymptotics in Theorem 1 depend in an elegant manner on the training data labels (scores) ỹ. Since the variance σ2i is independent of the classes, we easily determine the vector ỹ = ỹ ? minimizing the misclassification probability for Task i as
ỹ? = arg max ỹ∈R2k
(mi1 −mi2)2
σ2i = arg max ỹ∈R2k
‖ỹT(I2k −D 1 2 ∆̄ ΓD− 1 2 ∆̄ )(e [2k] i1 − e [2k] i2 )‖2
ỹTD 1 2
∆̄ Γ(DKi. c ⊗12 + Vi)ΓD
1 2 ∆̄ ỹ
the solution of which is explicit
ỹ? = D− 1 2 ∆̄ Γ−1H[(A⊗121T2 ) M]D − 12 ∆̄ (e [2k] i1 − e [2k] i2 ), H ≡ (DKi. c ⊗12 + Vi)−1 (4)
with corresponding (asymptotically) optimal classification error i1 (equation 3) given by ?i1 = Q ( 1
2
√ (e
[2k] i1 − e [2k] i2 ) TG(e[2k]i1 − e [2k] i2 )
) , G = D 1 2
∆̄ [(A⊗121T2 ) M ]V−1i [(A⊗121 T 2 ) M ]D
1 2 ∆̄ .
The only non-diagonal matrices in equation 4 are Γ and Vi in whichM plays the role of a “variance profile” matrix. In particular, assuming ∆µTi ∆µ` = 0 for all ` 6= i (i.e., the statistical means of all tasks are orthogonal to those of Task i), then the two rows and columns of M associated to Task i are all zero but on the 2 × 2 diagonal block. Therefore, ỹ? must be filled with zero entries but on its Task i two elements. All other values at the zero entry locations of ỹ? (such as the usual ỹ = [1,−1, . . . , 1,−1]T) would be suboptimal and possibly severely detrimental to the classification performance of Task i (not by altering the means mi1,mi2 but by increasing the variance σ2i ). This extreme example strongly suggests that, in order to maximize the MTL performance on Task i, one must impose low scores ỹjl to all Tasks j strongly different from Task i.
The choice ỹ = [1,−1, . . . , 1,−1]T can also be very detrimental when ∆µTi ∆µj < 0 for some i, j: that is, when the mapping of the two classes within each task is reversed (e.g., if Class 1 in Task 1 is closer to Class 2 than Class 1 in Task 2). In this setting, it is easily seen that ỹ = [1,−1, . . . , 1,−1]T works against the classification and performs much worse than a single-task LS-SVM.
Another interesting conclusion arises from the simplified setting of equal number of samples per task and per class, i.e., n11 = . . . = nk2. In this case, ỹ? = Γ−1H ( (A⊗ 121T2 ) M ) (e [2k] i1 − e [2k] i2 ) in which all matrices are organized in 2× 2 blocks of equal entries. This immediately implies that ỹ?j1 = −ỹ?j2 for all j. So in particular the detection threshold 12 (mi1 +mi2) of the averaged-mean test (equation 2) is zero, as conventionally assumed. In all other settings for the njl’s, it is very unlikely that ỹ?i1 = −ỹ?i2 and the optimal decision threshold must also be estimated. These various conclusions give rise to an improved MTL LS-SVM algorithm. A pseudo-code (Algorithm 1) along with discussions on (i) the estimation of the statistics met in Theorem 1 (Remark 1) and on (ii) its multi-class extension (Remark 2) are covered next. Matlab and Julia implementations of Algorithm 1 and its extensions are proposed in the supplementary material.

heading: 5 EXPERIMENTS

text:
Our theoretical results (the data-driven optimal tuning of MTL LS-SVM, as well as the anticipation of classification performance) find various practical applications and consequences. We exploit them here in the context of transfer learning, first on a binary decision on synthetic data, and then on a multiclass classification on real data.
To this end, and before proceeding to our experimental setup, a few key remarks to practically exploit Theorem 1 are in order. Remark 1 (On the estimation of mij and σi). All quantities defined in Theorem 1 are a priori known, apart from the k2 inner products ∆µTi ∆µj for 1 ≤ i, j ≤ k. For these, define Sil ⊂ {1, . . . , nil} (l = 1, 2) and the corresponding indicator vector jil ∈ Rni with [jij ]a = δa∈Sij . For i = j, further let S ′il ⊂ {1, . . . , nil} with S ′il ∩ Sil = ∅ and the corresponding indicator vector j′il ∈ Rni . Then, the following estimates hold:
∆µTi ∆µj − (
ji1 |Si1| − ji2 |Si2|
)T XTi Xj ( jj1 |Sj1| − jj2 |Sj2| ) = O ( (p min l∈{1,2} {|Sil|, |Sjl|})− 1 2 ) ∆µTi ∆µi − ( ji1 |Si1| − ji2 |Si2| )T XTi Xi ( j′i1 |S ′i1| − j ′ i2 |S ′i2| ) = O ( (p min l∈{1,2} {|Sil|, |S ′il|})− 1 2 ) .
Observe in particular that a single sample (two when i = j) per task and per class (|Sil| = 1) is sufficient to obtain a consistent estimate for all quantities, so long that p is large. Remark 2 (From binary to multiclass MTL). Accessing the vector m in Theorem 1 allows for the extension of the MTL framework to a multiclass-per-task MTL by discarding the well known inherent biases of multiclass SVM. In the context of Li classes for Task i, using a one-versusall approach (for each ` ∈ {1, . . . , Li}, one MTL LS-SVM algorithms with Class “1” being the target class ` and Class “2” all other classes at once), one needs to access Li pairs of values (mi1(`),mi2(`)) and, for a new x, decide on the genuine class of x based on the largest value among gi(x; 1)−mi1(1), . . . , gi(x;Li)−mi1(Li) with gi(x; `) the output score for “Class ` versus all”. For simplicity, from Remark 1 in the supplementary material, one may choose smart shift vectors ȳ(`) ∈ Rk for the scores ỹ(`) ∈ R2k (i.e., replace ỹ(`) by ỹ(`) + ȳ(`)⊗ 12) so that mi1(`) = 0 for each `. Under this shift, the selected class is the one for which gi(x; `) is maximum.
β = −1
β = 0
β = 0.5
β = 1

heading: 5.1 EFFECT OF INPUT SCORE (LABEL) AND THRESHOLD DECISION CHOICES

text:
In order to support the theoretical insights drawn in the article, our first experiment illustrates the effects of the bias in the decision threshold for gi(x) (in general not centered on zero) and of the input score (label) optimization ỹ? on synthetic data.
Specifically, MTL-LSSVM is first applied to the following two-task (k = 2) setting: for Task 1, x
(j) 1l ∼ N ((−1)jµ1, Ip) and for Task 2, x (j) 2l ∼ N ((−1)jµ2, Ip), where µ2 = βµ1 + √ 1− β2µ⊥1 and µ⊥1 is any vector orthogonal to µ1 and β ∈ [0, 1]. This setting allows us to tune, through β, the similarity between tasks. For four different values of β, Figure 1 depicts the distribution of the binary output scores gi(x) both for the classical MTL-LSSVM (top displays) and for our proposed random matrix improved scheme, with optimized input labels (bottom displays).
As a first remark, note that both theoretical prediction and empirical outputs closely fit for all values of β, thereby corroborating our theoretical findings. In practical terms, the figure supports (i) the importance to estimate the threshold decision which is non-trivial (not always close to zero) and (ii) the relevance of an appropriate choice of the input labels to improve the discrimination performance between both classes, especially when the two tasks are not quite related as shown by the classification error presented in red in the figure.

heading: 5.2 MULTICLASS TRANSFER LEARNING

text:
We next turn to the classical Office+Caltech256 (Saenko et al., 2010; Griffin et al., 2007) real data (images) benchmark for transfer learning, consisting of the 10 categories shared by both datasets. For fair comparison with previous works, we compare images using p = 4096 VGG features. Half of the samples of the target is randomly selected for the test data and the accuracy is evaluated over 20 trials. We use here Algorithm 1, the results of which (Proposed) are reported in Table 1 against the non-optimized LS-SVM (Xu et al., 2013) and alternative state-of-the-art algorithms: MMDT, CDLS and ILS.3
3MMDT: Max Margin Domain Transform, proposed in (Hoffman et al., 2013), applies a weighted SVM on a learnt transformation of the source and target. CDLS: Cross-Domain Landmark Selection, proposed in (Hubert Tsai et al., 2016), derives a domain invariant feature space. ILS: Invariant Latent Space, proposed in (Herath et al., 2017), learns an invariant latent space to reduce the discrepancy between source and target.
Algorithm 1 Proposed Multi Task Learning algorithm.
S/T c→ w
w→ c c→ a
a→ c
w→ a a→ d
d→ a
w→ d c→ d
d→ c
a→ w
d→ w
Mean score
LSSVM 96.69 89.90 92.90 90.00 93.80 78.70 93.50 95.00 85.00 90.20 94.70 100 91.70 MMDT 93.90 87.05 90.83 84.40 94.17 86.25 94.58 97.50 86.25 87.23 92.05 97.35 90.96 ILS 77.89 73.55 86.85 76.22 86.22 71.34 74.53 82.80 68.15 63.49 78.98 92.88 77.74 CDLS 97.60 88.30 93.54 88.30 93.54 92.50 93.54 93.75 93.75 88.30 97.35 96.70 93.10 Ours 98.68 89.90 94.40 90.60 94.40 93.80 94.20 100 92.50 89.90 98.70 99.30 94.70
Table 1 demonstrates that our proposed improved MTL LS-SVM, despite its simplicity and unlike the competing methods used for comparison, has stable performances and is quite competitive. We further recall that, in additions to these high levels of performance, the method comes along with theoretical guarantees, which none of the competing works are able to provide.

heading: 6 CONCLUSION: BEYOND MTL

text:
Through the example of transfer learning (and more generally multitask learning), we have demonstrated the capability of random matrix theory to (i) predict and improve the performance of machine learning algorithms and, most importantly, to (ii) turn simplistic (and in theory largely suboptimal) methods, such as here LS-SVM, into competitive state-of-the-art algorithms. As Gaussian mixtures are quite “universal” and thus already appropriate to handle real data (as shown in supplementary material), one may surmise the optimality of the least square approach, thereby opening the possibility to prove that MTL LS-SVM is likely close-to-optimal even in real data settings.
This is yet merely a first step into a generalized use of random matrix theory and large dimensional statistics to devise much-needed low computational cost and explainable, yet highly competitive, machine learning methods from elementary optimization schemes.

heading: ACKNOWLEDGMENTS

text: Couillet’s work is partially supported by MIAI at University Grenoble-Alpes (ANR-19-P3IA-0003) and the HUAWEI LarDist project.


summaries: This paper addresses an important symmetry in meta-learning.  Namely, the context data consists of a set of datapoints in arbitrary order.  The model should thus be permutation equivariant to their order.  At the same time, the data itself may have its own symmetries, e.g. rotation, which the network should likewise be equivariant to.  The authors follow a theory-driven approach, proving in Thm 2 that a function with these two types of symmetries may be factored and represented by a composition of functions reflecting each symmetry individually.  They then design a Neural Process (NP) model, EquivCNP, which reflects this result.  Other works have used permutation equivariance and translation equivariance in NPs, but this is the first to incorporate other symmetry groups.

summaries: The paper provides an extension of convolution conditional neural processes CNPs to more general Lie group equivariant CNPs. The development of the theory seems sufficiently clear to someone more familiar with the field. However, for newer readers, it seems important to be familiar with background concepts and prior work. This is not a penalizing point but rather just an observation.

summaries: This paper presents EquivCNP which extends Conditional Neural Processes (CNP) to incorporate symmetries of the data, e.g. rotation and scaling. The approach utilizes a combination of LieConv (Finzi et al., 2020)  and DeepSet (Zaheer et al., 2017) to achieve the equivariance in the data space and permutation invariance across the samples in a dataset. They provide empirical results on a 1D regression task with synthetic and 2D image completion tasks using digital clock digits dataset which they constructed.

abstractText: We present the group equivariant conditional neural process (EquivCNP), a metalearning method with permutation invariance in a data set as in conventional conditional neural processes (CNPs), and it also has transformation equivariance in data space. Incorporating group equivariance, such as rotation and scaling equivariance, provides a way to consider the symmetry of real-world data. We give a decomposition theorem for permutation-invariant and group-equivariant maps, which leads us to construct EquivCNPs with an infinite-dimensional latent space to handle group symmetries. In this paper, we build architecture using Lie group convolutional layers for practical implementation. We show that EquivCNP with translation equivariance achieves comparable performance to conventional CNPs in a 1D regression task. Moreover, we demonstrate that incorporating an appropriate Lie group equivariance, EquivCNP is capable of zero-shot generalization for an image-completion task by selecting an appropriate Lie group equivariance.

authors:
Brandon Anderson
Truong Son Hy
Risi Kondor

title: Cormorant: Covariant molecular neural networks

venue: In Advances in Neural Information Processing Systems,

authors:
Marcin Andrychowicz
Misha Denil
Sergio Gomez
Matthew W Hoffman
David Pfau
Tom Schaul
Brendan Shillingford
Nando De Freitas

title: Learning to learn by gradient descent by gradient descent

venue: In Advances in neural information processing systems,

title: B-spline cnns on lie groups

venue: arXiv preprint arXiv:1909.12057,

title: Xception: Deep learning with depthwise separable convolutions

authors:
Taco Cohen
Max Welling

title: Group equivariant convolutional networks

authors:
Taco S Cohen
Mario Geiger
Maurice Weiler

title: A general theory of equivariant cnns on homogeneous spaces

authors:
Michaël Defferrard
Nathanaël Perraudin
Tomasz Kacprzak
Raphael Sgier

title: Deepsphere: towards an equivariant graph-based spherical cnn

venue: In ICLR Workshop on Representation Learning on Graphs and Manifolds,

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks

venue: In Proceedings of the 34th International Conference on Machine Learning-Volume

authors:
Marc Finzi
Samuel Stanton
Pavel Izmailov
Andrew Gordon Wilson

title: Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data

venue: arXiv preprint arXiv:2002.12880,

authors:
Alexandre Galashov
Jonathan Schwarz
Hyunjik Kim
Marta Garnelo
David Saxton
Pushmeet Kohli
SM Eslami
Yee Whye Teh

title: Meta-learning surrogate models for sequential decision making

authors:
Marta Garnelo
Dan Rosenbaum
Chris J Maddison
Tiago Ramalho
David Saxton
Murray Shanahan
Yee Whye Teh
Danilo J Rezende
SM Eslami

title: Conditional neural processes

venue: arXiv preprint arXiv:1807.01613,

authors:
Jonathan Gordon
Wessel P Bruinsma
Andrew YK Foong
James Requeima
Yann Dubois
Richard E Turner

title: Convolutional conditional neural processes

authors:
Samuel Greydanus
Misko Dzamba
Jason Yosinski

title: Hamiltonian neural networks

authors:
Zhiwu Huang
Chengde Wan
Thomas Probst
Luc Van Gool

title: Deep learning on lie groups for skeleton-based action recognition

authors:
Lukasz Kaiser
Aidan N Gomez
Francois Chollet

title: Depthwise separable convolutions for neural machine translation

venue: arXiv preprint arXiv:1706.03059,

authors:
Hyunjik Kim
Andriy Mnih
Jonathan Schwarz
Marta Garnelo
Ali Eslami
Dan Rosenbaum
Oriol Vinyals
Yee Whye Teh

title: Attentive neural processes

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Risi Kondor
Shubhendu Trivedi

title: On the generalization of equivariance and convolution in neural networks to the action of compact groups

venue: arXiv preprint arXiv:1802.03690,

authors:
Christiane Lemke
Marcin Budka
Bogdan Gabrys

title: Metalearning: a survey of trends and technologies

venue: Artificial intelligence review,

authors:
Christos Louizos
Xiahan Shi
Klamer Schutte
Max Welling

title: The functional neural process

authors:
Robin Quessard
Thomas D Barrett
William R Clements

title: Learning group structure and disentangled representations of dynamical environments

venue: arXiv preprint arXiv:2002.06991,

authors:
Sachin Ravi
Hugo Larochelle

title: Optimization as a model for few-shot learning

venue: In Proceedings of the 5th International Conference on Learning Representation,

authors:
Andrei A Rusu
Dushyant Rao
Jakub Sygnowski
Oriol Vinyals
Razvan Pascanu
Simon Osindero
Raia Hadsell

title: Meta-learning with latent embedding optimization

venue: arXiv preprint arXiv:1807.05960,

authors:
Alvaro Sanchez-Gonzalez
Victor Bapst
Kyle Cranmer
Peter Battaglia

title: Hamiltonian graph networks with ode integrators

venue: arXiv preprint arXiv:1909.12790,

authors:
Adam Santoro
Sergey Bartunov
Matthew Botvinick
Daan Wierstra
Timothy Lillicrap

title: Metalearning with memory-augmented neural networks

authors:
Jake Snell
Kevin Swersky
Richard Zemel

title: Prototypical networks for few-shot learning

authors:
Koh Takeuchi
Hisashi Kashima
Naonori Ueda

title: Angle-based convolution networks for extracting local spatial features

venue: In Neurips Workshop on Modeling and decision-making in the spatiotemporal domain,

authors:
Peter Toth
Danilo Jimenez Rezende
Andrew Jaegle
Sébastien Racanière
Aleksandar Botev
Irina Higgins

title: Hamiltonian generative networks

venue: arXiv preprint arXiv:1909.13789,

authors:
Maurice Weiler
Gabriele Cesa

title: General e (2)-equivariant steerable cnns

authors:
Wenxuan Wu
Zhongang Qi
Li Fuxin

title: Pointconv: Deep convolutional networks on 3d point clouds

authors:
Jin Xu
Jean-Francois Ton
Hyunjik Kim
Adam R Kosiorek
Yee Whye Teh

title: Metafun: Metalearning with iterative functional updates

authors:
Yaofeng Desmond Zhong
Biswadip Dey
Amit Chakraborty

title: Symplectic ode-net: Learning hamiltonian dynamics with control

venue: arXiv preprint arXiv:1909.12077,

title: EQUIVCNP ARCHITECTURE The architecture of EquivCNP is following that of ConvCNP (Gordon et al., 2019), so that we can fairly compare them. It is difficult to determine radius r of LieConv because the radius is varied substantially between the different groups due to the different distance functions

heading: 1 INTRODUCTION

text:
Data symmetry has played a significant role in the deep neural networks. In particular, a convolutional neural network, which play an important part in the recent achievements of deep neural networks, has translation equivariance that preserves the symmetry of the translation group. From the same point of view, many studies have aimed to incorporate various group symmetries into neural networks, especially convolutional operation (Cohen et al., 2019; Defferrard et al., 2019; Finzi et al., 2020). As example applications, to solve the dynamics modeling problems, some works have introduced Hamiltonian dynamics (Greydanus et al., 2019; Toth et al., 2019; Zhong et al., 2019). Similarly, Quessard et al. (2020) estimated the action of the group by assuming the symmetry in the latent space inferred by the neural network. Incorporating the data structure (symmetries) into the models as inductive bias, can reduce the model complexity and improve model generalization.
In terms of inductive bias, meta-learning, or learning to learn, provides a way to select an inductive bias from data. Meta-learning use past experiences to adapt quickly to a new task T ∼ p(T ) sampled from some task distribution p(T ). Especially in supervised meta-learning, a task is described as predicting a set of unlabeled data (target points) given a set of labeled data (context points). Various works have proposed the use of supervised meta-learning from different perspectives (Andrychowicz et al., 2016; Ravi & Larochelle, 2016; Finn et al., 2017; Snell et al., 2017; Santoro et al., 2016; Rusu et al., 2018). In this study, we are interested in neural processes (NPs) (Garnelo et al., 2018a;b), which are meta-learning models that have encoder-decoder architecture (Xu et al., 2019). The encoder is a permutation-invariant function on the context points that maps the contexts into a latent representation. The decoder is a function that produces the conditional predictive distribution of targets given the latent representation. The objective of NPs is to learn the encoder and the decoder, so that the predictive model generalizes well to new tasks by observing some points of the tasks. To achieve the
objective, an NP is required to learn the shared information between the training tasks T , T ′ ∼ p(T ): the data knowledge Lemke et al. (2015). Each task T is represented by one dataset, and multiple datasets are provided for training NPs to tackle a meta-task. For example, we consider a meta-task that completing the pixels that are missing in a given image. Often, images are taken by the same condition in each dataset, respectively. While the datasets contain identical subjects of images (e.g., cars or apples), the size and angle of the subjects in the image may be different; the datasets have group symmetry, such as scaling and rotation. Therefore, it is expected that pre-constraining NPs to have group equivariance improves the performance of the NPs at those datasets.
In this paper, we investigate the group equivalence of NPs. Specifically, we try to answer the following two questions, (1) can NPs represent equivariant functions? (2) can we explicitly induce the group equivariance into NPs? In order to answer the questions, we introduce a new family of NPs, EquivCNP, and show that EquivCNP is a permutation-invariant and group-equivariant function theoretically and empirically. Most relevant to EquivCNP, ConvCNP (Gordon et al., 2019) shows that using general convolution operation leads to the translation equivariance theoretically and experimentally; however it does not consider incorporation of other groups. First, we introduce the decomposition theorem for permutation-invariant and group-equivariant maps. The theorem suggests that the encoder maps the context points into a latent variable, which is a functional representation, in order to preserve the data symmetry. Thereafter, we construct EquivCNP by following the theorem. In this study, we adopt LieConv (Finzi et al., 2020) to construct EquivCNP for practical implementation. We tackle a 1D synthetic regression task (Garnelo et al., 2018a;b; Kim et al., 2019; Gordon et al., 2019) to show that EquivCNP with translation equivariance is comparable to conventional NPs. Furthermore, we design a 2D image completion task to investigate the potential of EquivCNP with several group equivariances. As a result, we demonstrate that EquivCNP enables zero-shot generalization by incorporating not translation, but scaling equivariance.

heading: 2 RELATED WORK

heading: 2.1 NEURAL NETWORKS WITH GROUP EQUIVARIANCE

text:
Our works build upon the recent advances in group equivariant convolutional operation incorporated into deep neural networks. The first approach is group convolution introduced in (Cohen & Welling, 2016), where standard convolutional kernels are used and their transformation or the output transformation is performed with respect to the group. This group convolution induces exact equivariance, but only to the action of discrete groups. In contrast, for exact equivariance to continuous groups, some works employ harmonic analysis so as to find the basis of equivariant functions, and then parameterize convolutional kernels in the basis (Weiler & Cesa, 2019). Although this approach can be applied to any type of general data (Anderson et al., 2019; Weiler & Cesa, 2019), it is limited to local application to compact, unimodular groups. To address these issues, LieConv (Finzi et al., 2020) and other works (Huang et al., 2017; Bekkers, 2019) use Lie groups. Our EquivCNP chooses LieConv to manage group equivariance for simplicity of the implementation.
There are several works that study deep neural networks using data symmetry. In some works, in order to solve machine learning problems such as sequence prediction or reinforcement learning, neural networks attempt to learn a data symmetry of physical systems from noisy observations directly (Greydanus et al., 2019; Toth et al., 2019; Zhong et al., 2019; Sanchez-Gonzalez et al., 2019). While both these studies and EquivCNP can handle data symmetries, EquivCNP is not limited to specific domains such as physics.
Furthermore, Quessard et al. (2020) let the latent space into which neural networks map data, have group equivariance, and estimated the parameters of data symmetries. In terms of using group equivariance in the latent space, EquivCNP is similar to this study but differs from being able to use various group equivariance.

heading: 2.2 FAMILY OF NEURAL PROCESSES

text:
NPs (Garnelo et al., 2018a;b) are deep generative models for regression functions that map an input xi ∈ Rdx into an output yi ∈ Rdy . In particular, given an arbitrary number of observed data points (xC , yC) := {(xi, yi)}Ci=1, NPs model the conditional distribution of the target value yT at some new,
unobserved target data point xT , where (xT , yT ) := {(xj , yj)}Tj=1. Fundamentally, there are two NP variants: deterministic and probabilistic. Deterministic NPs (Garnelo et al., 2018a), known as conditional NPs (CNPs), model the conditional distribution as:
p(yT |xT , xC , yC) := p(yT |xT , rC), where r represents a function that maps data sets (xC , yC) into a finite-dimensional vector space in a permutation-invariant way and rC := r(xC , yC) ∈ Rd is the feature vector. The function r can be implemented by DeepSets (Zaheer et al., 2017). The likelihood p(yT |xT , rC) is modeled by Gaussian distribution factorized across the targets (xj , yj) with mean and variance of prediction {(xj , yj)}Tj=1 by passing inputs rC and xj through the MLP. The CNP is trained by maximizing the likelihood.
Probabilistic NPs include a latent variable z. The NP infers q(z|rC) given an input rC using the reparametrization trick (Kingma & Welling, 2013) and models such a conditional distribution as:
p(yT |xT , xC , yC) := ∫ p(yT |xT , rC , z)q(z|rC)dz
and it is trained by maximizing an ELBO: L(φ, θ) = Ez∼qφ(z|xT ,yT )[log pθ(yT |xT )] − KL[qφ(z|xT , yT )‖pθ(z|xC , yC)]. NPs have various useful properties: i) Scalability: the computational cost of NPs scales as O(n+m) with respect to n contexts and m targets of data, ii) Flexibility: NPs can define a conditional distribution of an arbitrary number of target points, conditioning an arbitrary number of observations, iii) Permutation invariance: the encoder of NPs uses Deepsets (Zaheer et al., 2017) to make the target prediction permutation invariant. Thanks to these properties, Galashov et al. (2019) replace Gaussian processes in Bayesian optimization, contextual multi-armed bandit, and Sim2Real tasks.
While there are many NP variants (Kim et al., 2019; Louizos et al., 2019; Xu et al., 2019) to improve the performance of NPs, those do not take group equivariance into account yet. The most similar to EquivCNP, ConvCNP (Gordon et al., 2019) incorporated only translation equivariance. In contrast, EquivCNP can incorporate not only translation but also other groups such as rotation and scaling.

heading: 3 DECOMPOSITION THEOREM

text:
In this section, we consider group convolution. We first prepare some definition and teminology. Let X and Y ⊂ R be the input space and output space, respectively. We define ZM = (X × Y)M as a collection of M input-output pairs, Z≤M = ⋃M n=1Zn as the collection of at most M pairs, and
Z = ⋃∞ m=1Zm as the collection of finitely many pairs. Let [n] = {1, . . . , n} for n ∈ N, and let Sn be the permutation group on [n]. The action of Sn on Zn is defined as πZn := ((xπ−1(1),yπ−1(1)), . . . , (xπ−1(n),yπ−1(n))),
where π ∈ Sn and Zn ∈ Zn. We define the multiplicity of Zn = ((x1,y1), . . . , (xn,yn)) ∈ Zn by mult(Zn) := sup {|{i ∈ [n] : xi = x̂}| : x̂ = x1, . . . ,xn} and the multiplicity of Z ′ ⊆ Z by mult(Z ′) := supZn∈Z′ mult(Zn). Then, a collection Z ′ ⊆ Z is said to have multiplicity K if mult(Z ′) = K. Mathematically, symmetry is described in terms of group action. The following group equivariant maps represent to preserve the symmetry in data. Definition 1 (Group Equivariance and Invariance). Suppose that a group G acts on sets S and S ′. Then, a map Φ : S → S ′ is called G-equivariant when Φ(g · s) = g ·Φ(s) holds for arbitrary g ∈ G and s ∈ S. In particular, when G acts on S ′ trivially (i.e., g · s′ = s′ for g ∈ G and s′ ∈ S ′), the G-equivariant map is said to be G-invariant: Φ(g · s) = Φ(s).
Then, we can derive the following theorem, which decompose a permutation-invariant and group equivariant function into two tractable functions. Note that this theorem has been proved by Gordon et al. (2019) when G is a translation group. Theorem 2 (Decomposition Theorem). Let G be a group. Let Z ′≤M ⊆ (X ×Y)≤M be topologically closed, permutation-invariant and G-invariant with multiplicity K. For a function Φ : Z ′≤M → Cb(X ,Y), the following conditions are equivalent:
Algorithm 1 Prediction of Group Equivariant Conditional Neural Process Input: ρ =LieConv, RBF kernel ψ, context {xi, yi}Ni=1, target {x∗j}Mj=1
lower, upper← range((xi)Ni=1 ∪ (xj)Mj=1) (tk) T k=1 ← uniform_grid(lower, upper; γ) // Encoding the context information into representation h(i.e. Encoder) h← ∑N i=1 φK+1(yi)ψ([x ∗ j , tk],xi) h. (µj ,Σj) > = LieConvNet(h)(x∗j ) // Decoder
Output: {(µj ,Σj)}Mj=1
(I) Φ is continuous, permutation-invariant and G-equivariant.
(II) There exist a function spaceH and a continuous G-equivariant function ρ : H → Cb(X ,Y) and a continuous G-invariant interpolating kernel ψ : X 2 → R such that
Φ(Z) = ρ ( m∑ i=1 φK+1 (yi)ψxi ) where φK+1 : Y → RK+1 is defined by φK+1(y) := [1, y, y2, . . . , yK ]>.
Thanks to the Theorem 2, we can construct the permutation-invariant and group-equivariant NPs whose form of encoder and decoder is determined. In this paper, we call Φ as EquivDeepSet.

heading: 4 GROUP EQUIVARIANT CONDITIONAL NEURAL PROCESSES

text:
In this section, we represent EquivCNP that is a permutation-invariant and group-equivariant map. EquivCNP models the same conditional distribution as well as CNPs:
p(YT |XT ,DC) = N∏ n=1 p (yn|Φθ(DC) (xn))
= N∏ n=1 N (yn;µn,Σn) with (µn,Σn) = Φθ(DC) (xn)
where N denotes the density function of a normal distribution, DC = (XC ,YC) = {(xc, yc)}Ci=1 is the observed context data and φ is a EquivDeepSet. The important components of EquivCNP to be determined are ρ, φ, and ψ. The algorithm is represented in Algorithm 1.
To describe in more detail, first, Section 4.1 introduce the definition of group convolution, and then Section 4.2 explains LieConv (Finzi et al., 2020) used for EquivCNP to implement group convolution. Finally, we describe the architecture of proposed EquivCNP in Section 4.3.

heading: 4.1 GROUP CONVOLUTION

text:
When X is a homogenous space of a group G, the lift of x ∈ X is the element of group G that transfers a fixed origin o to x：Lift(x) = {u ∈ G : uo = x}. That is, each pair of coordinates and
features is lifted into K elements1：{(xi, fi)}Ni=1 → {(uik, fi)} N,K i=1,k=1. When the group action is transitive, the space on which it acts on is a homogenous space. More generally, however, the action is not transitive, and the total space contains an infinite number of orbits. Consider a quotient space Q = X/G, which consists of orbits of G in X . Then each element q ∈ Q is a homogenous space of G. Because many equivariant maps use this information, the total space should be G×X/G, not G. Hence, x ∈ X is lifted to the pair (u, q), where u ∈ G and q ∈ Q. Group convolution is a generalization of convolution by translation, which is used in images, etc., to other groups. Definition 3 (Group Convolution (Kondor & Trivedi, 2018; Cohen et al., 2019)). Let g, f : G×Q→ R be functions, and let µ(·) be a Haar measure on G． For any u ∈ G, the convolution of f by g is defined as
h(u, q) = ∫ G×Q g(v−1u, q, q′)f(v, q′)dµ(v)dq′.
By the definition, we can verify that the group convolution is G-equivariant. Moreover, Cohen et al. (2019) recently showed that a G-equivariant linear map is represented by group convolution when the action of a group is transitive.

heading: 4.2 LOCAL GROUP CONVOLUTION

text:
In this study, we used LieConv as a group convolution (Finzi et al., 2020)． LieConv is a convolution that can handle Lie groups in group convolutions. LieConv acts on a pair (xi, fi) N i=1 of coordinates xi ∈ X and values fi ∈ V in vector space V . First, input data xi is transformed (lifted) into group elements ui and orbits qi. Next, we define the convolution range based on the invariant (pseudo) distance in the group, and convolve it using a kernel parameterized by a neural network.
What is important for inductive bias and computational efficiency in convolution is that the range of convolutions is local; that is, if the distance between ui and uj is larger than r, gθ(ui, uj) = 0. First, we define distance in the Lie group to deal with locality in the matrix group2:
d(u, v) := ‖ log(u−1v)‖F , where log denotes the matrix logarithm, and F denotes the Frobenius norm. Because d(wu,wv) = ‖ log(u−1w−1wv)‖F = d(u, v) holds, this function is left-invariant and is a pseudo-distance.3
To further account for orbit q, we extend the distance to d((ui, qi), (vj , qj))2 = d(ui, vj)2 + αdO(qi, qj)
2, where dO(qi, qj) := infxi∈qi,xj∈qj dX (xi, xj) and dX is the distance on X . It is not necessarily invariant to the transformation in q.
Based on this distance, the neighborhood is nbhd(u) = {v, q|d((ui, qi), (vi, qj)) < r}. The radius r should be adjusted appropriately from the ratio of the range of convolutions to the total input, because the appropriate value is difficult to determine depending on the group treated. Therefore, the Lie group convolution is
h(u, q) = ∫ v,q′∈nbhd(u) gθ(v −1u, q, q′)f(v, q′)dµ(v)dq′.
Radius r of the neighborhood corresponds to the inverse of the density channel h(0) in Gordon et al. (2019).
Discrete Approximation. Given a lifted input data point {(vj , qj)Nj=1} and a function value fj = f(vj , qj) at each point, we need to select a target {(ui, qi)Ni=1} to convolve so that we can approximate the integral of the equation. Because the convolutional range is limited by nbhd(u), LieConv can approximate the integrals by the Monte Carlo method：
h(u, q) = (g∗̂f)(u, q) = 1 n ∑ vj ,q′j∈nbhd(u,q) g(v−1j u, q, q ′ j)f(vj , q ′ j)
1K is a hyperparameter and we randomly pick K elements {uik}Kk=1 in the orbit corresponding to xi. 2We assume that we have a finite-dimensional representation. 3This is because the triangle inequality is not satisfied.
The classical convolutional filter kernel g(·) is only valid for discrete values and is not available for continuous group elements. Therefore, pointconv/Lieconv uses a multilayered neural network gθ as a convolutional kernel. However, because neural networks are good at computation in Euclidean space, and inputG is not a vector space, we let gθ be a map in the Lie algebra g. Therefore, we use Lie groups and logarithmic maps exist in each element of the group. That is, let gθ(u) = (g ◦ exp)θ(log u), and parameterize g̃θ = (g ◦ exp)θ by MLP. We use g̃θ : g → Rcout×cin . Therefore, the convolution of the equation is
hi = 1
ni ∑ j∈ nbhd (i) g̃θ ( log ( v−1j ui ) , qi, qj ) fj .
Here, the input to the MLP is aij = Concat ( [log(v−1j ui), qi, qj)] ) .

heading: 4.3 IMPLEMENTATION

text:
First, we explain the form of φ. Because most real-world data have a single output per one input location, we treat the multiplicity of DC as one, K = 1, and define φ(y) = [1 y]> based on (Zaheer et al., 2017). The first dimension of output φi indicates whether the data located at xi is observed, so that the model can distinguish between the observed data, and the unobserved data whose value is zero (yi = 0).
Then, we describe the form of ψ. Following our Theorem 2, ψ is required to be stationary, nonnegative, and a positive–definite kernel. For EquivCNP, we change ψ depending on whether the input data is continuous or discrete. With continuous input data (e.g. 1D regression), we use RBF kernels for ψ. An RBF kernel has a learnable bandwidth parameter and scale parameter and is optimized with EquivCNP. A functional representation E(Z) is made up by multiplying the kernel ψ with φ. On the other hand, when the inputs are discrete (e.g. images), we use not an RBF kernel but LieConv.
Finally, we explain the form of ρ. With our Theorem2, because ρ needs to be a continuous group equivariant map between function spaces, we use LieConv for ρ. In this study, under the hypothesis of separability (Kaiser et al., 2017), we implemented separable LieConv in the spatial and channel directions, to improve the efficiency of computational processing. The details are given in the Appendix B. EquivCNP requires to compute the convolution of E(Z). However, since E(Z) itself is a functional representation, it cannot be computed in computers as it is. To address this issue, we discretize E(Z) over the range of context and target points. We space the lattice points (ti)ni=1 ⊆ X on a uniform grid over a hypercube covering both the context and target points. Because the conventional convolution that is used in ConvCNP requires discrete lattice input space to operate on and produces discrete outputs, we need to back the outputs to continuous functions X → Y . While ConvCNP regards the outputs as weights for evenly-spaced basis functions (i.e., RBF kernel), LieConv does not require the input location to be lattice and can produce continuous functions output directly. Note that the algorithm of EquivCNP can be the same as ConvCNP; it can also use evenly-spaced basis functions. The obtained functions are used to output the Gaussian predictive mean and the variance at the given target points. We can evaluate EquivCNP by log-likelihood using the mean and variance.

heading: 5 EXPERIMENT

text: To investigate the potential of EquivCNP, we constructed three questions: 1) Is EquivCNP comparable to conventional NPs such as ConvCNP? and 2) Can EquivCNP have group equivariance in addition to translation equivariance and 3) does it preserve the symmetries? To compare fairly with ConvCNP, the architecture of EquivCNP follows that of ConvCNP; details are given in the Appendix C.

heading: 5.1 1D SYNTHETIC REGRESSION TASK

text:
To answer the first question, we tackle the 1D synthetic regression task as has been done in other papers (Garnelo et al., 2018a;b; Kim et al., 2019). At each iteration, a function f is sampled from a given function distribution, then, some of the context DC and target DT points are sampled from function f . In this experiment, we selected the Gaussian process with RBF kernel, Matern– 52 and periodic kernel for the function distribution. We chose translation equivariance T (1) to incorporate
into EquivCNP. We compared EquivCNP with GP (as an oracle), with CNP (Garnelo et al., 2018a) as a baseline, and with ConvCNP.
Table 1 shows the log–likelihood means and standard deviations of 1000 tasks. In this task, both contexts and targets are sampled from the range [−2, 2]. From Table 1, we can see that EquivCNP with translation equivariance is comparable to ConvCNP throughout all GP curve datasets. That is, EquivCNP has the model capacity to learn the functions as well as ConvCNP.
We also conducted the extrapolation regression proposed in (Gordon et al., 2019) as shown in Figure 2. The first two columns show the models trained on an RBF kernel and the last two columns on a Matern– 52 kernel. The top row shows the predictive distribution when the observation is given within the same training region; the bottom row for the observation is not only the training region but also the extrapolation region: [−4, 4]. As a result, EquiveCNP can generalize to the observed data whose range is not included during training. This result was expected because Gordon et al. (2019) has mentioned that translation equivariance enables the models to adapt to this setting.

heading: 5.2 2D IMAGE-COMPLETION TASK

text:
An image-completion task aims to investigate that EquivCNP can complete the images when it is given an appropriate group equivariance. The image-completion task can be regarded as a regression task that predicts the value of y∗i at the 2D image coordinates x ∗ i , given the observed pixels DC = {(xn, yn)}Nn=1 (∈ R3 for the colored image input, and ∈ R for the grayscale image input). The framework of the image completion can apply not only to the images but also to other real-world applications, such as predicting spatial data (Takeuchi et al., 2018).
To evaluate the effect of EquivCNP with a specific group equivariance, we introduce a new dataset digital clock digits as shown in Figure 3. Since previous works use the MNIST dataset for image completion, we also conduct the image completion task with rotated-MNIST. However, we cannot find a significant difference between the group equivariance models (the result of rotated-MNIST is depicted in Appendix E). We think that this happens because (1) original MNIST contains various
Table 2: Log-likelihood of 2D image-completion task
Group Log–likelihood
T (2) 1.0998± 0.4115 SO(2) −2.4275± 6.8856 R>0 × SO(2) 1.8398± 0.5368 SE(2) 1.1655± 0.5420 Figure 3: The example of training data(top) and test data (bottom).
Context
SE(2)
×1.0 ×0.75 ×0.5 25% 25% 25%75% 75%25%75% ×0.25
T(2) (ConvCNP)
R×SO(2)
SO(2)
Figure 4: Image-completion task results. The top row shows the given observation and the other rows show the mean of the conditional distribution predicted by EquivCNP with the specific group equivariance: T (2), SO(2), R>0 × SO(2), and SE(2). Two of each column shows the same image, and the difference between two columns is the percentage of context random sampling: 25% and 75%. When the size of digits is the same as that of the training set (i.e. not scaling but rotation equals SO(2) symmetry), T (2) and SE(2) have a good quality, but when the size of digits is smaller than that of training set, R>0 × SO(2) has a good performance.
data symmeries including translation, scaling, and rotation, and (2) we cannot specify them precisely. Thus, we provide digital clock digits dataset anew.
In this experiment, we used four kinds of group equivariance; translation group T (2), the 2D rotation group SO(2), the translation and rotation group SE(2), and the rotation-scale group R>0 × SO(2). The size of the images is 64 × 64 pixels, and the numbers are in the center with the same vertical length. For the test data, we transform the images by scaling within [0.15, 0.5] and rotating within [−90◦,+90◦]. Image completion with our digits data becomes an extrapolation task in that the test data is never seen during training, though the number shapes are the same in both sets.
The log–likelihood of image completion by EquivCNP with the group equivariance is reported in Table 2. The mean and standard deviation of the log–likelihood is calculated over 1000 tasks (i.e. evaluating the digit transformed in 100 times respectively). As a result, EquivCNP withR>0×SO(2) performed better than other group equivarinace. On the other hand, the model with SO(2) had the worst performance. This might happen because the SO(2) is not able to generalize EquivCNP to scaling. In fact, the log–likelihood of SE(2), which is the group equivariance combining translation T (2) and rotation SO(2), is not improved than that of T (2).
Figure 4 shows the qualitative result of image completion by EquivCNP with each group equivariance. We demonstrate that EquivCNP was able to predict digits smaller than the training digits4. While T (2) completes the images most clearly when the sizes of digits and the number of observations
4When the scaling is ×1.0, it equals to SO(2) symmetry.
are large, other groups also complete the images. The smaller the size of digits is compared to the training digits, the worse the quality of T (2) completion becomes, and R>0 × SO(2) completes the digits more clearly. This is because the convolution region of T (2) is invariant to the location, while that of R>0 × SO(2) is adaptive to the location. As a result, for the images transformed by scaling, we can see that EquivCNP with R>0 × SO(2) preserved scaling group equivariance.

heading: 6 DISCUSSION

text: We presented a new neural process, EquivCNP, that uses the group equivariant adopted from LieConv. Given a specific group equivariance, such as translation and rotation as inductive bias, EquivCNP has a good performance at regression tasks. This is because the kernel size changes depending on the specific equivariance. Real–world applications, such as robot learning tasks (e.g. using hand-eye camera) will be left as future work. We also hope EquivCNPs will help in learning group equivariance (Quessard et al., 2020) by data–driven approaches for future research.

heading: A. PROOF OF THEOREM 2

text:
First, we prove that (II) implies (I). We define the action ofG on the set of univariate maps f : X → R by
(g · f)(x) := f(g−1 · x).
and define the action of G on the set of bivariate maps ψ : X 2 → R by (g · ψ)(x,x′) := ψ(g−1 · x, g−1 · x′).
Lemma 4. For a map ψ : X 2 → Rd and a sample x ∈ X , a sample dependent function ψx : X → Rd is defined by
ψx(x ′) := ψ(x′,x)
Here, ψ is G-invariant if and only if the map X 3 x 7→ ψx ∈ Map(X ,Rd) is G-equivariant.
The above lemma is derived as follows:
ψg·x′(x) = ψ(x, g · x′) = ψ(g−1 · x,x′) = ψx′(g−1 · x) = (g · ψx′)(x). The left hand side represents the action on the sample space and the right hand side does the action on the function space.
When we denote the set of all G-equivariant maps from S to S ′ by Equiv(S,S ′), the above lemma is represented as
Inv(X 2,Rd) ∼= Equiv(X ,Map(X ,Rd)).
Thus, since ψ : X 2 → R is invariant from (II), x 7→ ψx is equivariant. Then, for Z ∈ Z ′≤M , the correspondence Z 7→ ∑m i=1 φK+1(yi)ψxi is also equivariant. Since ρ is equivariant from (II), we obtain (I) because the composition ofequivariant maps is equivariant.
Next, we prove that (I) implies (II). We prepare some notations and lemmas in the following. Let ψ be an interpolating continuous kernel that satisfies ψ (x,x′) ≥ 0. Then, for m ∈ N and Z ′m ⊆ (X × Y)m, define
Hm(Z ′m) := { m∑ i=1 φK+1 (yi)ψ (·,xi) : (xi, yi)mi=1 ⊆ Z ′ m } ⊆ HK+1,
where HK+1 = H × · · · × H is the (K + 1)-dimensional-vector-valued-function Hilbert space constructed from the RKHS H for which ψ is a reproducing kernel and endowed with the inner product 〈f, g〉HK+1 = ∑K+1 i=1 〈fi, gi〉H, where 〈·, ·〉H is the inner product of the RKHS H. When the permutation group Sm acts on a set (X × Y)m, the set of equivalence classes of this action is denoted by (X × Y)m/Sm. Then, for an element Z ∈ (X × Y)m, the equivalent class of the action is denoted by [Z]. Similarly, for a subset Z ′m ⊂ (X × Y)m, the set of equivalent classes is denoted by [Z ′m] := {[Z]|Z ∈ Z ′m}. Furthermore, we denote as[
Z ′≤M ] := M⋃ m=1 [Z ′m] and H≤M := M⋃ m=1 Hm(Z ′m).
Lemma 1 and Lemma 3 in Gordon et al. (2019) provides the following lemma. Lemma 5. For m ∈ N, let Z ′m ⊆ (X × Y)m be a set with multiplicity K and ψ be an interpolating continuous kernel. Then, (Hm(Z ′m)) M m=1 are pairwise disjoint and the embedding E is injective and continuous:
E : [ Z ′≤M ] → H≤M (Z ′m), E([Z]) := Em([Z]) if [Z] ∈ [Z ′m] ,
where Em : [Z ′m]→ Hm(Z ′m), Em ([(x1, y1) , . . . , (xm, ym)]) := m∑ i=1 φK+1 (yi)ψ (·,xi)
Similarly, Lemma 2 and Lemma 4 in Gordon et al. (2019) provides the following lemma.
Lemma 6. Suppose that Z ′M is a topologically closed set in (X × Y)M and permutation-invariant, and that ψ satisfies (i) ψ ≥ 0, (ii) ψ(x, x) = σ2 > 0 for any x, and (iii) ψ(x, x′)→ 0 as ‖x‖ → ∞. Let Φ : [ Z ′<M ] → Cb(X ,Y) be a map such that every restriction Φ|[z′m] is continuous. Then, Φ ◦ E−1 : H<M → Cb(X ,Y) is continuous.
When a G-equivariant function f is injective, f−1|Imf is also G-equivariant on the image of f . Denoting Φ ◦ E−1 by ρ, we can rewrite as Φ = ρ ◦ E.

heading: B. SEPARABLE LIECONV

text:
In this section, we introduce the separable LieConv that we design and implement for EquivCNP. LieConv (Finzi et al., 2020) is based on PointConv (Wu et al., 2019), which is proposed for point cloud convolution. That is, the lifted inputs are convolved by PointConv. Therefore, we can adopt techniques that are used for general convolution. One of such techniques is separable convolution(Chollet, 2017). Separable convolution consists of depthwise convolution and pointwise convolution (as known as 1 x 1 convolution). The mathematical formulation of normal convolution, the pointwise convolution, and the depthwise convolution is as follow:
Conv(W, y)(i, j) = K,L,M∑ k,l,m W(k,l,m) · y(i+k,j+l,m)
PointwiseConv(W, y)(i, j) = M∑ m Wm · y(i,j,m)
DepthwiseConv(W, y)(i, j) = K,L∑ k=1 W(k,l) y(i+k,j+l)
SepConv (Wp,Wd, y) (i, j) = PointwiseConv(i, j) (Wp,Depthwise Conv (i, j) (Wd, y))
Thanks to the assumption that convolution operation is separable to the spatial direction and the channel direction, the separable convolution provides the way to operate convolution more efficiently than general convolution. Note that the difference of the efficiency between LieConv and separable LieConv is slight; the difference between the matrix production and element-wise product. Following the equation above, we design and implemented separable LieConv. Figure 5 illustrates the processing of (a) normal LieConv and (b) separable LieConv. The memory consumption is also different that the output shape of after the convolutional weights (kernel) is calculated in normal LieConv is B ×NMC × Cmid while that of separable LieConv is B ×NMC × Cin.

heading: C. EQUIVCNP ARCHITECTURE

text:
The architecture of EquivCNP is following that of ConvCNP (Gordon et al., 2019), so that we can fairly compare them. It is difficult to determine radius r of LieConv because the radius is varied substantially between the different groups due to the different distance functions. Instead, we parametrized the radius by specifying the average fraction of the total number of convolved elements that would fall into this radius. Therefore, we describe the value of the average fraction instead of kernel size that is described in other papers as usual. Simultaneously, while the conventional convolutional layer has a parameter called stride that determines the target elements (pixels) to be convolved, LieConv has a parameter sampling fraction instead of stride to subsample the group elements; sampling fraction is 1.0.
C.1 1D SYNTHETIC REGRESSION TASK
For 1D regression tasks, we use 4-layer LieConv architecture with ReLU activations. The average fraction of those LieConv is 532 and the number of MC sampling is 25. The channels of LieConv are [16, 32, 16, 8]. Functional representation E(Z) is concatenated with target point xT , followed by
lifting. After operating convolution to the lifted inputs, we use a softplus activation following the last fully-connected layer (FC) as a standard deviation. Note that the output of EquivCNP, mean and standard deviation, is sliced to get those of yT . The architecture of EquivCNP for a 1D regression task is illustrated in Figure 6.
C.2 2D IMAGE-COMPLETION TASK
For the 2D image-completion task, we use LieConv Convθ instead of RBF kernels as ψ. The channels of this LieConv is 128, the average fraction is 110 , and the number of MC sampling is 121. After the LieConv of ψ, we use four residual blocks. Each block is composed by two separable LieConv layers
and residual connections as shown in Figure 7. The channel of each residual block is 128, the average fraction is 115 , and the number of MC sampling is 81.
We employ the same procedure of ConvCNP (Gordon et al., 2019) for image-completion as follows:
1. Given an input image I ∈ RC×H×W , where C is color channel, H and W represents height and width respectively, sample context points features := I Mc from bernoulli distribution. Mc means the density as same as we define φ during 1D regression task.
2. After lifting the inputs, apply a LieConv to both I Mc and Mc to get functional representation: E(Z) = Convθ([Mc, I Mc]) ∈ R(128+128)×H×W .
3. Then, functional representation E(Z) is passed through one FC followed by four residual blocks: h = ResBlocks(FC(E(Z))) ∈ R128×H×W .
4. Finally, we use one FC to get mean and standard deviation channels and split the output ∈ R2C×H×W into those statistics.

heading: D. EXPERIMENT DETAILS

text:
In this section, we describe the experiments in more detail. Code and dataset are available on https://github.com/makora9143/EquivCNP.
D.1 1D SYNTHETIC REGRESSION TASK
The kernels used in Section 5.1 for generating the data via Gaussian Processes are defined as follows:
• RBFKernel:
k(x1, x2) = exp
( − (x1 − x2) 2
2 ) • Matern- 52
k(x1, x2) =
( 1 + √ 5d+ 5 3 d2 ) exp ( − √ 5 2 d ) with d = ‖x1 − x2‖2
• Periodic k(x1, x2) = exp (−2 sin(π‖x1 − x2‖2))
To train all NPs, the GPs generate the context and target points; the number of context points and target points is random-sampled uniformly from [3, 50] respectively. All NPs were trained for 200 epochs by 256 batches per epoch and the size of each batch is 16, We used Adam optimizer (Kingma & Ba, 2014) with learning rate 10−3. An architecture of CNP was based on the original code5. We visualize the result of periodic kernel regression at Figure 8.
We also demonstrate EquivCNP with the algorithm following that of ConvCNP (Gordon et al., 2019); regarding the output of EquivCNP as weights for evenly-spaced basis functions (i.e. RBF kernel) in Figure 9. The result of predictive distribution is much smoother than the result of our Algorithm 1 though using RBF kernel is redundant.
5https://github.com/deepmind/neural-processes
D.2 2D IMAGE-COMPLETION TASK
The original image of the digital clock number is shown in Figure 10. We first inverted in colors of black and white of the image. Then, we cropped the image so that each cropped image contains one digit and resize them to 64× 64. Note that the vertical size of each number is set up to 56, while the horizontal size is not fixed. The values of all pixels are devided by 255 to rescale them to the [0, 1] range.
As we mentioned in Section C.2, the context points are sampled from bernoulli distribution. The parameter of bernoulli distribution, probability p that the value is 1, is determined at a rate of the number uniformly from U(ntotal100 , ntotal 2 ) per ntotal. The batch size is 4, epoch is 100, and the optimizer is Adam (Kingma & Ba, 2014) whose learning rate is 5× 10−4.

heading: E. ADDITIONAL COMPLETION TASK: MNIST

text:
We also conduct the image completion task using rotated MNIST. It is thought that (1) original MNIST contains various data symmetries including translation, scaling, and rotation, and (2) we cannot specify them precisely. Figure 11 shows the actual images from the original MNIST datasets. We can confirm that yet we did not conduct any transformation, the images have been already rotated.
Moreover, factors other than symmetry such as personal habit exist. Indeed, the original MNIST is not good for verify the effectiveness of EquivCNP.
The result is depicted in Figure 12. During this experiment, the batch size is 16, epoch is 30, and the optimizer is Adam whose leraning rate is 5× 10−4. As a result, the model misses the completion when the number of context points is quite a few. On the other hand, when the number of context points is sufficient, the completion results seem well except the SO(2)-equivariant model.

title: GROUP EQUIVARIANT CONDITIONAL NEURAL PRO- CESSES