summaries: This paper proposes a simple and effective technique to improve disentanglement by coupling the latent spaces of different VAE models. It builds on Duan et al. (2019)’s proposed method to rank the representations of different models. By learning a VAE ensemble with linear transformations between the latent spaces and an additional “cross-model” reconstruction loss, the authors show that they can achieve significantly better disentangling.

summaries: The authors introduce a novel VAE-based approach for unsupervised learning of disentangled representations of image data.  The approach trains an ensemble of VAEs along with pair-wise linear transformations between their latent spaces.  The objective includes the ELBO objectives for each VAE as well as two additional pressures:  (i) An L2 similarity objective that pressures samples from each VAE latent space to match under linear transformations samples from the other VAE latent spaces, and (ii) A cross-model decoding objective that encourages decoding accuracy of the linearly transformed latent samples.  The authors provide a theoretical argument that the linear transformations should learn to be orthogonal, and show some experimental results indicating that their model performs well compared to baselines when evaluated with an established disentangling metric.

summaries: This submission proposes an ensemble framework to improve learning disentangled representations with Variational Autoencoders (VAEs). The approach builds on the assumption that entangled latent representations learned by VAEs show some “uniqueness” in their latent space structure, while disentangled representations exhibit some “similarity”; an assumption corroborated by recent studies. On that basis, a VAE ensemble approach is proposed where several VAEs are connected through linear mappings between the individual latent spaces to encourage alignment of latent representations and thus disentanglement. A formal derivation of the framework is provided and the formal validity of the underlying assumption demonstrated. Furthermore, empirical evaluation of the proposed approach in comparison to the standard VAE, beta-VAE and FactorVAE on the datasets dSprites (main results, main text) and CelebA (appendix) is performed, yielding improved results on the FactorVAE disentanglement metric (all baseline methods considered) as well as the Distance to Orthogonality (DtO) metric (only standard VAE considered).

abstractText: Variational Autoencoder (VAE) based frameworks have achieved the state-of-the1 art performance on the unsupervised disentangled representation learning. A re2 cent theoretical analysis shows that such success is mainly due to the VAE im3 plementation choices that encourage a PCA-like behavior locally on data sam4 ples. Despite this implied model identifiability, the VAE based disentanglement 5 frameworks still face the trade-off between the local orthogonality and data re6 construction. As a result, models with the same architecture and hyperparameter 7 setting can sometimes learn entangled representations. To address this challenge, 8 we propose a simple yet effective VAE ensemble framework consisting of multi9 ple VAEs. It is based on the assumption that entangled representations are unique 10 in their own ways, and the disentangled representations are “alike” (similar up to a 11 signed permutation transformation). In the proposed VAE ensemble, each model 12 not only maintains its original objective, but also encodes to and decodes from 13 other models through pair-wise linear transformations between the latent repre14 sentations. We show both theoretically and experimentally, the VAE ensemble 15 objective encourages the linear transformations connecting the VAEs to be triv16 ial transformations, aligning the latent representations of different models to be 17 “alike”. We compare our approach with the state-of-the-art unsupervised disen18 tangled representation learning approaches and show the improved performance. 19

title: Understanding disentangling in β -VAE

authors:
Emile Mathieu
Tom Rainforth
N Siddharth
Yee Whye Teh

title: Disentangling disentanglement

authors:
Loic Matthey
Irina Higgins
Demis Hassabis
Alexander Lerchner

authors:
Ari Morcos
Maithra Raghu
Samy Bengio

authors:
Maithra Raghu
Justin Gilmer
Jason Yosinski
Jascha Sohl-Dickstein

authors:
Danilo Jimenez Rezende
Shakir Mohamed
Daan Wierstra

venue: arXiv preprint arXiv:1401.4082,

authors:
Michal Rolinek
Dominik Zietlow
Georg Martius

title: The strength of weak learnability

venue: Machine learning,

authors:
Karen Simonyan
Andrew Zisserman

venue: arXiv preprint arXiv:1409.1556,

authors:
Sjoerd van Steenkiste
Francesco Locatello
Jürgen Schmidhuber
Olivier Bachem

authors:
Liwei Wang
Lunjia Hu
Jiayuan Gu
Zhiqiang Hu
Yue Wu
Kun He
John Hopcroft

authors:
Nicholas Watters
Loic Matthey
Matko Bosnjak
Christopher P Burgess
Alexander Lerchner

text: Variational Autoencoder (VAE) based frameworks have achieved the state-of-the- 1 art performance on the unsupervised disentangled representation learning. A re- 2 cent theoretical analysis shows that such success is mainly due to the VAE im- 3 plementation choices that encourage a PCA-like behavior locally on data sam- 4 ples. Despite this implied model identifiability, the VAE based disentanglement 5 frameworks still face the trade-off between the local orthogonality and data re- 6 construction. As a result, models with the same architecture and hyperparameter 7 setting can sometimes learn entangled representations. To address this challenge, 8 we propose a simple yet effective VAE ensemble framework consisting of multi- 9 ple VAEs. It is based on the assumption that entangled representations are unique 10 in their own ways, and the disentangled representations are “alike” (similar up to a 11 signed permutation transformation). In the proposed VAE ensemble, each model 12 not only maintains its original objective, but also encodes to and decodes from 13 other models through pair-wise linear transformations between the latent repre- 14 sentations. We show both theoretically and experimentally, the VAE ensemble 15 objective encourages the linear transformations connecting the VAEs to be triv- 16 ial transformations, aligning the latent representations of different models to be 17 “alike”. We compare our approach with the state-of-the-art unsupervised disen- 18 tangled representation learning approaches and show the improved performance. 19

heading: 1 INTRODUCTION 20

text:
Disentangled representation learning aims to capture the semantically meaningful compositional 21 representation of data (Higgins et al., 2018; Mathieu et al., 2018), and is shown to improve the 22 efficiency and generalization of supervised learning (Locatello et al., 2019), reinforcement learning 23 (Watters et al., 2019), and reasoning tasks (van Steenkiste et al., 2019). The current state-of-the- 24 art unsupervised disentangled representation learning deploy the Variational Autoencoder (VAE) 25 (Kingma & Welling, 2013; Rezende et al., 2014). The main challenge is to reduce the trade-off 26 between learning a disentangled representation and reconstructing input data. Most of the recent 27 works extend the original VAE objective with carefully designed augmented objective to address 28 this trade-off (Higgins et al., 2017; Burgess et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; 29 Kumar et al., 2017). A recent study in (Locatello et al., 2018) compared these methods and showed 30 that their performance is sensitive to initialization and hyperparameter setting of the augmented 31 objective function. 32
Recently, Duan et al. (Duan et al., 2019) developed an unsupervised model selection method named 33 Unsupervised Disentanglement Ranking (UDR) to address the challenge of hyperparameter search 34 and model selection. UDR leverages the finding in (Rolinek et al., 2019) that the implementation 35 choices of VAE encourage a local PCA-like behavior locally on data samples. As a result, disen- 36 tangled representations by VAEs are “alike” as they are similar up to signed permutation transfor- 37 mations. On the contrary, the entangled representations by VAEs are “unique” as they are similar 38 at least up to non-degenerate rotation matrices. UDR uses multiple models trained with different 39 initializations and hyperparameter settings, and builds a similarity matrix measuring the pair-wise 40 similarity between the latent variables from different models. A higher score is given to the model 41 that can match its representations to many others models. The results show close match between 42 UDR and commonly used supervised metrics, as well as the performance of downstream tasks using 43 the latent representations. 44
Inspired by the findings from these studies, we propose a simple yet effective VAE ensemble frame-45 work to improve the disentangled representation by VAE. The proposed VAE ensemble consists of46 multiple VAEs. The latent variables in every pair of these models are connected through linear lay-47 ers to force the latent representations in the ensemble to be similar up to a linear transformation.48 We show that the VAE ensemble objective encourages these pair-wise linear transformations to con-49 verge to trivial transformations, making latent representations of different VAEs in the ensemble to50 be “alike”, thus disentangled. In this paper, we make the following contributions: (1) We introduce51 a simple yet effective VAE ensemble framework to improve the disentangled representation learning52 using the original VAE. (2) We show in theoretical analysis that the linear transformations connect-53 ing the latent representations of the individual models in the ensemble tend to converge to trivial54 transformations thus encourage disentangled representation, and verify this result with experiments.55 (3) We evaluate our approach using the original VAE model, and show the improved state-of-the-art56 performance across different datasets.57
2 RELATED WORK58
Variatioanl Autoencoder is a deep directed probabilistic graphical model consisting of an encoder59 and a decoder (Kingma & Welling, 2013; Rezende et al., 2014). The encoder qφ(z|x) maps the60 input data x ∈ Rn to a probabilistic distribution as the latent representation z ∈ Rd, and the decoder61 qθ(x|z) maps the latent representation to the data space noted as qθ(x|z), where φ and θ represent62 model parameters. The VAE objective is to maximize the marginalized log-likelihood of data. Direct63 optimization of this objective is not tractable and it is approximated by the evidence lower bound64 (ELBO) as:65 LV AE = Eqφ(z|x)[log qθ(x|z)]−KL(qφ(z|x) ‖ p(z)), (1) In practice, the first term is estimated by reconstruction error. The second term is the Kullback-66 Leibler divergence between the posterior qφ(z|x) and the prior p(z) commonly chosen as an67 isotropic unit Gaussian p(z) ∼ N (0, I).68 Disentangled representation by VAE has achieved the state-of-the-art performance (Higgins et al.,69 2017; Burgess et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017), despite the70 fact that the VAE objective only models the marginal distribution of the data instead of the desired71 joint distribution over data and latent variables. The reason for this success is the implementation72 choices of the VAE framework (Rolinek et al., 2019). In practice, the latent variables in VAE often73 work in “polarized” modes. The “passive” mode is defined by µ2j (x) 1 and σ2j (x) ≈ 1, while the74 “active” mode is defined by σ2j (x) 1. The “passive” latent variables closely approximate the prior75 and have little effect on the decoder. The “active” latent variables, on the other hand, are closely76 related to both the per sample KL loss and the decoder output. The “polarized regime” enables a77 reformulated VAE objective showing that VAEs optimize a trade-off between data reconstruction78 and orthogonality of the linear approximation of decoder Jacobian locally around a data sample.79 This PCA-like behavior near data points encourages an identifiable disentangled latent space by80 VAE. Furthermore, it was suggested that finding an appropriate “polarized regime” is dependent81 on the initialization and the hyperparameter tuning of the state-of-the-art approaches. In this study,82 we show that the proposed VAE ensemble aligns the “polarized regime” of individual VAE models83 towards the disentangled representation.84
Model selection In practice, we often observe neural networks achieve similar performance with85 different internal representations when trained with the same hyperparameters (Raghu et al., 2017;86 Wang et al., 2018; Morcos et al., 2018). For the unsupervised disentanlged representation, as dis-87 cussed in (Locatello et al., 2018; Duan et al., 2019), we often observe high variance in the perfor-88 mance from the model trained with the same architecture and hyperparameter setting. This poses89 a challenge for choosing the model in practice. Duan et al. (2019) proposed Unsupervised Disen-90 tanglement Ranking (UDR) to address this challenge. The extensive empirical evaluations on UDR91 using both the supervised metric measurement and the performance of downstream tasks validates92 its effectiveness. They also confirm that disentangled representations are “alike” and entangled rep-93 resentations are unique in their own ways. The proposed VAE ensemble leverages this finding.94
Identifiable VAE Built on the recent breakthroughs in nonlinear Independent Component Analy-95 sis (ICA) literature (Hyvarinen & Morioka, 2016; 2017; Hyvarinen et al., 2019), Khemakhem et96 al. show that the identification of the true joint distribution over observed and latent variables is97
possible up to very simple transformations (Khemakhem et al., 2019). They proposed identifiable 98 VAE (iVAE) that requires a factorized prior distribution over the latent variables conditioned on an 99 additionally observed variable, such as a class label or almost any other observation. We believe the 100 proposed VAE ensemble is related to such framework where the latent representation from one VAE 101 model can be regarded as the auxiliary observations for another. 102
Ensemble learning The idea of ensemble learning is to combine multiple learning models (poten- 103 tially weak learners) to improve the task performance or robustness over a single model (Schapire, 104 1990). It achieves the improved performance by averaging the bias, reducing the variance thus pre- 105 venting the over-fitting (Drucker et al., 1994; Breiman, 1996). Early works in neural networks have 106 used the ensemble learning to achieve top performance in the related competition (Krizhevsky et al., 107 2012; Simonyan & Zisserman, 2014). In this work, we apply the ensemble learning to enforce the 108 alignment among the latent representations of different models. This results in latent representations 109 that are similar among each other in the ensemble up to a trivial transformation. 110
3 THE VAE ENSEMBLE FRAMEWORK 111
As illustrated in Figure 1, the proposed VAE ensemble consists of n original VAE models with the 112 same architecture but different initializations. It also consists of n× (n−1) linear layers connecting 113 the latent representations of every two VAE models. Each model in the ensemble maintains its 114 original VAE objective as Eq. 1. In addition, l2 loss is used to force mapping between latent 115 representations via pair-wise linear layers (cross-model linear transformation). The decoder of each 116 VAE model generates the input reconstruction from not only their corresponding encoder (within- 117 model reconstruction), but also the linearly transformed encodings from other encoders (cross-model 118 reconstruction). Overall, the VAE ensemble is trained with the following objective: 119
L(θ,φ) = n∑ i=1 n∑ j=1 Eqφij(zij |x)[log qθj(x|zij)]− n∑ i=1 KL(qφii(zii|x) ‖ p(zii))
− γ n∑ i=1 n∑ j=1 Eqφij(zij |x) ‖zjj − zij‖ 2 ,
(2)
where n is the number of models in the ensemble; φ := (φij) is the encoders parameters where φij120 represents the encoder of VAEi and its associated linear layer mapping the latent representation from121 VAEi to VAEj (notice that φii represents the encoder parameters of VAEi only and its associated122 linear transformation can be be regarded as an identity transformation); θ := (θi) represents the de-123 coder parameters of VAEi; and zjj represents the latent representation of VAEj while zij represents124 the linearly transformed latent representation from VAEi to VAEj . γ is a hyperparameter to balance125 the effect of the estimation error between the latent representaitons. p(zii) ∼ N (0, I) is assume to126 be the prior as defined in the original VAE objective.127
Comparing to the original VAE objective in Eq. 1, the objective of each individual VAE model in128 the ensemble, L′V AE , can be written as:129
L′V AE(θ, φ) = LV AE(θ, φ) + n−1∑ j=1 { Eqφj(zj |x)[log qθ(x|zj)]− γEqφj(zj |x) ‖zj − z‖ 2 }
︸ ︷︷ ︸ Ensemble Regularization
, (3)
where n is the number of models in the ensemble, φj stands for the parameters of the encoder and130 its linear transformation layers from other VAEs, and zj represents the linear transformed latent131 representation of the encoding from other encoders.132
In this form, the VAE ensemble regularizes each VAE model with additional terms on the encoder133 as γEqφj(zj |x) ‖zj − z‖ 2, and on the decoder as ∑n j=1 Eqφj(zj |x)[log qθ(x|zj)]. These regulariza-134 tions directly constrain the latent representations among different VAE models in the ensemble to135 be similar. In particular, for a given input data, ‖zj − z‖2 encourages the encoders to generate136 similar encodings up to the linear transformations; and ∑n j=1 Eqφj(zj |x)[log qθ(x|zj)] emphasizes137 the similar effect on the data reconstruction from the latent variables such that the decoders can138 reconstruct the input data with both the original encoding z and the linearly transformed encoding139 zj . As we shall discuss in the next section, together these regularizations encourage similar latent140 representation up a trivial transformation by different models in the ensemble.141
We also introduce the hyperparameter γ to balance the trade-off between these two regularizations:142 higher value forces closer mapping between the encoders and reduce the cross-model reconstruction143 error of the decoders; lower value relaxes the mapping between the encoders and increases the cross-144 model reconstruction error of the decoders. As we show in Section 5, both components are important145 and balancing the trade-off between them is important as the ensemble size increases.146
Computational complexity It is a common practice to train a number of seeds per hyperparameter147 setting for the current state-of-the-art unsupervised disentanglement VAE models (Locatello et al.,148 2018; Duan et al., 2019). Comparing to training n original VAEs, the proposed VAE ensemble149 requires additional n × (n − 1) linear layers. While this addition does not increase the size of the150 model much, the estimation of the linear transformations loss and the cross-model reconstruction151 losses grow with n× (n− 1), which may be computationally expensive especially when n is large.152 That being said, the results in Section 5 show that the VAE ensemble achieves more stable results153 comparing to the current state-of-the-art models. Also, its computation is highly parallelisable.154
4 THEORETICAL JUSTIFICATION155
In this section, we present the theoretical analysis on why the proposed VAE ensemble can improve156 the disentangled representation. We start with analysing the l2 objective in Eq. 2 of the pair-wise157 liner transformations in the VAE ensemble, and show that: (1) the pair-wise linear transformations158 encourage similar “polarized” regime (see Sec. 2) among the VAEs in the ensemble; (2) the linear159 transformations are close to the orthogonal transformations. Based on these two properties, we then160 discuss how the cross-model reconstructions by the VAE ensemble encourage learning a disentan-161 gled representation over its entangled counterpart.162
4.1 THE EFFECT OF LINEAR TRANSFORMATION BETWEEN LATENT REPRESENTATIONS163
Let VAEi and VAEj be two different VAE models in the ensemble, and Mji be the linear trans-164 formation that maps the latent representation of a given input x by VAEj to the one by VAEi, as165
zj(x) ∼ N (µj(x), diag(σj(x)2)) to zi(x) ∼ N (µi(x), diag(σi(x)2)). In the following we re- 166 move the input notation from the VAE latent representations for simplicity (i.e. zj(x) is simplified 167 as zj), while keeping in mind that the analysis is based on the local latent representation of a given 168 input x. 169
For VAEj , the l2 term of the VAE ensemble loss in Eq. 2 aims to find Mji and zj that minimize 170 E ‖zi −Mjizj‖2, where the expectation is over the stochasticity of VAEj . We can write zi and zj 171 as zi = µi + i and zj = µj + j , where i ∼ N (0, diag(σ2i )) and j ∼ N (0, diag(σ2j )). Hence 172 using bias-variance decomposition, the l2 term can be written as: 173
min Mji,zj∼N (µj ,diag(σ2j ))
E ‖zi −Mjizj‖2
= min Mji,zj∼N (µj ,diag(σ2j ))
{ ‖µi −Mjiµj‖2 + E
zj ‖Mjiµj −Mjizj‖2 + E zi ‖µi − zi‖2 } = min Mji,zj∼N (µj ,diag(σ2j )) { ‖µi −Mjiµj‖2 + E zj ‖Mjiµj −Mjizj‖2 } + C1
= min Mji,zj∼N (µj ,diag(σ2j ))
{ ‖µi −Mjiµj‖2 + E
j ‖Mji j‖2
} + C1
(4)
where the constant C1 arises from the fact E zi ‖µi − zi‖2 does not depend on Mji and zj. Eq. 4 con- 174 sists of a deterministic component of ‖µi −Mjiµj‖2 and a stochastic component of E j ‖Mji j‖2. 175 The deterministic component can be minimized by adjusting the parameters in VAEj such that its 176 mean encoding µj is optimized for any given Mji. This simplifies our analysis to focus on the 177 stochastic component. Between Mji and j in this stochastic component, we separately optimize 178 one while having the other fixed. 179
We start with fixed Mji and optimizing for j ∼ N (0, diag(σ2j )). Notice that σ2j is associated with 180 VAEj objective. In (Rolinek et al., 2019), the VAE objective is reformulated into the deterministic 181 reconstruction, the stochastic reconstruction and the KL loss. The last two components define the 182 stochastic loss of VAE. It is formulated as: 183
min V,σ2j ∑ X E j∼N (0,diag(σ2j )) ‖D j‖2 s.t. ∑ X LKL = C1, (5)
where X represents the dataset, D represents the local linear approximation of the Jacobian of the 184 decoder with singular value decomposition as D = UΣV T . Furthermore, the KL loss LKL = 185 1 2 ∑d k=1(µ 2 jk +σ 2 jk− log σ2jk− 1) can be simplified as L≈KL = 12 ∑ k∈“active”(µ 2 jk− log σ2jk− 1) 186 based on the “polarized” regime of VAE. (Rolinek et al., 2019) shows that σ2j act as the precision 187 control allowed for each latent variable where more influential ones receive more precision. Com- 188 bining the stochastic loss of the linear transformation in Eq. 4 and the stochastic loss of the original 189 VAE in Eq. 5, the overall stochastic loss on σ2j can be formulated as: 190
min σ2j
E j∼N (0,diag(σ2j )) [‖Mji j‖2 + ‖D j‖2] s.t. ∑ k − log σ2jk = C2, (6)
where σ2jk is the kth element of σ 2 j . Here we further simplify LKL with the L≈KL up 191 to additive constants C2 when µj is fixed. In addition to the precision control of σ2j on 192 VAEj , this objective also aims to find an optimal distribution of σ2j that aligns the “polarized 193 regime” among different VAEs. To see why, let ck be the kth column of Mji, we then have 194 E ‖Mji j‖2 = ∑ k ‖ck‖ 2 σ2jk. The Arithmetic-Mean–Geometric-Mean (AM/GM) inequality sug- 195
gests that ∑ k ‖ck‖ 2 σ2jk ≥ n (∏ k ‖ck‖ 2 σ2jk )1/n = n (∏ k ‖ck‖ 2 )1/n
exp(−C), where the equal- 196 ity is achieved when ‖cm‖2 σ2jm = ‖cn‖ 2 σ2jn for any m 6= n. This suggests that latent variables 197 with high ‖ck‖2 mapping from zj to zi will have smaller variance. Hence, these latent variables in zj 198 are encouraged to stay in the “active” mode. On contrary, the latent variables that do not share sim- 199 ilar generative factors between zj and zi will be assigned larger variance, and being pushed towards 200 the “passive” mode. 201
Now we fix the optimal distribution of σ2j , and optimize for Mji. Since j ∼ N (0, diag(σ2j )), this202 objective can be understood as optimally rotating the latent space of VAEj such that the stochastic203 component in Eq. 4 is minimized. Specifically, we have the following objective:204
min Mji ‖Mji j‖2 = min R ∥∥MjiRT j∥∥2 (7) where R is an orthogonal transformation. Let c′k be the kth column of MjiR. Similar as before, the205 AM/GM inequality suggests
∥∥MjiRT j∥∥2 = ∑k ‖c′k‖2 σ2jk ≥ ∏k ‖c′k‖2 exp(−C3). Hadamard’s206 inequality suggests that ∏ k ‖c′k‖
2 ≥ |det(MjiR)|, and the equality is satisfied when c′ks are pair-207 wise orthogonal. This can be understood from the geometric perspective where ∏ k ‖c′k‖
2 gives an208 upper bound on Volume( { MjiR Tx : x ∈ [0, 1]d }
). As a result, the optimization of Mji will lead to209 an orthogonal transformation. Together the optimization of Eq. 6 and Eq. 7 encourages the align-210 ment of the “polarized regime” among different models under orthogonal linear transformations.211 They force different models in the ensemble to capture the same mixture of the generative factors.212 In the next section, we discuss the effect of the cross-model reconstruction in the VAE ensemble that213 encourages the disentangled representation over the entangled ones.214
4.2 THE EFFECT OF CROSS-MODEL RECONSTRUCTION215
In an entangled representation, each latent variable captures a mixture of generative factors in its216 unique way. Since different generative factors typically have different effects on data variations217 (Duan et al., 2019), the orthogonal transformation from one entangled representation zj to another218 one zi introduces different encoding variance. Some of the transformed latent variables in Mjizj219 carry larger variance comparing to the corresponding ones in zi. This discrepancy leads to larger220 cross-model reconstruction of VAEi than the within model reconstruction. This error forces both221 VAEi and VAEj to adjust their representations until the effect on the data reconstruction by indi-222 vidual latent variables matches between Mjizj and zi. The process applies to all models in the223 ensemble and eventually leads to a one-to-one mapping of latent variables between different mod-224 els, where Mji becomes a trivial transformation (signed permutation matrix). In particular, if one of225 the models in the VAE ensemble learns a disentangled representation, other models in the ensemble226 will converge to it. This is because the orthogonal transformation from an entangled representation227 to a disentangled representation introduces larger cross-model encoding variance due to the mixture228 of different generative factors in the former, thus a larger cross-model reconstruction by the disen-229 tangled model. On contrary, the orthogonal transformation from a disentangled representation to an230 entangled representation would not introduce larger cross-model encoding variance than the within231 model encoding, thus similar cross-model reconstruction as within model reconstruction by the en-232 tangled model. Such a gap encourages the entangled representations to align with the disentangled233 representation. We illustrate the geometric interpretation of such a case in Appendix C.234
From these discussions, we conclude that the VAE ensemble encourages different individual mod-235 els to capture similar generative factors, thus learn representations that are “alike” up to a trivial236 transformation. In the next section, we verify these analytic results with experiments.237
5 EXPERIMENTS238
Our experiments are designed to confirm the discussions in the previous sections. Particularly we239 ask the following questions: Q1: Do the linear transformations in the ensemble converge to trivial240 transformation? Q2: Do the VAEs in the ensemble work in similar “polarized” regime? Q3: Does241 VAE ensemble improve the unsupervised disentangled representation learning, and what is the ef-242 fect of ensemble size? Q4: What are the effects of the cross-model reconstruction loss, the linear243 transformation loss and the hyperparameter γ in the VAE ensemble objective?244
We analyze the inner working of the proposed VAE ensemble using the benchmark dSprite dataset245 (Matthey et al., 2017) with fully known generative processes, and the real-world CelebA dataset (Liu246 et al., 2015) with unknown generative process. Furthermore, for dSprites dataset, we compare our247 results with the original VAE model and the state-of-the-art disentanglement VAE models including248 β-VAE (Higgins et al., 2017), FactorVAE (Kim & Mnih, 2018), TC-VAE (Chen et al., 2018) and249 DIP-VAE (Kumar et al., 2017). We use two widely used supervised metrics including FactorVAE250 metric (Kim & Mnih, 2018) and DCI Disentanglement scores (Eastwood & Williams, 2018) as the251
quantitative measurements. They are shown to correlate with other common supervised metrics 252 (Locatello et al., 2018). For example, FactorVAE metric and β-VAE metric (Higgins et al., 2017) 253 capture similar notions, while DCI Disentanglement and Mutual Information Gap (MIG) (Chen 254 et al., 2018) capture similar notions. In addition, DCI Disentanglement is closely related to the 255 unsupervised model selection method UDR (Duan et al., 2019). For CelebA dataset, we show the 256 latent traversal visulization as a qualitative measurement in Appendix E. We provide the details of 257 the experiments in Appendix D. 258
Q1: We use the Distance to Orthogonality (DtO) (Rolinek et al., 2019) to check if the linear trans- 259 formations in the ensemble converge to a signed permutation matrix during training. DtO is the 260 Frobenius norm of the difference between a matrix M and its closest signed permutation matrix 261 P (M). It is solved with mixed-integer linear programming (MILP) formulation. The details on 262 DtO can be found in Appendix B. In Figure 2, we show the DtO estimation of the linear transfor- 263 mations in the VAE ensemble of different ensemble size for the dSprite dataset. We show the mean 264 and standard deviation of DtO across all linear transformations over 10 different runs. Furthermore, 265 we compare these results with a baseline measurement where DtO is estimated for the linear trans- 266 formations between the mean latent representations of well-trained individual models. Specifically, 267 we use ten well-trained individual models and report the mean and standard deviation of the DtO 268 estimations. As seen in the figure, the VAE ensemble models with different ensemble size all ap- 269 proach to trivial transformations between the individual models, while other VAE models do not 270 have such property. In Fig. 6, we show that during training, the VAE ensemble remains maintains 271 low DtO while the original VAEs do not have such property. A similar result for models trained on 272 the CelebA dataset with different latent dimensions is shown in Fig. 7. Further discussion on these 273 results are provided in Appendix E. 274
Q2: To check if the models in the VAE ensemble work in similar “polarized” regime, we estimate 275 the relative error between LKL and L≈KL as ∆ = LKL−L≈KLLKL for each latent variable. Smaller ∆ 276 indicates closer matching between LKL and L≈KL of a latent variable, thus more “active”. Figure 277 3(a) and 3(b) show log(∆) of the 10 latent variables of individual models in VAE E2 and VAE E3 278 with different γ settings trained on the dSprite dataset. The results show that individual models 279 in the ensemble do work in similar ”polarized regime’. In Figure 3(a), we also compare the VAE 280 ensemble with the β-VAE where β = 4. This setting was found previously to be the optimal setting 281 for the dSprite data for β-VAE (Higgins et al., 2017). We see that the VAE ensemble encourages 282 more “active” latent variables than β-VAE. When we compare Fig. 3(a) and 3(b), we see that as 283 the ensemble size increases, individual models are forced to have more “active” latent variables by 284 decomposing the generative factors. This can be observed in the latent traversals shown in Appendix 285 E. The dSprites dataset contains five ground truth generative factors. The VAE E2 models can 286 have up to eight “active” latent variables depending on input, and these representations capture a 287 decomposition of the ground truth generative factors. 288
Q3: In Table 1, we compare the disentangled representation performance between the proposed289 VAE ensemble and the state-of-the-art models. For the VAE ensemble, we report the performance290 of the first model in the ensemble. We also report the results for the VAE ensemble with different291 ensemble size and γ values. As shown in the table, the VAE ensemble significantly improves the292 performance over the original VAE model. In many settings, the VAE ensemble achieves similar293 or better performance over the state-of-the-art models. In Table 2, we evaluate the consistency294 among the models in the ensemble by reporting the standard deviation of the evaluation metrics295 using different models in the same ensemble. The small values confirm that different models in296 the ensemble learn similar latent representations. Furthermore, Table 1 shows the joint effect of297 ensemble size and γ setting. When γ = 1, the performance of VAE ensemble increases as the298 ensemble size increases, indicated by the higher mean and smaller variance of both the FactorVAE299 and DCI Disentanglement metrics. This behavior is consistent with the characteristic of ensemble300 learning where the increase in performance becomes smaller as the size of ensemble increases.301 However, as γ increases, having larger ensemble size can reduce the performance. We believe this302
is due to the increased difficulty of balancing between the cross-model and within model objectives 303 of VAE ensemble for larger ensembles. The reduced alignment of the latent representations among 304 different models can also be seen in Table 2 where difference in the performance among individual 305 models in the ensemble increases as ensemble size increases. 306
Q4: We conduct the ablation study to further understand the effect of the linear transformation loss 307 and the cross-model reconstruction loss in the VAE ensemble objective. As shown in Fig. 4, remov- 308 ing either component leads to a lower FactorVAE metric for the VAE ensemble. Without the linear 309 transformation loss, the performance of VAE ensemble decreases significantly across different en- 310 semble sizes. Without the cross-model reconstruction loss, the performance of VAE ensemble also 311 decreases but the gap becomes smaller as γ increases. This matches the discussion in Section 3 that 312 higher γ forces closer mapping between the encoders and reduce the cross-model reconstruction er- 313 ror of the decoders. However, this also reduces the effect of cross-model reconstruction as discussed 314 in Section 4.2. A similar result is also found for the DCI Disentanglement metric as shown in Fig. 315 8 in Appendix E. Overall, the results from the ablation study confirms the importance of both the 316 linear transformation loss and the cross-model reconstruction loss in the VAE ensemble objective. 317
6 CONCLUSION 318
In this study, we propose a simple yet effective VAE ensemble framework consisting of multiple 319 original VAEs to learn disentangled representation. The individual models in the ensemble are 320 connected through linear layers that regularize both encoders and decoders to align the latent repre- 321 sentations to be similar up to a linear transformation. We show in theory that the regularization by 322 the VAE ensemble forces the linear transformations to be trivial transformations and show improved 323 performance on the unsupervised disentangled representation learning. The theoretical discussion 324 in Section 4 is based on the original VAE objective, and our experiments also focus on the ensemble 325 with original VAE. We believe such framework can be extended to other disentangled VAE models, 326 or even a mixture of different VAE models, as long as the regularization by the ensemble does not 327 conflict with the augmented objective of these models. 328
REFERENCES329 Leo Breiman. Bagging predictors. Machine learning, 24(2):123–140, 1996.330
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins,331 and Alexander Lerchner. Understanding disentangling in β -VAE. (NIPS), 2017.332
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating Sources of Disentangle-333 ment in Variational Autoencoders. (ICLR):1–13, 2018.334
Harris Drucker, Corinna Cortes, Lawrence D Jackel, Yann LeCun, and Vladimir Vapnik. Boosting335 and other ensemble methods. Neural Computation, 6(6):1289–1301, 1994.336
Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Chris Burgess, Alexander Lerchner, and337 Irina Higgins. Unsupervised model selection for variational disentangled representation learning.338 In International Conference on Learning Representations, 2019.339
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of340 disentangled representations. ICLR, 2018.341
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,342 Shakir Mohamed, and Alexander Lerchner. β-Vae: Learning Basic Visual Concepts With a Con-343 strained Variational Framework. ICLR 2017, (July):1–13, 2017.344
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and345 Alexander Lerchner. Towards a Definition of Disentangled Representations. arXiv, pp. 1–29,346 2018.347
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning348 and nonlinear ica. In Advances in Neural Information Processing Systems, pp. 3765–3773, 2016.349
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and350 generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence351 and Statistics, pp. 859–868, 2019.352
AJ Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources.353 Proceedings of Machine Learning Research, 2017.354
Ilyes Khemakhem, Diederik P Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlin-355 ear ica: A unifying framework. arXiv preprint arXiv:1907.04809, 2019.356
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. Proceedings of Machine Learning357 Research, 80:2649–2658, 10–15 Jul 2018.358
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114,359 2013.360
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo-361 lutional neural networks. In Advances in neural information processing systems, pp. 1097–1105,362 2012.363
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan-364 gled latent concepts from unlabeled observations. CoRR, abs/1711.00848, 2017.365
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.366 In Proceedings of International Conference on Computer Vision (ICCV), December 2015.367
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and L G Dec.368 Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representa-369 tions. arXiv, pp. 1–33, 2018.370
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and371 Olivier Bachem. On the fairness of disentangled representations. In Advances in Neural Informa-372 tion Processing Systems, pp. 14584–14597, 2019.373
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in 374 variational autoencoders. arXiv preprint arXiv:1812.02833, 2018. 375
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement 376 testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017. 377
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural 378 networks with canonical correlation. In Advances in Neural Information Processing Systems, pp. 379 5727–5736, 2018. 380
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector 381 canonical correlation analysis for deep learning dynamics and interpretability. In Advances in 382 Neural Information Processing Systems, pp. 6076–6085, 2017. 383
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and 384 approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014. 385
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pca direc- 386 tions (by accident). In Proceedings of the IEEE Conference on Computer Vision and Pattern 387 Recognition, pp. 12406–12415, 2019. 388
Robert E Schapire. The strength of weak learnability. Machine learning, 5(2):197–227, 1990. 389
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image 390 recognition. arXiv preprint arXiv:1409.1556, 2014. 391
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- 392 tangled representations helpful for abstract visual reasoning? In Advances in Neural Information 393 Processing Systems, pp. 14222–14235, 2019. 394
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards 395 understanding learning representations: To what extent do different neural networks learn the 396 same representation. In Advances in Neural Information Processing Systems, pp. 9584–9593, 397 2018. 398
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. 399 Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven 400 exploration. arXiv preprint arXiv:1905.09275, 2019. 401
A TECHNICAL LEMMAS402
In this section, we give the lemmas used in the theoretical discussion in Section 4.403
Lemma 1. (Jensen’s inequality) If g(x) is a convex transformation on x, then this convex transfor-404 mation of a mean g[E(x)] is less than or equal to the mean of the convex transformed value E[g(x)];405 it is a simple corollary that the opposite is true of concave transformations.406
Lemma 2. (AM-GM inequality) As an extension of Jensen’s inequality, given a list of non-negative407 real numbers x1, x2, . . . , xn, the arithmetic mean of this list 1n ∑n i=1 xi is greater than or equal408
to the geometric mean of the same list ( ∏n i=1 xi) 1 n ; and further, the equality holds if and only if409 x1 = x2 = · · · = xn.410 Lemma 3. (Hadamard’s inequality). if M is the matrix having columns ci, then |det(M)| ≤411 ∏n i=1 ‖ci‖; and the equality in Hadamard’s inequality is achieved if and only if the vectors are412 orthogonal.413
B DISTANCE TO ORTHOGONALITY (DTO)414
In this section, we introduce the detail of Distance to Orthogonality (DtO) that is used in our exper-415 iment to check if the linear transformations in the VAE ensemble approach trivial transformations.416 This measurement is also used in (Rolinek et al., 2019) for a similar purpose. DtO is the Frobe-417 nius norm of the difference between a square matrix M and its closest signed permutation matrix418 P (M). Finding P (M) can be formulated as a mixed-integer linear programming (MILP) problem419 as following:420
min P ∑ i,j |Mi,j − P (M)i,j | s.t. P (M)i,j ∈ {−1, 0, 1} , ∀(i, j)∑ i
|Pi,j | = 1, ∀j∑ j |Pi,j | = 1, ∀i
(8)
By introducing new variables P+i,j , P − i,j ∈ {0, 1} and Di,j = |Mi,j − P (M)i,j |, we can reformulate421 the above optimization problem as:422
min P ∑ i,j Di,j
s.t. (P+i,j − P − i,j)−Mi,j ≤ Di,j , ∀(i, j)
Mi,j − (P+i,j − P − i,j) ≤ Di,j , ∀(i, j)∑
i (P+i,j + P − i,j) = 1, ∀j∑
j
(P+i,j + P − i,j) = 1, ∀i
(9)
Using this optimization formulation, DoT of a given matrix M ∈ Rn×n is defined as:423
DoT = 1
n2 ∑ i,j |Mi,j − P (M)i,j | (10)
C GEOMETRIC INTERPRETATION OF THE EFFECT OF CROSS-MODEL 424 RECONSTRUCTION 425
Given a disentangled and entangled latent representation space, Fig. 5 illustrates the effect of the 426 cross-model reconstruction by VAE ensemble. The left part shows the orthogonal transformation 427 from a disentangled representation to an entangled space, and the right part shows the transfor- 428 mation in the opposite direction. As shown in the figure, the orthogonal transformation from the 429 disentangled representation to the entangled space does not introduce larger variance than the en- 430 tangled representation. Hence, we can expect similar cross-model reconstruction and within model 431 reconstruction. However, the transformation from the entangled representation to the disentangled 432 space introduces larger variance (yellow shaded area over blue area on the right) than the disentan- 433 gled representation. This leads to larger cross-model reconstruction by the disentangled model. 434
D MODEL ARCHITECTURE AND TRAINING DETAILS 435
We conducted our experiments, including training and evaluating the current state-of-the- 436 art disentanglement models as well as evaluating the proposed VAE ensembles, using the 437 disentanglement lib 1 open-source library (Locatello et al., 2018). 438
Table 3 shows the encoder and the decoder architecture of the VAE model used in our experiments. 439 This architecture is the same as the one used in the original β-VAE Higgins et al. (2017). 440
Table 4 shows the hyperparameters setting used throughout the experiments. These parameters are 441 fixed for all the experiments. 442
E ADDITIONAL EXPERIMENTAL RESULTS 443
In this section, we present the additional results including the DtO and “polarized regime” analysis 444 on the models trained on the CelebA dataset similar to the ones conducted on dSprite dataset in 445 Section 5; the ablation results with DCI-Disentanglement metric and the DtO estimation; and the 446
1https://github.com/google-research/disentanglement_lib
latent traversal of the trained model on both dSprite and CelebA dataset along with further discuss447 the effect of VAE ensemble on the latent representation.448
E.1 CHARACTERIZATION OF THE LINEAR TRANSFORMATION IN VAE ENSEMBLE449
In Figure 6, we show the DtO estimation of the linear transformations in the ensemble during train-450 ing for the dSprite dataset. We report the mean and standard deviation of DtO across all linear451 transformations over 10 different runs. Furthermore, we compare these results with a VAE baseline452 where DtO is estimated for the linear transformations between original VAEs. Specifically, we train453 ten different VAEs separately and estimate the DtO of the pairwise linear transformation among454 these models during training. Similarly we report the mean and standard deviation of these DtO es-455 timations. As seen in the figure, the VAE ensemble models with different ensemble size all approach456 to trivial transformations between the individual models, while the original VAEs do not have such457 property. A similar result is also found in models trained for CelebA dataset. Similar to the results458 in Figure 6, we observe decreased DtO of the linear transformations in the VAE ensemble during459 training.460
We also compare models trained with different latent dimension size. We observe decreased DtO461 as the latent dimension of the model increases in Figure 7. This is because, as discussed in the462 main paper, the VAE ensemble encourages more “active” latent variables. Models with higher latent463 dimension likely to learn a decomposition of generative factors. As a result, the alignment of the464 latent variable between different models are easier thus the linear transformations between the latent465 representations is closer to the trivial transformation. On the contrary when there are less latent466 variables in the model than the generative factors, some of the latent variables will capture more467 than a single generative factor. As a result, the one-to-one mapping between the latent variables of468 different models will not lead to a trivial transformation.469
In Figure 9 and Figure 10, we show the “polarized regime” estimation for models in VAE E2 and 470 VAE E3 trained for CelebA datasets, respectively. Similar to the results in Figure 3, individual 471 models in the VAE ensemble tend to have similar ‘polarized regime”, and higher γ enforces the “po- 472 larized regime” by separating “passive” latent variables from the “active” ones. When we compare 473 between VAE E2 and VAE E3, we observe increased “active” latent variables similar to the result 474 on dSprite dataset in Section 5. More importantly, as discussed earlier, latent variables in a model 475 with limited latent dimensions need to capture more than a single generative factor, especially for 476 a complicated real-world dataset such as CelebA. This makes the linear transformation between the 477 latent representations less trivial. As the latent dimension size grows, such constraint is relaxed and 478 the linear transformations are closer to trivial. 479
These additional results confirm the conclusion in Section 5: (1) as the ensemble size increases, 480 DtO increases due to the difficulty of aligning the latent representations among different models; 481 (2) as the model latent dimension increases, DtO decreases due to the increased model capacity, and 482 encourages the one-to-one mapping between latent variables in different models; (3) hyperparameter 483 γ does not affect DtO significantly, but plays an important role on separating “active” and “passive” 484 latent variables, especially when the latent dimension is large enough. 485
Furthermore, we believe the DtO measurement of the linear transformation in VAE ensemble could 486 be a useful indicator for latent dimension size. As shown in Figure 7 and Figure 9, when the latent 487 dimension is sufficient for a given dataset, the DtO of the linear transformation is small and some 488 latent variables are pushed to “passive” mode. 489
E.2 ABLATION STUDY 490
Similar to the ablation result shown in Section 5, here we show the same ablation study using the DCI 491 Disentanglement metric in Fig. 8(a) as well as the DtO measurement 8(b). Similar as the results of 492 the FactorVAE metric in Fig. 4, removing either component leads to a lower DCI Disentanglement 493 metric for the VAE ensemble. Without the linear transformation loss, the performance of VAE 494 ensemble decreases significantly across different ensemble sizes. Fig. 8(b) shows that for VAE 495
ensemble without the cross-model reconstruction, the linear transformations among models are close496 to a trivial transformation (signed permutation). This implies the orthonormal transformation of the497 linear transformations. This result further supports our intuitive justification in Appendix C that the498 cross-model objective encourages entangled models to align to disentangled models. Indeed, we see499 that adding the cross-model reconstruction can further reduce the DtO of the linear transformations500 among the models in the ensemble.501
E.3 LATENT TRAVERSAL502
In this section we show the latent traversal of models trained on both dSprites and CelebA datasets.503 For a fixed input image, to extract the latent traversal we change the value of a single latent variable504 zi in the corresponding encoding, and observe the generated output image to understand the effect505 of zi. The range of the value are usually chosen to be from -3 to 3 due to the standard Gaussian506 prior.507
In Figure 11, we show the latent traversal for both VAE E2 and a single VAE model with 10 latent508 dimensions trained on dSprites dataset. Three images as shown in the last column of each block509
are used as input. Both models are able to capture certain generative factors of the data including 510 position, shape, rotation and scale. In Section 5, we argue that the representation by VAE ensemble 511 encourages more “active” latent variables, thus can capture a decomposition of the ground truth gen- 512 erative factors. Especially from the “polarized regime” estimation in Figure 3, we observe that some 513 latent variables in the VAE ensemble are in-between “active” and “passive” modes. This suggests 514 that the VAE ensemble model generates input-dependent factors based on the input complexity. In 515 Figure 11, we observe this behavior highlighted with color boxes. The traversal on the second la- 516 tent variable z2 shows that an ellipse shape does not lead to an “active” latent variable. However, 517 both heart and square shape lead to an “active” latent variable that changes the output. In contrast, 518 the single VAE model does not have such behavior where the “active” modes are consistent across 519 different input data. 520
In Figure 12 and Figure 13, we show the latent traversal for both VAE E2 and a single VAE model 521 with 16 latent dimensions trained on CelebA dataset, respectively. In this real-world dataset, the 522 generative factors are unknown. We observe different factors including background, azimuth, gen- 523 der, hair style being captured by both models. Similar as before, the single VAE model maintains 524 similar “active” mode for all latent variables where similar traversal patterns are observed for both 525 input images. However, VAE E2 shows semantically consistent but input-dependent “active” mode. 526 This is translated into different traversal effects and more realistic and sharper images by VAE E2, 527 especially for the first input image that is less common in the dataset. We believe this is important 528 towards a meaningful compositional latent representation learning. 529
Overall, the latent traversal results in this section confirm the findings on the inner working of the 530 VAE ensemble shown in the previous section as well as the discussion in Section 5. 531


summaries: The paper proposes an efficient way to automatically choose the best or most suitable pipeline for different datasets. The proposed method can accelerate the AutoML using a pre-trained meta module. In particular, the AutoML job of a new supervised learning task can be accomplished without model evaluations, namely zero-shot / real-time AutoML. The meta module is constructed as a graph structure in which each node represents a dataset used for meta-training.

summaries: The problem that the authors attempt to solve is to determine what ML pipeline will perform best on any new dataset, without incurring in the extra cost of actually running a large number of such pipelines, as is typically done in AutoML algorithms. The way this paper tackles the problem is to train a neural network that given a new dataset as input, will output a pipeline that is predicted to perform well on that dataset. This neural network is trained on other datasets, for which high performing pipelines are already known. Predicting a pipeline for a new dataset thus only require a forward pass through their NN.

summaries: This paper presents a very interesting idea of utilizing the documentation for the data and the operators in the pipeline to generate meta-features for meta-learning. This is a very novel application of graph neural networks GNNs and language models for AutoML meta-learning. This view of meta-learning takes a very intuitive on a very high level. The use of the outputs of existing AutoML systems (such as auto-sklearn, TPOT, etc) is also very intuitive and well motivated. All these intuitive ideas are put together into a novel AutoML recommendation architecture making use of modern deep learning components.

abstractText: We present a new zero-shot approach to automated machine learning (AutoML) that predicts a high-quality model for a supervised learning task and dataset in real-time without fitting a single model. In contrast, most AutoML systems require tens or hundreds of model evaluations. Hence our approach accelerates AutoML by orders of magnitude. Our method uses a transformer-based language embedding to represent datasets and algorithms using their free-text descriptions and a meta-feature extractor to represent the data. We train a graph neural network in which each node represents a dataset to predict the best machine learning pipeline for a new test dataset. The graph neural network generalizes to new datasets and new sets of datasets. Our approach leverages the progress of unsupervised representation learning in natural language processing to provide a significant boost to AutoML. Performance is competitive with state-of-the-art AutoML systems while reducing running time from minutes to seconds and prediction time from minutes to milliseconds, providing AutoML in real-time.

authors:
Dan Brickley
Matthew Burgess
Natasha Noy

title: Google Dataset Search: Building a search engine for datasets in an open web ecosystem

venue: In The World Wide Web Conference,

authors:
Boyuan Chen
Harvey Wu
Warren Mo
Ishanu Chattopadhyay
Hod Lipson

title: Autostacker: A compositional evolutionary learning system

venue: The Genetic and Evolutionary Computation Conference,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
Iddo Drori
Yamuna Krishnamurthy
Remi Rampin
Raoni Lourenco
Jorge One
Kyunghyun Cho
Claudio Silva
Juliana Freire

title: AlphaD3M: Machine learning pipeline synthesis

venue: In ICML International Workshop on Automated Machine Learning,

authors:
Iddo Drori
Yamuna Krishnamurthy
Remi Rampin
Raoni Lourenco
Kyunghyun Cho
Claudio Silva
Juliana Freire

title: Automatic machine learning by pipeline synthesis using model-based reinforcement learning and a grammar

authors:
Iddo Drori
Lu Liu
Sharath Koorathota
Nian Yi
Jie Li
Antonio Khalil Moretti
Juliana Freire
Madeleine Udell

title: Automl using metadata language embeddings

venue: In NeurIPS Workshop on Meta-Learning,

authors:
Matthias Feurer
Aaron Klein
Katharina Eggensperger
Jost Springenberg
Manuel Blum
Frank Hutter

title: Efficient and robust automated machine learning

venue: In Advances in Neural Information Processing Systems,

authors:
Nicolo Fusi
Rishit Sheth
Melih Elibol

title: Probabilistic matrix factorization for automated machine learning

authors:
Xin He
Kaiyong Zhao
Xiaowen Chu

title: AutoML: A survey of the state-of-the-art

venue: arXiv preprint arXiv:1908.00709,

authors:
Efi Kokiopoulou
Anja Hauth
Luciano Sbaiz
Andrea Gesmundo
Gabor Bartok
Jesse Berent

title: Fast task-aware architecture inference

venue: arXiv preprint arXiv:1902.05781,

authors:
Tomas Mikolov
G.s Corrado
Kai Chen
Jeffrey Dean

title: Efficient estimation of word representations in vector space

venue: In International Conference on Learning Representations, pp. 1–12,

authors:
Mitar Milutinovic
Atilim Gunes Baydi
Robert Zinkov
William Harvey
Dawn Song
Frank Wood

title: End-to-end training of differentiable pipelines across machine learning frameworks

venue: In NIPS Workshop on Autodiff,

title: Announcing automated ML capability in Azure Machine Learning, 2019

venue: URL https://azure.microsoft.com

authors:
Randal S Olson
Jason H Moore

title: TPOT: A tree-based pipeline optimization tool for automating machine learning

venue: In Automated Machine Learning,

authors:
F. Pedregosa
G. Varoquaux
A. Gramfort
V. Michel
B. Thirion
O. Grisel
M. Blondel
P. Prettenhofer
R. Weiss
V. Dubourg
J. Vanderplas
A. Passos
D. Cournapeau
M. Brucher
M. Perrot
E. Duchesnay

title: Scikit-learn: Machine learning in Python

venue: Journal of Machine Learning Research,

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer

authors:
Mohammad Shoeybi
Mostofa Patwary
Raul Puri
Patrick LeGresley
Jared Casper
Bryan Catanzaro

title: Megatron-LM: Training multi-billion parameter language models using GPU model parallelism

authors:
Jasper Snoek
Hugo Larochelle
Ryan P Adams

title: Practical Bayesian optimization of machine learning algorithms

authors:
Francisco J Solis
Roger J-B Wets

title: Minimization by random search techniques

venue: Mathematics of Operations Research,

authors:
Thomas Swearingen
Will Drevo
Bennett Cyphers
Alfredo Cuesta-Infante
Arun Ross
Kalyan Veeramachaneni

title: ATM: A distributed, collaborative, scalable system for automated machine learning

venue: In IEEE International Conference on Big Data,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Chengrun Yang
Yuji Akimoto
Dae Won Kim
Madeleine Udell

title: OBOE: Collaborative filtering for AutoML model selection

venue: In International Conference on Knowledge Discovery & Data Mining,

authors:
Chengrun Yang
Ziyang Wu
Jicong Fan
Madeleine Udell

title: AutoML pipeline selection: Efficiently navigating the combinatorial space

venue: In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD),

authors:
Marc-André Zöller
Marco F Huber

title: Benchmark and survey on automated machine learning

venue: arXiv preprint arXiv:1904.12054,

heading: 1 INTRODUCTION

text:
A data scientist facing a challenging new supervised learning task does not generally invent a new algorithm. Instead, they consider what they know about the dataset and which algorithms have worked well for similar datasets in past experience. Automated machine learning (AutoML) seeks to automate these tasks to enable widespread use of machine learning by non-experts. A major challenge is to develop fast, efficient algorithms to accelerate applications of machine learning (Kokiopoulou et al., 2019). This work develops automated solutions that exploit human expertise to learn which datasets are similar and what algorithms perform best. We use a transformer-based language model (Devlin et al., 2018) allowing our AutoML system to process text descriptions of datasets and algorithms, and a feature extractor (BYU-DML, 2019) to represent the data itself. Using such models for our representation brings in large-scale data. We allow to train our model on other existing AutoML system solutions, specifically AutoSklearn (Feurer et al., 2015), AlphaD3M (Drori et al., 2018), OBOE (Yang et al., 2019), and TPOT (Olson & Moore, 2019), tapping into their diverse set of solutions. Our approach fuses these representations (dataset description, data, AutoML pipeline descriptions) and represents datasets as nodes in a graph of datasets.
Generally, graph neural networks are used for three main tasks: (i) node prediction, (ii) link prediction, and (iii) sub-graph or entire graph classification. In this work we use a GNN for node prediction, which predicts the machine learning pipeline for an unseen dataset. Specifically, we use a graph attention network (GAT) (Veličković et al., 2018) with neighborhood aggregation, in which an attention function adaptively controls the contribution of neighbors. An advantage of using a GNN for AutoML is boosting AutoML performance by sharing information between datasets (graph nodes): including description and algorithm, by message passing between the nodes in the graph. In addition, GNNs generalize well to a new unknown dataset using the aggregated weights learnt over the training datasets. GNN weights are shared with the test dataset for prediction. GNNs generalize to entire new sets of datasets. Finally, prediction is in real-time, within milliseconds.
A simple idea is to use machine learning pipelines that performed well (for the same task) on similar datasets. What constitutes a similar dataset? The success of an AutoML system often hinges on this question, and different frameworks have different answers: for example, AutoSklearn (Feurer et al., 2015) computes a set of meta-features, which are features describing the data features, for each dataset, while OBOE (Yang et al., 2019) uses the performance of a few fast, informative models to compute latent features. More generally, for any supervised learning task, one can view the list of
recommended algorithms generated by any AutoML system as a vector describing that task. This work is the first to use the information that a human would check first: a summary description of the dataset and algorithms, written in free text. These dataset features induce a metric structure on the space of datasets. Under an ideal metric, a model that performs well on one dataset would also perform well on nearby datasets. The methods we develop in this work show how to learn such a metric using the recommendations of an AutoML framework together with the dataset description. We provide a new zero-shot AutoML method that predicts accurate machine learning pipelines for an unseen dataset and classification task in real-time and runs the pipeline in a few seconds.
We use a transformer-based language model to embed the description of the dataset and pipelines and a feature extractor to compute meta-features from the data. Based on the description embedding and meta-features, we build a graph as the input to a graph neural network (GNN). Each dataset represents a node in the graph, together with its corresponding feature vector. The GNN is trained to predict a machine learning pipeline for a new node (dataset). Therefore, given a new dataset, our real-time AutoML method predicts a pipeline with good performance within milliseconds. The running time of our predicted pipeline is a few seconds and the accuracy of the predicted pipeline is competitive with state-of-the-art AutoML methods that are given one minute. This work makes several contributions by using language embeddings and GNNs for AutoML for the first time, and leveraging existing AutoML systems. The result is a real-time high-quality AutoML system.
Real-time. Our system predicts a machine learning pipeline for a new dataset in milliseconds and then runs the pipeline and tunes its hyper-parameters within three seconds. This reduces computation time by orders of magnitude compared with state-of-the-art AutoML systems, while improving performance.
GNN architecture. Our work achieves real-time AutoML by introducing several architectural components that are new to AutoML. These include embeddings for dataset descriptions and algorithm descriptions using a state-of-the-art transformer-based language model in addition to (standard) embeddings for data; a non-Euclidean embedding of datasets as a graph; and a predictive model employing a GNN on the graph of datasets. Importantly, the GNN recommends a pipeline for a new dataset by adding a node to the graph of datasets and sharing the GNN weights with the new node. Using the information and relationships between all datasets boosts AutoML performance.
Embeddings. Bringing techniques from NLP to AutoML, specifically using a large-scale transformer-based language model to embed the description of the dataset and algorithms, brings in information from a large corpra of text. This allows our zero-shot AutoML to train on a small set of datasets with state-of-the-art test set performance.
Leveraging existing AutoML systems. Our flexible architecture can use pipeline recommendations from any number of other AutoML systems to improve performance.

heading: 2 RELATED WORK

text:
AutoML is an emerging field of machine learning with the potential to transform the practice of Data Science by automatically choosing a model to best fit the data. Several comprehensive surveys of the field are available (He et al., 2019; Zöller & Huber, 2019).
Processing each dataset in isolation. The most straightforward approach to AutoML considers each dataset in isolation and asks how to choose the best hyper-parameter settings for a given algorithm. While the most popular method is still grid search, other more efficient approaches include Bayesian optimization (Snoek et al., 2012) or random search (Solis & Wets, 1981).
Recommender systems. These methods learn (often, exhaustively) what algorithms and hyperparameter settings performed best for a training set-of-datasets and use this information to select better algorithms on a test set without exhaustive search. This approach reduces the time required to find a good model. An example is OBOE (Yang et al., 2019; 2020), which fits a low rank model to learn the low-dimensional representations for the models (or pipelines) and datasets that best predict the cross-validated errors, among all bilinear models. To find promising models for a new dataset,
OBOE runs a set of fast but informative algorithms on the new dataset and uses their cross-validated errors to infer the feature vector for the new dataset. A related approach (Fusi et al., 2018) using probabilistic matrix factorization powers Microsoft Azure’s AutoML service (Mukunthu, 2019).
Search trees. Auto-Tuned Models (Swearingen et al., 2017) represent the search space as a tree with nodes being algorithms or hyper-parameters and searches for the best branch using a multiarmed bandit.
Model-based reinforcement learning. AlphaD3M (Drori et al., 2018; 2019a) formulated AutoML as a single player game. The system uses reinforcement learning with self-play and a pretrained model which generalizes from many different datasets and similar tasks.
Genetic programming. TPOT (Olson & Moore, 2019) and Autostacker (Chen et al., 2018) use genetic programming to choose both hyper-parameter settings and a topology of a machine learning pipeline. TPOT represents pipelines as trees, whereas Autostacker represents them as layers.
Bayesian optimization. AutoSklearn (Feurer et al., 2015) chooses a model for a new dataset by first computing (ad hoc) data meta-features to find nearest-neighbor datasets. The best-performing methods on the neighbors are refined via Bayesian optimization and used to form an ensemble.
Differentiable programming. End-to-end learning of machine learning pipelines is performed using differentiable primitives (Milutinovic et al., 2017) forming a directed acyclic graph.
Algorithmic primitives. One major factor in the performance of an AutoML system is the base set of algorithms it can use to compose more complex pipelines. For a fair comparison, in our numerical experiments we compare our proposed methods only to other AutoML systems that use Scikit-learn (Pedregosa et al., 2011) primitives.
Embeddings. Language has a common unstructured representation as a sequence of words, sentences, or paragraphs. The most significant recent progress in NLP is large-scale transformer-based models and embeddings (Devlin et al., 2018; Shoeybi et al., 2019; Raffel et al., 2019) based on attention mechanisms (Vaswani et al., 2017). An unsupervised corpus of text is transformed into a supervised dataset by defining content-target pairs along the entire text: for example, target words that appear in each sentence, or target sentences which appear in each paragraph. A language model is first trained to learn a low dimensional embedding of words or sentences followed by a map from low dimensional content to target (Mikolov et al., 2013). This embedding is then used on a new, unseen and small dataset in the same low-dimensional space. Our work uses such embeddings for automatic machine learning. In a similar fashion to recent work (Drori et al., 2019b) we use an embedding for the dataset and algorithm descriptions. In this work we model the non-linear interactions between these embedding using a neural network as well.

heading: 3 METHODS

text: Our zero-shot AutoML predicts a machine learning pipeline for a classification task on a dataset based on the dataset description and data, and based on other datasets, their relationships, and their recommended pipelines by AutoML systems. We embed the dataset description and extract data meta-features to construct a graph of datasets where each node represents a dataset. The graph is processed using a graph neural network (GNN). Each node of the graph contains a feature vector which is the fusion of the description embedding and data meta-features, and the GNN node representations includes other AutoML solutions. The machine learning pipeline for a new dataset is predicted by the GNN. A detailed architecture is illustrated in Figure 1 and described by Algorithms 2 and 3. The notation used in this work are given in Table 1.

heading: 3.1 PRE-PROCESSING

text:
Our pre-processing consists of (i) dataset description embedding; (ii) dataset meta-feature extraction; and (iii) pipeline computation and description embeddings, as described next and summarized in Algorithm 1.
Dataset Description Embedding. We create a feature vector by embeding the descriptionM(D) of each dataset as a 1024-dimensional vector FM = E(M(D)) ∈ R1024 using BERT (Devlin et al., 2018). The supplementary material shows examples of dataset descriptions embedded using our approach.
Data meta-features. We compute meta-features FD ∈ R148 for the dataset D using a feature extractor (BYU-DML, 2019), restricting to meta-features that can be computed in one second on any of the datasets used in our experiments. Meta-features include statistics of the datasets and results of simple algorithms.
Pipelines and pipeline embedding. For each dataset, we compute the recommended pipeline returned by AutoML systems OBOE (O), AutoSklearn (S), AlphaD3M (A), and TPOT (T). We create feature vectors for recommended pipelines by embedding the Scikit-learn documentations for pre-processor or feature selector and estimator (which is unique within each pipeline). Again, we use the BERT embedding, to form a 1024-dimensional embeddingE(M(PC(D))) ∈ R1024 for each pipeline, where C ranges over the AutoML methods O, S, A, and T. The best-performing pipeline P? returned by any AutoML system serves as our training label: we train our system to recommend this pipeline.
Fused dataset representations. The combined representation of dataset Di with description M(Di) fuses together the dataset description embedding and data meta-features using a neural network (whose weights are learned):
Fi = fφ([FDi ,FMi ]) ∈ R512. (1) We also represent the dataset and its best pipeline by fusing this representation with the pipeline embedding using a second neural network:
ui = gθ([Fi,FP?(Di)]) ∈ R 512 (2)
Experimentally, these fused representations improve performance compared to concatenation.
Algorithm 1 Zero-shot AutoML pre-processing Input: training datasets {(Di,Mi)}i∈V . Output: features {FMi,FDi,FP?(Di)}i∈V . for i = 1 to n do
compute embedding of description FMi = E(Mi) compute data meta-features FDi for all C ∈ O, S, A, T do
compute recommended pipeline PC(Di) compute performance on dataset R(PC,Di)
end for select best performing pipeline P?(Di) embed pipeline FP?(Di) = E(M(P?(Di)))
end for

heading: 3.2 GRAPH REPRESENTATION

text:
We build a graph G = (V,E) where each node i ∈ V represents the dataset Di and has feature vector vi.
Nodes. The feature vector vi = [Fi,FPC(Di)] ∈ R1536 for node i representing dataset Di with descriptionM(Di) concatenates the fused dataset representation (described above) Fi ∈ R512 and the pipeline embeddingFP = E(P?) ∈ R1024 for the pipelineP? that performed best on the dataset. During training, we mask the pipeline embedding from the feature vector and learn to predict a node using the GNN.
Edges. To compute the edges of the graph G, we compute the distance d between each pair of datasets i, j as d = ‖Fi − Fj‖2 where Fi and Fj are the fused dataset representations (described above) for the datasets. Two datasets are connected by an edge if dataset j is one of the k nearest neighbors of dataset i or vice versa. In our experiments, we chose k = 20: we found that our method is reasonably robust to the choice of k; that Euclidean distance outperforms cosine similarity; and that a k-NN graph outperforms a threshold-based graph.
At training time, we build this graph on the training datasets. At test time, given a new test dataset, we dynamically connect the new node to the graph using its fused feature representation fφ([FMtest ,FDtest ]) to choose edges. Notice that the edges for the new dataset are chosen quickly, without fitting a single machine learning model.

heading: 3.3 NEURAL NETWORK ARCHITECTURE

text:
The neural networks we train for zero-shot AutoML consist of two fusion networks and a graph attention network (a type of GNN). The fusion networks are used to capture the non-linear inter-
actions between the features corresponding to the dataset description, the data meta-features, and the pipeline embedding. The GNN predicts the best pipeline for a new dataset based the weights optimized during training as described next.
Graph Attention Network. A graph attention network (GAT) (Veličković et al., 2018) is used to predict the best pipeline for a new dataset. Each layer l = 1, ..., L of the GNN updates the feature vector at the i-th node as:
uli = αiiWu l−1 i + ∑ j∈N (i) αijWu l−1 j , (3)
where W is a learnable weight matrix, N (i) are the neighbors of the i-th node, and αij are the attention coefficients, defined as:
αij = exp(σ(z> [Wui,Wuj ]))∑
k∈N (i) exp(σ(z > [Wui,Wuk]))
, (4)
where z is a learnable vector, and σ(·) is the leaky ReLU activation function. Our GNN consists of 3 GAT layers. The last layer of our GNN is a softmax which computes a vector of probabilities over pipelines. Hence the output of the GAT is a probability distribution over pipelines for each node. The network recommends the pipeline that maximizes this probability. Alternatively, we may sample from this probability distribution to obtain several pipelines that can be combined into an ensemble.

heading: 3.4 TRAINING AND TESTING

text:
Training. Our training process is illustrated in Figures 1 and 2, and described in Algorithm 2. At each training iteration, we randomly select a node i. We mask the pipeline embedding of the i-th node as ui = gθ([Fi,0]). The true label is defined as the pipeline with best performance among the four AutoML systems P?(Di) on the i-th dataset. The resulting problem is a multi-class classification problem with as many classes as there are distinct algorithms.
The loss function is defined by cross-entropy between the probability p̂ of predicted algorithm P̂(Di) and one-hot encoding y of the best algorithm P?(Di):
L(P̂(Di),P?(Di)) = − m∑ l=1 yl log(p̂l). (5)
Algorithm 2 Zero-shot AutoML training Input: training datasets, descriptions {Di,M(Di)}i∈V . Output: datasets graph G, GNN hW,z , fusion networks fφ and gθ . pre-process: compute {FMi,FDi,FP?(Di)}i∈V initialize fusion networks weights φ, θ. initialize GNN weights W, z. for each backprop iteration do
generate updated datasets graph G: for i = 1 to n do
compute fused representation Fi = fφ(FDi,Mi) end for compute pairwise distances d(Fi,Fj)i,j∈V for i = 1 to n do
connect node i to k-NN nodes N (i) end for select random node i in G compute ui = gθ(Fi,0) for all j 6= i do
compute uj = gθ(Fj ,FP∗(Dj)) end for predict best pipeline P̂(Di) = hW,z(ui, {uj}j∈N (i)) compute loss L(P̂(Di),P?(Di)) update weights
end for
Testing. Our testing process is illustrated on the right path of Figure 1 and Algorithm 3. Given a new dataset D and description M, we compute the description embedding FM and data metafeatures FD and the fused dataset representation F . We use this representation to compute the edges of this new node in the graph of all datasets. Next, we add the new node, with features u = gθ([F ,0]), to the current graph, replacing the embedding of the pipeline with the zero vector. Finally, we use the graph neural network to recommend a pipeline for the test dataset.
Algorithm 3 Zero-shot AutoML testing Input: dataset Di, description M(Di), datasets graph G, GNN, s.t. i 6∈ V (disjoint train and test). Output: predict best pipeline P̂(Di) for task on dataset. generate new node i in G: compute embedding of description FM = E(M(Di)) compute data meta-features FD compute fused representation F = fφ(FD,FM) connect node i to k-NN nodes j ∈ N (i), V = V ∪ {i}. compute ui = gθ(F ,0) predict best pipeline P̂(Di) = hW,z(ui, {uj}j∈N (i))
Notice our method does not need to complete even a single model fit to recommend a model with hyper-parameters. On the other hand, we must fit the model (to learn the parameters) on the dataset to predict output values for new input data. Our method can always recommend a model in 3 seconds, but training is still needed for prediction.

heading: 4 RESULTS

text:
Table 2 shows our results for a representative set of test datasets, comparing our approach with state-of-the-art AutoML systems and baselines. For each dataset (row), Table 2 reports the mean evaluation accuracy of different AutoML methods. Figure 3 compares the of accuracy on the test set between our zero-shot approach given 3 seconds of computation, and other AutoML systems and random forest baseline given 1 minute of computation. Our new zero-shot AutoML approach is the only AutoML system that provides predictions within 3 seconds. OBOE requires at least 20 seconds to perform predictions, and then only on a few of the datasets. AlphaD3M reaches performance slightly better than our approach, however given a minute of computation. See the supplementary
material for additional results which validate the performance of our method. First, our zero-shot method generally outperforms other simple baselines given the same amount of computation time. Second, our zero-shot method, in 3 seconds, gives results comparable to state-of-the-art AutoML systems given 1 minute.
7/23/20, 11:23 AMfigure_plot.ipynb - Colaboratory
Page 3 of 37https://colab.research.google.com/drive/1t0Gt8c_Tp3gYFnLhcQ4SBnkO2vED3qbA#scrollTo=m7vlAj6LRZEU
([<matplotlib.axis.YTick at 0x7fb9b29999b0>, <matplotlib.axis.YTick at 0x7fb9b2999390>, <matplotlib.axis.YTick at 0x7fb9b2997b00>, <matplotlib.axis.YTick at 0x7fb9b2942828>, <matplotlib.axis.YTick at 0x7fb9b294ada0>, <matplotlib.axis.YTick at 0x7fb9b294af60>], [Text(0, 0, '0'), Text(0, 0, '0.2'), Text(0, 0, '0.4'), Text(0, 0, '0.6'), Text(0, 0, '0.8'), Text(0, 0, '1.0')])
fig, axes = plt.subplots()
axes.scatter(res_sklearn, res_zero, c='tab:red', marker='o', s=150)
t = np.arange(0,1.05,0.01)
axes.plot(t,t,'k--')
axes.set_xlim(0.0,1.05)
axes.set_ylim(0.0,1.05) axes.set_xlabel('AutoSklearn (1 minute)',fontsize=18) axes.set_ylabel('Zero-Shot AutoML (3 seconds)',fontsize=18) plt.axis('square') plt.xticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0', '0.2', '0.4', '0.6', '0.8', '1.0']) plt.yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0', '0.2', '0.4', '0.6', '0.8', '1.0'])
fig, axes = plt.subplots()
7/23/20, 11:23 AMfigure_plot.ipynb - Colaboratory
Page 4 of 37https://colab.research.google.com/drive/1t0Gt8c_Tp3gYFnLhcQ4SBnkO2vED3qbA#scrollTo=m7vlAj6LRZEU
([<matplotlib.axis.YTick at 0x7fb9b28f0a58>, <matplotlib.axis.YTick at 0x7fb9b28f0438>, <matplotlib.axis.YTick at 0x7fb9b28f4ba8>, <matplotlib.axis.YTick at 0x7fb9b291ab38>, <matplotlib.axis.YTick at 0x7fb9b28a3a20>, <matplotlib.axis.YTick at 0x7fb9b28a3eb8>], [Text(0, 0, '0'), Text(0, 0, '0.2'), Text(0, 0, '0.4'), Text(0, 0, '0.6'), Text(0, 0, '0.8'), Text(0, 0, '1.0')])
fig, axes = plt.subplots()
axes.scatter(res_oboe, res_zero, c='tab:purple', marker='o', s=150)
t = np.arange(0,1.05,0.01)
axes.plot(t,t,'k--')
axes.set_xlim(0.0,1.05)
axes.set_ylim(0.0,1.05)
axes.set_xlabel('OBOE (1 minute)',fontsize=18) axes.set_ylabel('Zero-Shot AutoML (3 seconds)',fontsize=18) # axes[1].set_ylabel('Ours',fontsize=14) plt.axis('square') plt.xticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], [' , '0.2', '0.4', '0.6', '0.8', '1.0']) plt.yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], [' , '0.2', '0.4', '0.6', '0.8', '1.0'])
fig, axes = plt.subplots()
7/23/20, 11:23 AMfigure_plot.ipynb - Colaboratory
Page 5 of 37https://colab.research.g ogle.com/drive/1t0Gt8c_Tp3gYFnLhcQ4SBnkO2vED3qbA#scrollTo=m7vlAj6LRZEU
([<matplotlib.axis.YTick at 0x7fb9b28c88d0>, <matplotlib.axis.YTick at 0x7fb9b28c82b0>, <matplotlib.axis.YTick at 0x7fb9b28c7a20>, <matplotlib.axis.YTick at 0x7fb9b2872b00>, <matplotlib.axis.YTick at 0x7fb9b287b9e8>, <matplotlib.axis.YTick at 0x7fb9b287be80>], [Text(0, 0, '0'), Text(0, 0, '0.2'), Text(0, 0, '0.4'), Text(0, 0, '0.6'), Text(0, 0, '0.8'), Text(0, 0, '1.0')])
# axes[2].scatter(res_tpot, res_zero, c='k', marker='o', s=150)
fig, axes = plt.subplots()
axes.scatter(res_tpot, res_zero, c='tab:orange', marker='o', s=150)
t = np.arange(0,1.05,0.01)
axes.plot(t,t,'k--')
axes.set_xlim(0.0,1.05)
axes.set_ylim(0.0,1.05) axes.set_xlabel('TPOT (1 minute)',fontsize=18) axes.set_ylabel('Zero-Shot AutoML (3 seconds)',fontsize=18) plt.axis('square') plt xticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0', '0.2', '0.4', '0.6', '0.8', '1.0']) plt yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0', '0.2', '0.4', '0.6', '0.8', '1.0'])
7/23/20, 11:23 AMfigure_plot.ipynb - Colaboratory
Page 6 of 37https://colab.research.google.com/drive/1t0Gt8c_Tp3gYFnLhcQ4SBnkO2vED3qbA#scrollTo=m7vlAj6LRZEU
([<matplotlib.axis.YTick at 0x7fb9b2822898>, <matplotlib.axis.YTick at 0x7fb9b2822278>, <matplotlib.axis.YTick at 0x7fb9b28999e8>, <matplotlib.axis.YTick at 0x7fb9b2849ac8>, <matplotlib.axis.YTick at 0x7fb9b28539b0>, <matplotlib.axis.YTick at 0x7fb9b2853e48>], [Text(0, 0, '0'), Text(0, 0, '0.2'), Text(0, 0, '0.4'), Text(0, 0, '0.6'), Text(0, 0, '0.8'), Text(0, 0, '1.0')])
fig, axes = plt.subplots()
axes.scatter(res_d3m, res_zero, c='tab:blue', marker='o', s=150)
t = np.arange(0,1.05,0.01)
axes.plot(t,t,'k--')
axes.set_xlim(0.0,1.05)
axes.set_ylim(0.0,1.05) axes.set_xlabel('AlphaD3M (1 minute)',fontsize=18) axes.set_ylabel('Zero-Shot AutoML (3 seconds)',fontsize=18) plt.axis('square') plt xticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0', '0.2', '0.4', '0.6', '0.8', '1.0']) plt yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0', '0.2', '0.4', '0.6', '0.8', '1.0'])
7/23/20, 11:23 AMfigure_plot.ipynb - Colaboratory
Page 7 of 37https://colab.research.g ogle.com/drive/1t0Gt8c_Tp3gYFnLhcQ4SBnkO2vED3qbA#scrollTo=m7vlAj6LRZEU
([<matplotlib.axis.YTick at 0x7fb9b5c82198>, <matplotlib.axis.YTick at 0x7fb9b2ab3b38>, <matplotlib.axis.YTick at 0x7fb9b3d56390>, <m tplotlib.axis.YTick at 0x7fb9b41d18d0>, <matplotlib.axis.YTick at 0x7fb9b41d1358>, <matplotlib.axis.YTick at 0x7fb9b3821240>], [Text(0, 0, '0'), Text(0, 0, '0.2'), Text(0, 0, '0.4'), Text(0, 0, '0.6'), Text(0, 0, '0.8'), Text(0, 0, '1.0')])
fig, axes = plt.subplots()
axes.scatter(res_rf_m, res_zero, c='tab:green', marker='o', s=150)
t = np.arange(0,1.05,0.01)
axes.plot(t,t,'k--')
axes.set_xlim(0.0,1.05)
axes.set_ylim(0.0,1.05) axes.set_xlabel('Random Forest (1 minute)',fontsize=18) axes.set_ylabel('Zero-Shot AutoML (3 seconds)',fontsize=18) plt.axis('square') plt xticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], ['0', '0.2', '0.4', '0.6', '0.8', '1.0']) plt yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], [' , '0.2', '0.4', '0.6', '0.8', '1.0'])
plt.rcParams['xtick.labelsize'] = 18 plt.rcParams['ytick.labelsize'] = 18 fig, axes = plt.subplots(1,5,figsize=(30,5))Figure 3: Comparison of accuracy on test set between our zero-shot approach given 3 seconds of computa ion and other AutoML systems and Random For st baseline given 1 minute of computation. Our zero-shot approach matches the performance of baselines while running 20 times faster.

heading: 5 CONCLUSIONS AND FUTURE WORK

text: We introduce a new zero-shot approach to AutoML that is able to recommend a good pipeline to use for a given dataset in real-time. Our system builds a graph from both NLP text embedding of the dataset and pipeline descriptions as well as data meta-features and uses a graph neural network to predict the best pipeline for a given dataset. Our approach matches the performance of other state-of-the-art AutoML systems and is significantly faster, reducing running time from minutes to seconds and prediction time from minutes to milliseconds. Future work will extend our approach to handle different types of data, including audio and images. In addition, we envision an extension to semi-supervised AutoML by using a GNN to embed a large unsupervised set of datasets without pipelines, such as the 25 Million datasets available on Google dataset search (Brickley et al., 2019), together with a small supervised set of datasets with AutoML pipelines. Finally, we will make our data, models, and code public upon publication.