summaries: This paper makes two contributions: 1 - It presents a framework for measuring linear symmetry based disentanglement (LSBD). In particular, it addresses the challenge that LSBD is only defined with respect to an optimal decomposition of the latent space $Z = Z_1 \oplus \ldots \oplus Z_k$ that needs to be identified by the metric. 2 - The metric serves as additional supervision in the training of a VAE on toy datasets with at most two symmetry transformations. The results show quantitative improvements in terms of disentanglement. Summary:  This paper aims to operationalize the symmetry-based disentanglement idea proposed in Higgins et al., 2018, and proposes a novel loss function that can be used to both evaluate and learn disentangled representations (including learning from datasets where only a subset of samples are fully labeled). The paper includes some experimental validation of the proposed ideas on synthetic datasets. It also includes cross-comparisons between both the LSBD-based metric and method to state of the art disentangling metrics and methods, which demonstrate the complementary aspects of the LSBD method (e.g., its ability to capture multi-dimensional disentangled factors, for example rotations in a euclidean plane).  Overall, the paper is clearly written and makes a positive contribution in pushing for disentangling metrics that are motivated by a novel formalism. The impact of this paper is slightly limited due to the strength of supervision required (full supervision of all factors of variation) for implementing both the D_LSBD metric and the loss. This work focuses on developing a metric evaluation metric and a semi-supervised VAE-based model for linear symmetry-based disentanglement (LSBD)[1]. When assuming knowledge of linearly disentangled group representation $\rho$, the authors deduce that LSBD is fulfilled if the last requirement is met (equivariance of a map h). The authors propose to measure the equivariance by simply quantify the equivariance of the encoding map w.r.t. to the group and observations. Although a straightforward thought, computing the metric is challenging. Therefore, the authors propose an approximation by calculating the dispersion of the inverse group representation obtained through a PCA. Similar to the evaluation metric, the authors propose LSBD-VAE, a semi-supervised version of $\Delta$-VAE to regularise the latent space based on its dispersion. They show through empirical evaluation that their method outperforms other unsupervised disentanglement approaches w.r.t. to the proposed evaluation metric.  This paper proposes a new metric $D_{LSBD}$ to measure Linear Symmetry Based Disentanglement. The metric is motivated from an idealized setting before outlining a practical upper bound that can be realized. The authors also apply their new metric as an additional loss term to a $\Delta$-VAE model and show that it greatly enhances Linear Disentanglement while also agreeing with past notions of Disentanglement. The experiments, while simple and using only $SO(2)$ as a subgroup, are demonstrative of the proposed approach.
abstractText: The definition of Linear Symmetry-Based Disentanglement (LSBD) formalizes 1 the notion of linearly disentangled representations, but there is currently no metric 2 to quantify LSBD. Such a metric is crucial to evaluate LSBD methods and to 3 compare to previous understandings of disentanglement. We propose DLSBD, a 4 mathematically sound metric to quantify LSBD, and provide a practical implemen5 tation. Furthermore, from this metric we derive LSBD-VAE, a semi-supervised 6 method to learn LSBD representations. We demonstrate the utility of our metric 7 by showing that (1) common VAE-based disentanglement methods don’t learn 8 LSBD representations, (2) LSBD-VAE as well as other recent methods can learn 9 LSBD representations, needing only limited supervision on transformations, and 10 (3) various desirable properties expressed by existing disentanglement metrics are 11 also achieved by LSBD representations. 12
references.authors: H. 345 Caselles-Dupré M.G. Ortiz D. Filliat
references.title: Symmetry-based disentangled representation
references.authors: T.Q. Chen X. Li R.B. Grosse D.K. Duvenaud
references.title: Isolating sources of disentanglement
references.authors: T. Cohen M. Welling
references.title: Learning the irreducible representations of commutative Lie
references.authors: T.S. Cohen M. Welling
references.title: Transformation properties of learned visual representations
references.authors: C. 355 Eastwood C.K. Williams
references.title: A framework for the quantitative evaluation of disentangled
references.title: β-VAE: Learning basic visual concepts with a constrained variational framework
references.authors: H. Kim A. Mnih
references.title: Disentangling by factorising
references.venue: In International Conference on Machine
references.authors: D.P. Kingma M. Welling
references.title: Auto-Encoding Variational Bayes
references.venue: In International Conference
references.authors: A. Kumar P. Sattigeri A. Balakrishnan
references.title: Variational inference of disentangled
references.authors: S.A. 371 Nene S.K. Nayar H Murase
references.title: Columbia object image library (coil-20)
references.authors: M. Painter A. Prugel-Bennett J. Hare
references.title: Linear disentangled representations and
references.authors: L.A. Perez Rey V. Menkovski J. Portegies
references.title: Diffusion variational autoencoders
references.authors: R. 377 Quessard T.D. Barrett W.R. Clements
references.title: Learning Group Structure and Disentangled
references.authors: D.J. 380 Rezende S. Mohamed D. Wierstra
references.title: Stochastic backpropagation and approximate
references.authors: K. 383 Ridgeway M.C. Mozer
references.title: Learning deep disentangled embeddings with the f-statistic
references.authors: I. 385 Sosnovik M. Szmaja A. Smeulders
references.title: Scale-Equivariant Steerable Networks
references.authors: Z. Wu S. Song A. Khosla F. Yu L. Zhang X. Tang J. Xiao
references.title: 3D ShapeNets: A
sections.text: The definition of Linear Symmetry-Based Disentanglement (LSBD) formalizes1 the notion of linearly disentangled representations, but there is currently no metric2 to quantify LSBD. Such a metric is crucial to evaluate LSBD methods and to3 compare to previous understandings of disentanglement. We propose DLSBD, a4 mathematically sound metric to quantify LSBD, and provide a practical implemen-5 tation. Furthermore, from this metric we derive LSBD-VAE, a semi-supervised6 method to learn LSBD representations. We demonstrate the utility of our metric7 by showing that (1) common VAE-based disentanglement methods don’t learn8 LSBD representations, (2) LSBD-VAE as well as other recent methods can learn9 LSBD representations, needing only limited supervision on transformations, and10 (3) various desirable properties expressed by existing disentanglement metrics are11 also achieved by LSBD representations.12
  1 Introduction13
  Learning low-dimensional representations that disentangle the underlying factors of variation in data14 is considered an important step towards interpretable machine learning with good generalization. To15 address the fact that there is no consensus on what disentanglement entails and how to formalize it,16 Higgins et al. (2018) propose a formal definition for Linear Symmetry-Based Disentanglement, or17 LSBD, arguing that underlying real-world symmetries give exploitable structure to data.18
  However, there is currently no metric to quantify LSBD. Such a metric is crucial to properly evaluate19 methods aiming to learn LSBD representations and to relate LSBD to previous definitions of disentan-20 glement. Although previous works have evaluated LSBD by measuring performance on downstream21 tasks (Caselles-Dupré et al., 2019) or by measuring specific traits related to LSBD (Painter et al.,22 2020; Quessard et al., 2020), none of these evaluation methods directly quantify LSBD according to23 its well-formalized definition.24
  We propose DLSBD, a well-formalized and generally applicable metric that quantifies the level of25 LSBD in learned data representations. We show an intuitive justification of this metric, as well26 as its theoretical derivation. We also provide a practical implementation to compute DLSBD for27 common symmetry groups. Furthermore, we show that our metric formulation can be used to derive28 a semi-supervised method to learn LSBD representations, which we call LSBD-VAE. To make29 LSBD-VAE more widely applicable, we also demonstrate how to disentangle symmetric properties30 from other non-symmetric properties, and how to quantify this disentanglement with DLSBD.31 We show the utility of DLSBD by quantifying LSBD in a number of settings, for a variety of datasets32 with underlying SO(2) symmetries and other non-symmetric properties. First, we evaluate common33 VAE-based disentanglement methods and show that most don’t learn LSBD representations. Second,34 we evaluate LSBD-VAE and other recent methods that specifically target LSBD, showing that they35
  Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
  can obtain much better DLSBD scores while needing only limited supervision on transformations.36 Third, we compare DLSBD with existing disentanglement metrics, showing that various desirable37 properties expressed with these metrics are also achieved by LSBD representations.38
  2 Related Work39
  Plenty of works have focused on learning and quantifying disentangled representations recently, but40 research has shown that there is little consensus about the exact definition of disentanglement and41 methods often do not achieve it as well as they proclaim (Locatello et al., 2019). To introduce some42 much-needed formalization, Higgins et al. (2018) proposed to define disentanglement with respect43 to symmetry transformations acting on the data. They used group theory to provide two formal44 definitions, which we refer to as (Linear) Symmetry-Based Disentanglement, or (L)SBD. In this45 paper we focus only on LSBD, not SBD.46
  Several methods have been proposed to learn LSBD representations (Caselles-Dupré et al., 2019;47 Painter et al., 2020; Quessard et al., 2020). These methods also learn to represent the transformations48 acting on the input data, assuming various levels of supervision on these transformations. Other49 methods have previously focused on capturing transformations of the data outside the context of50 disentanglement as well (Cohen and Welling, 2015; Sosnovik et al., 2019; Worrall et al., 2017).51
  3 Linear Symmetry-Based Disentanglement52
  Higgins et al. (2018) provide a formal definition of linear disentanglement that connects symmetry53 transformations affecting the real world (from which data is observed) to the internal representations54 of a model. The definition is grounded in concepts from group theory, we provide a more detailed55 description of these concepts in the Supplementary Material.56
  The definition1 considers a group G of symmetry transformations acting on the data space X through57 the group action · : G ×X → X . In particular, G can be decomposed as the direct product of K58 groups G = G1 × . . .×GK . A model’s internal representation of data is modeled with the encoding59 function h : X → Z that maps data to the embedding space Z. The definition for Linearly Symmetry-60 Based Disentangled (LSBD) representations then formalizes the requirement that a model’s encoding61 h should reflect and disentangle the transformation properties of the data, and that the transformation62 properties of the model’s encoding should be linear. The exact definition is as follows:63
  Definition: Linear Symmetry-Based Disentanglement (LSBD) A model’s encoding map h :64 X → Z, where Z is a vector space, is LSBD with respect to the group decomposition G =65 G1 × . . .×GK if66
  1. there is a decomposition of the embedding space Z = Z1 ⊕ . . . ⊕ ZK into K vector67 subspaces,68
  2. there are group representations for each subgroup in the corresponding vector subspace69 ρk : Gk → GL(Zk), k ∈ {1, . . . ,K}70
  3. the group representation ρ : G→ GL(Z) acts on Z as71
  ρ(g) · z = (ρ1(g1) · z1, . . . , ρK(gK) · zK), (1)
  for g = (g1, . . . , gK) ∈ G and z = (z1, . . . , zK) ∈ Z with gk ∈ Gk and zk ∈ Zk.72 4. the map h is equivariant with respect to the actions of G on X and Z, i.e. , for all x ∈ X73 and g ∈ G it holds that h(g · x) = ρ(g) · h(x).74
  Furthermore, we say that a group representation ρ is linearly disentangled with respect to the group75 decomposition G = G1 × . . .×GK if it satisfies criteria 1 to 3 from the LSBD definition above.76
  1The original definition actually considers an additional set of world states W , but our definition is more practical and can be shown to be the same under mild conditions, see Supplementary Material.
  4 Quantifying LSBD: DLSBD77
  4.1 Intuition: Measuring Equivariance with Dispersion78
  Figure 1: A dataset of images from a rotating object expressed in terms of the group G = SO(2) acting on a base image x0. It is possible to quantify the level of LSBD of an encoding map h by measuring its equivariance with respect to a group representation ρ. Since all data has been generated from x0, equivariance can be measured as the dispersion of the points {ρ(g−1n ) · h(xn)}Nn=1.
  To motivate our metric, let’s first assume a set-79 ting in which a suitable linearly disentangled80 group representation ρ is known. Let’s further81 assume that the dataset of observations can be82 expressed with respect toG acting on some base83 point x0 ∈ X , i.e. {xn}Nn=1 = {gn · x0}Nn=1.84 Formally, this assumes that the action of G on85 X is regular. In this case, we can use the inverse86 group elements g−1n to transform each data point87 toward the base point x0, i.e.88
  x0 = g −1 1 · x1 = . . . = g −1 N · xN . (2)
  Since ρ is linearly disentangled, we only need to89 measure the equivariance of the encoding map90 h to quantify LSBD. Equivariance is achieved91 when h(g · x) = ρ(g) · h(x), for all g ∈ G, x ∈92 X . Given the dataset described above, we can93 check this property for x ∈ {xn}Nn=1 and g ∈94 {gn}Nn=1.2 In particular, from Equation (2) we95 can see that we have equivariance if96
  h(x0) = ρ(g −1 1 ) · h(x1) = . . . = ρ(g −1 N ) · h(xN ). (3)
  This not only characterizes perfect equivariance, but also allows for an efficient way to quantify how97 close we are to true equivariance, by measuring the dispersion of the points {ρ(g−1n ) · h(xn)}Nn=1.398 Given a suitable norm ‖ · ‖Z in Z, we can thus quantify LSBD in this setting as99
  1
  N N∑ n=1 ∥∥∥∥∥ρ(g−1n ) · h(xn)− 1N N∑ n′=1 ρ(g−1n′ ) · h(xn′) ∥∥∥∥∥ 2
  Z
  , (4)
  i.e. we compute the mean of {ρ(g−1n ) · h(xn)}Nn=1 and use the average squared distance to this mean100 for points in {ρ(g−1n ) · h(xn)}Nn=1 as our LSBD metric, see Figure 1.101 However, this formulation requires knowing the right linearly disentangled group representation and102 a suitable norm in Z. Moreover, it implicitly assumes a uniform probability measure over the group103 elements {gn}Nn=1. In the next section we formulate our metric for a more general setting.104
  4.2 DLSBD: A Metric for LSBD105
  Generalizing the ideas from the previous section with concepts from measure theory, we propose a106 metric to measure the level of LSBD of any encoding h : X → Z given a data probability measure µ107 on X , provided that µ can be written as the pushforward GX(·, x0)#ν of some probability measure108 ν on G by the function GX(·, x0) for some base point x0. More formally,109
  µ(A) = GX(·, x0)#ν(A) = ν ({g ∈ G | GX(g, x0) ∈ A}) , (5) for Borel subsets A ⊂ X . Note that this is only possible if the action GX is transitive.110 For example, the situation of a dataset with N datapoints {xn}Nn=1 = {gn · x0}Nn=1 corresponds to111 the case in which ν and µ are empirical measures on the group G and data space X , respectively:112
  ν := 1
  N N∑ i=1 δgi , µ := 1 N N∑ i=1 δxi . (6)
  2Note that {gn}Nn=1 can be used to describe all known group transformations between elements in the dataset by means of composition and inverses, since xi = gi · (g−1j · xj). Thus it suffices to check equivariance for these N group transformations.
  3Note that we do not actually need to know x0 nor h(x0).
  We define the metric DLSBD for an encoding h and a measure µ as113
  DLSBD := inf ρ∈P(G,Z) ∫ G ∥∥ρ(g)−1 · h(g · x0)−Mρ,h,x0∥∥2ρ,h,µ dν(g), with Mρ,h,x0 = ∫ G ρ(g′)−1 · h(g′ · x0)dν(g′), (7)
  where the norm ‖·‖ρ,h,µ is a Hilbert-space norm depending on the representation ρ, the encoding map114 h : X → Z, and the data measure µ. More details of this norm can be found in the Supplementary115 Material. Moreover, P(G,Z) denotes the set of linearly disentangled representations of G in Z.116 Lower values of DLSBD indicate better disentanglement, zero being optimal.117
  4.3 Practical Computation of DLSBD118
  There are two main challenges for computing the metric of Equation (7). First, to calculate the119 integrals in the formula, all possible datapoints that can be expressed as g · x0 with g ∈ G =120 G1 × · · · × GK must be available. Second, the infimum of the integrals over all possible linearly121 disentangled representations must be estimated. This requires finding the possible invariant subspaces122 Z = Z1⊕· · ·⊕ZK induced by the encoding h over which the group representations are disentangled.123 We present a practical implementation of an upper bound to DLSBD for an encoding function h given124 a dataset X generated by some known group transformations. In particular, this approximation of125 DLSBD is designed for a group decomposition G = G1 × · · · ×GK where each Gk = SO(Dk) with126 k ∈ {1, . . . ,K} the group of rotations in Dk dimensions. This implementation approximates the127 integrals of Equation (7) by using the empirical distribution of X . The invariant subspaces of Z to the128 subgroup actions are found by applying a suitable change of basis. In the new basis, the disentangled129 group representations are expressed in a parametric form whose parameters are optimized to find the130 tightest bound to DLSBD. See Figure 2 for an intuitive description of the process.131 Assume there is a datasetX that can be modeled in terms of the group decompositionG = G1×· · ·Gk.132 For each Gk subgroup there is a set of known group elements Gk ⊆ Gk uniformly sampled such133 that the dataset is described in terms of all elements in G = G1 × · · · × GK and a base point x0 as134 X = {(g1, . . . , gK) · x0|gk ∈ Gk, k ∈ {1, . . . ,K}} .135 For each subgroup Gk we construct a set of encoded data Zk ⊆ Z whose variability should only de-136 pend on the action of Gk. The set Zk is given by Zk = {zk(g1, . . . , gK)|gj ∈ Gj , j ∈ {1, . . . ,K}},137 in which138
  zk(g1, . . . , gK) = h((g1, . . . , gK) · x0)− 1 |Gk| ∑ g′∈Gk h((g1, . . . , gk−1, g ′, gk+1, . . . , gK) · x0). (8)
  Similar to (Cohen and Welling, 2014), we find a suitable change of basis that exposes the invariant139 subspace Zk corresponding to the k-th subgroup Gk. The new basis is obtained from the eigenvectors140 resulting from applying Principal Component Analysis (PCA) to Zk. Each element in Zk is projected141 into the first Dk eigenvectors. The new set is denoted as Z ′k ⊆ RDk with elements z′k(g1, . . . , gK) ⊆142 RDk that are the projected versions of zk(g1, . . . , gK).143 (Quessard et al., 2020) describes how one could parameterize the subgroup representations of SO(Dk)144 for arbitrary Dk but here we will focus on Gk = SO(2). In this case, we can parameterize each145 subgroup representation in terms of a single integer parameter ω ∈ Z as ρk,ω(gk) corresponding146 to a 2 × 2 rotation matrix whose angle of rotation is ω multiplied by the known angle associated147 to the group element gk ∈ Gk = SO(2). For this subgroup we can approximate the Mρ,h,x0 from148 Equation (7) as Mk,ω given by149
  Mk,ω = 1 |G| ∑
  (g1,...,gK)∈G
  ρk,ω(g −1 k ) · z ′(g1, . . . , gK). (9)
  Similar to Equation (7) we would like to find the optimal ρk,ω that minimizes the integral over the150 group representations. We can define a parameter search space Ω ⊆ Z, e.g. Ω = [−10, 10] for finding151 the optimal ω ∈ Ω that minimizes the dispersion, this is expressed in the following equation152
  D(k)LSBD = min ω∈Ω
  1 |G| ∑
  (g1,...,gK)∈G
  ‖ρk,ω(g−1k ) · z ′(g1, . . . , gK)−Mk,ω‖2. (10)
  EachD(k)LSBD measures the degree of equivariance of the projected embeddings for each k-th subgroup153 corresponding to the best fitting group representation. The upper bound to the metric is finally obtained154 by averaging across all subgroups DLSBD ≤ 1K ∑K k=1D (k) LSBD.155
  5 Learning LSBD Representations: LSBD-VAE156
  In this section we present LSBD-VAE, a semi-supervised VAE-based method to learn LSBD repre-157 sentations. The main idea is to train an unsupervised Variational Autoencoder (VAE) (Kingma and158 Welling, 2014; Rezende et al., 2014) with a suitable latent space topology, and use our metric as an159 additional loss term for batches of transformation-labeled data.160
  Assumptions LSBD-VAE requires some knowledge about the group structure G that is to be161 disentangled. Concretely, the group and its decomposition G = G1 × . . .×GK should be known,162 as well as a suitable linearly disentangled group representation ρ : G→ GL(Z) and a latent space163 Z = Z1 ⊕ . . . ⊕ ZK . Moreover, we assume there exists an embedded submanifold ZG ⊆ Z such164 that the action of G on Z restricted to ZG is regular, and ZG is invariant under the action. Only ZG165 will then be used as the codomain for the encoding map, h : X → ZG.166 We demonstrate the assumptions above for the common group structure G = SO(2)× SO(2). For167 the group representation ρ = ρ1 ⊕ ρ2, with Z = R2 ⊕ R2, we can use rotation matrices in R2 for168 ρ1 and ρ2. We can then use 1-spheres S1 = {z ∈ R2 : ‖z‖ = 1} for the embedded submanifold:169 ZG = S
  1 × S1. In this case, the action of G on Z restricted to ZG is indeed regular, and ZG is170 invariant under the action.171
  Unsupervised Learning on Latent Manifold To learn encodings only on the latent manifold172 ZG, we use a Diffusion Variational Autoencoder (∆VAE) (Perez Rey et al., 2020). ∆VAEs can173 use any closed Riemannian manifold embedded in a Euclidean space as a latent space (or latent174 manifold), provided that a certain projection function from the Euclidean embedding space into the175 latent manifold is known and the scalar curvature of the manifold is available. The ∆VAE uses176 a parametric family of posterior approximates obtained from a diffusion process over the latent177 manifold. To estimate the intractable terms of the negative ELBO, the reparameterization trick is178 implemented via a random walk.179
  In the case of S1 as a latent (sub)manifold, we consider R2 as the Euclidean embedding space, and180 the projection function4 Π : R2 → S1 normalizes points in the embedding space: Π(z) = z/|z|. The181 scalar curvature of S1 is 0.182
  4This projection function is not defined for z = 0, but this value does not occur in practice.
  Semi-Supervised Learning with Transformation Labels Caselles-Dupré et al. (2019) proved183 that LSBD representations cannot be inferred from a training set of unlabeled observations, but that184 access to the transformations between data points is needed. They therefore use a training set of185 observation pairs with a given transformation between them.186
  However, we posit that only a limited amount of supervision is sufficient. Since obtaining supervision187 on transformations is typically more expensive than obtaining unsupervised observations, it is188 desirable to limit the amount of supervision needed.189
  Therefore, we augment the un-190 supervised ∆VAE with a super-191 vised method that makes use of192 transformation-labeled batches, i.e.193 batches {xm}Mm=1 such that xm =194 gm · x1 for m = 2, . . . ,M , where195 the transformations gm (and thus196 their group representations ρ(gm)) are197 known and are referred to as transfor-198 mation labels. The simplified version199 of the metric from Equation (4) can200 then be used for each batch as an ad-201 ditional loss term (with x0 = x1), as202 it is differentiable under the assump-203 tions described above (using the Eu-204 clidean norm).205
  We make a small adjustment to Equation (4) for the purpose of our method, since the mean computed206 there does not typically lie on the latent manifold ZG. Thus, we use the projection Π from the ∆VAE207 to project the mean onto ZG. Writing the encodings as zm := h(xm), the additional loss term for a208 transformation-labeled batch {xm}Mm=1 then becomes209
  LLSBD = 1
  M M∑ m=1 ∥∥∥∥∥ρ(g−1m ) · zm −Π ( 1 M M∑ m=1 ρ(g−1m ) · zm )∥∥∥∥∥ 2 , (11)
  where g1 = e, the group identity.210 Moreover, instead of feeding the encodings zm to the decoder, we use ρ(gm) · z, where z =211 Π (
  1 M ∑M m=1 ρ(g −1 m ) · zm ) . This encourages the decoder to follow the required group structure.212 This only affects the reconstruction loss component of the ∆VAE.213
  Figure 3 illustrates the supervised part of our method for a transformation-labeled batch {xm}Mm=1.214 The loss function is the regular ELBO (but with adjusted decoder input as described above) as used215 in ∆VAE plus an additional term γ · LLSBD, where γ is a weight hyperparameter to control the216 influence of the supervised loss component. By alternating unsupervised and supervised training217 (using the same encoder and decoder), we have a method that makes use of both unlabeled and218 transformation-labeled observations.219
  6 Experimental Setup220
  We evaluate the disentanglement of several models on three different image datasets (Square, Arrow,221 and Airplane) with a known group decomposition G = SO(2)× SO(2) describing the underlying222 transformations. For each subgroup a fixed number of |Gk| = 64 with k ∈ {1, 2} transformations is223 selected. The datasets exemplify different group actions of SO(2): periodic translations, in-plane224 rotations, out-of-plane rotations, and periodic hue-shifts.225
  In real settings, not all variability in the data can be modelled by the actions of a group. Therefore,226 we also evaluate the same models on two datasets ModelNet40 (Wu et al., 2014) and COIL-100227 (Nene et al., 1996) that consist of images from various objects (i.e. non-symmetric variation) under228 known out-of-plane rotations (SO(2) symmetries). In many settings it is easy to obtain labels for229 such rotations, e.g. when the camera or object angle is controlled by an agent. See Figure 4 for230 examples of the datasets. For more details, see the Supplementary Material.231
  For the Square, Arrow, and Airplane datasets we test LSBD-VAE with transformation-labeled batches232 of size M = 2. More specifically, for each experiment we randomly select L disjoint pairs of data233 points, and label the transformation between the data points in each pair. We vary the number of234 labeled pairs L from 0 (corresponding to a ∆VAE) to N/2 (in which case each data point is involved235 in exactly one labeled pair). We set the weight γ of the supervised loss component to γ = 100236 for all experiments. We choose M = 2 for our experiments since it is the most limited setting for237 LSBD-VAE. Higher values of M would provide stronger supervision, so successful results with238 M = 2 imply that good results can also be achieved for higher values of M (but not necessarily vice239 versa).240
  For the COIL-100 and ModelNet40 datasets, we train LSBD-VAE on batches containing images of241 one particular object from all different angles (72 and 64 for COIL-100 and ModelNet40, respectively).242 Each batch is labelled with transformations (g1, e), . . . , (gM , e), where gm represent rotations, and243 the unit transformation e indicates that the object is unchanged. To represent the rotations we use244 a S1 latent space as in ∆VAE, whereas for the object identity we use a 5-dimensional Euclidean245 space with standard Gaussian prior as in regular VAEs. LSBD is measured as the disentanglement of246 rotations in the latent space. For these experiments we used γ = 1.247
  We furthermore test a number of known disentanglement methods for comparison, including tra-248 ditional disentanglement methods as well as methods focusing on LSBD. In particular, we use249 disentanglement_lib (Locatello et al., 2019) to train a regular VAE (Kingma and Welling, 2014;250 Rezende et al., 2014), β-VAE (Higgins et al., 2017), CC-VAE (Burgess et al., 2018), FactorVAE251 (Kim and Mnih, 2018), and DIP-VAE-I/II (Kumar et al., 2018). Furthermore we evaluate the method252 from Quessard et al. (2020) that focuses on LSBD. We also tested ForwardVAE (Caselles-Dupré253 et al., 2019), but show only limited results since we were not able to reproduce any reasonable results254 for our datasets.255
  We use encodings from all these methods to evaluate DLSBD, as well as common traditional disentan-256 glement metrics from disentanglement_lib: Beta (Higgins et al., 2017), Factor (Kim and Mnih,257 2018), SAP (Kumar et al., 2018), DCI Disentanglement (Eastwood and Williams, 2018), Mutual258 Information Gap (MIG) (Chen et al., 2018), and Modularity (MOD) (Ridgeway and Mozer, 2018).259
  More information about the architecture, epochs and hyperparameters can be found in the Supplemen-260 tary Material. For the traditional disentanglement methods trained on Square, Arrow and Airplane261 datasets the latent spaces have 4 dimensions, since these are the minimum number of dimensions262 necessary to learn LSBD representations for an underlying SO(2) × SO(2) symmetry group, see263 (Higgins et al., 2018; Caselles-Dupré et al., 2019). For COIL-100 and ModelNet40 we use latent264 spaces with 7 dimensions for a fair comparison with the LSBD-VAE method.265
  7 Results: Evaluating LSBD with DLSBD266
  We now highlight four key observations from our experimental results. In particular, we differentiate267 between the methods (VAE, β-VAE, CC-VAE, FACTOR, DIP-I, DIP-II) and metrics (BETA,268 FACTOR, SAP, DCI, MIG, MOD) that approach disentanglement in the traditional sense, and269 methods (∆VAE, QUESSARD, LSBD-VAE) and metric (DLSBD) that focus specifically on LSBD.270 The full quantitative results can be found in the Supplementary Material.271
  7.1 Standard Disentanglement Methods Don’t Learn LSBD Representations272
  Figure 5 summarizes the DLSBD scores (lower is better) for all methods on all datasets. Bars show273 the mean scores over 10 runs for each method, the vertical lines represent standard deviations.274 LSBD-VAE/L indicates our method trained on L labelled pairs (LSBD-VAE/0 corresponds to the275 unsupervised ∆VAE), LSBD-VAE/full indicates our method trained on batches containing a single276 object in all known transformations (for datasets with non-symmetric variation). Note that LSBD-277 VAE obtained very good scores (nearly 0) on the Arrow and Pixel datasets, hence the missing278 bars.279
sections.heading: LSB
sections.text: None of the traditional disentanglement methods achieve good DLSBD scores, even if they score well280 on other traditional disentanglement metrics. This implies that LSBD isn’t achieved by traditional281 methods. Moreover, from the full results in the Supplementary Material we see that the traditional282 methods on these datasets do not achieve good scores on all traditional metrics. In particular, SAP,283 DCI, and MIG scores are low. We believe this is a result of the cyclic nature of the symmetries284 underlying our datasets, further emphasizing the need for disentanglement methods that can capture285 such symmetries.286
  The SAP and MIG scores measure to what extent generative factors are disentangled into a single287 latent dimension. However, since the factors in our dataset are inherently cyclic due to their symmetry288 structure, they cannot be properly represented in a single latent dimension, as shown by Perez Rey289 et al. (2020). Instead, at least two dimensions are needed to continuously represent each cyclic factor290 in our data. A similar conclusion was made by Caselles-Dupré et al. (2019) and Painter et al. (2020).291
  DCI disentanglement measures whether a latent dimension captures at most one generative factor.292 This is accomplished by measuring the importance of each latent dimension in predicting the293 true generative factor using boosted trees. However, since the generative factors are cyclic, the294 performance of the boosted tree classifiers is far from optimal, thus providing more importance to295 several dimensions in predicting the generative factors and giving overall lower DCI scores.296
  7.2 LSBD-VAE and other LSBD Methods Can Learn LSBD Representations with Limited297 Supervision on Transformations298
  From Figure 5 we observe that methods focusing specifically on LSBD can score higher on DLSBD,299 showing that they are indeed more suitable to learn LSBD representations. In particular, LSBD-VAE300 got very good DLSBD scores for all datasets. Moreover, our experiments on the Arrow, Airplane, and301 Pixel datasets also show that only limited supervision suffices to obtain good DLSBD scores with low302 variability.303
  We only partially managed to reproduce the results from Quessard et al. (2020) on our datasets. Their304 method scored fairly well on the Airplane, ModelNet40, and COIL-100 datasets, but did not do well305 on the Square and Arrow dataset in our experiments.306
  Furthermore, we tested ForwardVAE by Caselles-Dupré et al. (2019), but we did not manage307 to produce any reasonable results on our datasets, trying both their original architecture and the308
  architecture we used for our other experiments. Therefore, we do not include scores for this method.309 We did however manage to reproduce ForwardVAE’s results on the Flatland dataset, which was used310 in their paper. For those experiments, we computed a mean DLSBD score of 0.012 with standard311 deviation 0.001 over 10 runs, indicating that ForwardVAE indeed learns LSBD representations for312 Flatland.313
  7.3 LSBD Representations Also Satisfy Previous Disentanglement Notions314
  Our results also indicate that LSBD captures various desirable properties that are expressed by315 traditional disentanglement metrics. In Figure 6 we compare DLSBD scores with scores for previous316 disentanglement metrics, for all our experiments. Note that for DLSBD lower is better, whereas for317 all other metrics higher is better. As we noted before, good scores on traditional disentanglement318 metrics don’t necessarily imply good DLSBD scores. Conversely however, methods that score well319 on DLSBD also score well on many traditional disentanglement metrics, often even outperforming320 the traditional methods. In particular, from the full results (see Supplementary Material) we see321 that LSBD-VAE matches or outperforms the traditional methods on the BETA, FACTOR and MOD322 metrics, and achieves much better scores for the DCI metric where traditional methods scored poorly.323
  The MIG and SAP scores are still low for methods focusing on LSBD. This is expected however, as324 explained earlier in Section 7.1. This was also observed by Painter et al. (2020) for different datasets.325
  8 Conclusion326
  We presented DLSBD, a metric to quantify Linear Symmetry-Based Disentanglement (LSBD) as327 defined by Higgins et al. (2018). We further used this metric formulation to motivate LSBD-VAE,328 a semi-supervised method to learn LSBD representations given some expert knowledge on the329 underlying group symmetries that are to be disentangled.330
  We used DLSBD to evaluate various disentanglement methods, both traditional methods and recent331 methods that specifically focus on LSBD, and showed that LSBD-VAE can learn LSBD representa-332 tions where traditional methods fail to do so. We also comparedDLSBD to traditional disentanglement333 metrics, showing that LSBD captures many of the same desirable properties that are expressed by334 existing disentanglement methods. Conversely, we also showed that traditional disentanglement335 methods and metrics do not usually achieve or measure LSBD.336
  Challenges that remain are expanding and testing LSBD-VAE and DLSBD on different group struc-337 tures, towards more practical applications, as well as focusing on the utility of LSBD representations338 for downstream tasks.339
  Broader Impact The work is fairly theoretical, and practical methods derived from this work have340 no obvious negative societal impact. However, the ideas presented are relevant to representation341 learning and could be, in particular, used in computer vision and agent control applications.342
  References343 Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. (2018).344 Understanding disentangling in β-VAE. arXiv preprint arXiv:1804.03599.345
  Caselles-Dupré, H., Ortiz, M. G., and Filliat, D. (2019). Symmetry-based disentangled representation346 learning requires interaction with environments. In Advances in Neural Information Processing347 Systems, pages 4606–4615.348
  Chen, T. Q., Li, X., Grosse, R. B., and Duvenaud, D. K. (2018). Isolating sources of disentanglement349 in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2615–350 2625.351
  Cohen, T. and Welling, M. (2014). Learning the irreducible representations of commutative Lie352 groups. 31st International Conference on Machine Learning, pages 3757–3770.353
  Cohen, T. S. and Welling, M. (2015). Transformation properties of learned visual representations. In354 3rd International Conference on Learning Representations.355
  Eastwood, C. and Williams, C. K. (2018). A framework for the quantitative evaluation of disentangled356 representations. In International Conference on Learning Representations.357
  Higgins, I., Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., and Lerchner, A. (2018).358 Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230.359
  Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner,360 A. (2017). β-VAE: Learning basic visual concepts with a constrained variational framework. In361 International Conference on Learning Representations.362
  Kim, H. and Mnih, A. (2018). Disentangling by factorising. In International Conference on Machine363 Learning, pages 2649–2658.364
  Kingma, D. P. and Welling, M. (2014). Auto-Encoding Variational Bayes. In International Conference365 on Learning Representations.366
  Kumar, A., Sattigeri, P., and Balakrishnan, A. (2018). Variational inference of disentangled latent367 concepts from unlabeled observations. In International Conference on Learning Representations.368
  Locatello, F., Bauer, S., Lucic, M., Gelly, S., Schölkopf, B., and Bachem, O. (2019). Challenging369 common assumptions in the unsupervised learning of disentangled representations. In International370 Conference on Machine Learning.371
  Nene, S. A., Nayar, S. K., Murase, H., et al. (1996). Columbia object image library (coil-20).372
  Painter, M., Prugel-Bennett, A., and Hare, J. (2020). Linear disentangled representations and373 unsupervised action estimation. Advances in Neural Information Processing Systems, 33.374
  Perez Rey, L. A., Menkovski, V., and Portegies, J. (2020). Diffusion variational autoencoders. In375 Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20,376 pages 2704–2710.377
  Quessard, R., Barrett, T. D., and Clements, W. R. (2020). Learning Group Structure and Disentangled378 Representations of Dynamical Environments. Advances in Neural Information Processing Systems,379 33.380
  Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate381 inference in deep generative models. In Proceedings of the 31st International Conference on382 International Conference on Machine Learning-Volume 32.383
  Ridgeway, K. and Mozer, M. C. (2018). Learning deep disentangled embeddings with the f-statistic384 loss. In Advances in Neural Information Processing Systems, pages 185–194.385
  Sosnovik, I., Szmaja, M., and Smeulders, A. (2019). Scale-Equivariant Steerable Networks. Interna-386 tional Conference on Learning Representations, pages 1–14.387
  Worrall, D. E., Garbin, S. J., Turmukhambetov, D., and Brostow, G. J. (2017). Harmonic networks:388 Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer389 Vision and Pattern Recognition, pages 5028–5037.390
  Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. (2014). 3D ShapeNets: A391 Deep Representation for Volumetric Shapes.392
  Checklist393
  1. For all authors...394 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s395 contributions and scope? [Yes]396 (b) Did you describe the limitations of your work? [Yes]397 (c) Did you discuss any potential negative societal impacts of your work? [Yes] See398 Section 8399 (d) Have you read the ethics review guidelines and ensured that your paper conforms to400 them? [Yes]401 2. If you are including theoretical results...402
  (a) Did you state the full set of assumptions of all theoretical results? [Yes]403 (b) Did you include complete proofs of all theoretical results? [N/A] The theoretical404 contributions are new definitions and methodology, these have been motivated but do405 not require proofs.406
  3. If you ran experiments...407 (a) Did you include the code, data, and instructions needed to reproduce the main experi-408 mental results (either in the supplemental material or as a URL)? [Yes] In Supplemen-409 tary Material410
  (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they411 were chosen)? [Yes] In paper or Supplementary Material412
  (c) Did you report error bars (e.g., with respect to the random seed after running experi-413 ments multiple times)? [Yes] Each experiment was run 10 times, we report means and414 standard deviation, and show error bars where needed.415
  (d) Did you include the total amount of compute and the type of resources used (e.g., type416 of GPUs, internal cluster, or cloud provider)? [Yes] In Supplementary Material we417 described the hardware. Tracked time of training for experiments with LSBD-VAE418 and Quessard approach, also reported in the Supplementary Material. The times for419 training the traditional methods in disentanglement_lib were not measured.420
  4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...421 (a) If your work uses existing assets, did you cite the creators? [Yes]422 (b) Did you mention the license of the assets? [Yes] In Supplementary Material423 (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]424 Included code in the Supplementary Material425 (d) Did you discuss whether and how consent was obtained from people whose data you’re426 using/curating? [N/A] No consent was needed.427 (e) Did you discuss whether the data you are using/curating contains personally identifiable428 information or offensive content? [N/A] It clearly doesn’t.429 5. If you used crowdsourcing or conducted research with human subjects...430
  (a) Did you include the full text of instructions given to participants and screenshots, if431 applicable? [N/A]432
  (b) Did you describe any potential participant risks, with links to Institutional Review433 Board (IRB) approvals, if applicable? [N/A]434
  (c) Did you include the estimated hourly wage paid to participants and the total amount435 spent on participant compensation? [N/A]436
title: Quantifying and Learning Linear Symmetry-Based Disentanglement