summaries: The paper proposes applying the theory of independent causal mechanisms (ICM), from causal discovery, to nonlinear ICA approaches to the blind source separation problem (BSS).  ICM is based on the idea that variables in a system are algorithmically independent or do not share information with each other. The paper shows, however, that ICM is not sufficient for BSS as spurious solutions exist which are not equivalent to the truth.  To address these difficulties, the paper proposes a new condition for identifiability which places an orthogonality condition on the columns of the Jacobian and provides information theoretic and geometric interpretations. The paper then shows theoretically that a large class of spurious solutions are not admitted under the IMA formulation.   Experiments on toy examples demonstrate that the model is identifiable under these spurious solutions.

summaries: This work proposes a new regularization scheme to improve identifiability in the _nonlinear blind source separation problem_ (nonlinear ICA) inspired by the notion of _independence of mechanism_ coming from causal inference. The idea is to consider only mixing functions which have a Jacobian matrix with orthogonal columns everywhere. They show how this extra assumption allows to exclude two classical types of degeneracies in nonlinear ICA, although without showing identifiability. The theoretical claims are validated by synthetic experiments.

summaries: The paper proposes a criterion for non-linear ICA as applied to Blind Source Separation inspired by Independence of Causal Mechanisms in the causal discovery literature. The general idea is to constrain the gradients of the output with respect to the input sources to be orthogonal to each other; this is linked to the notion that the mixing mechanism is in some sense independent (or not "too fine-tuned") to the input sources. The authors show that this constraint eliminates a number of known counterexamples to identifiability in non-linear ICA, and test it experimentally to show that helps to characterize true solutions (both as an evaluation metric, and as a regularizer). The authors conclude with a discussion putting their contributions into context.

summaries: The authors propose to borrow concepts from causality, in particular "independent causal mechanisms" , to propose a novel framework termed "independent mechanism analysis" (IMA), which can provide non-spurious solutions to nonlinear blind source separation. IMA can be seen as a restricted nonlinear ICA model, where the contributions of the latent variables $z_i$ to the nonlinear mixing $\mathbf{f}$ are independent, i.e. the columns of the Jacobian $J_\mathbf{f}$ are orthogonal.

summaries: The paper aims at the identifiability of nonlinear ICA. It proposes a principle as a constraint of mixture functions based on the modified independent causal mechanism principle in the causal discovery literature; investigates the identifiability of nonlinear ICA under such an assumption; then applies it to nonlinear ICA. It also shows that the identifiability results can cover some non-identifiable cases in the existed literature.

abstractText: Independent component analysis provides a principled framework for unsupervised representation learning, with solid theory on the identifiability of the latent code that generated the data, given only observations of mixtures thereof. Unfortunately, when the mixing is nonlinear, the model is provably nonidentifiable, since statistical independence alone does not sufficiently constrain the problem. Identifiability can be recovered in settings where additional, typically observed variables are included in the generative process. We investigate an alternative path and consider instead including assumptions reflecting the principle of independent causal mechanisms exploited in the field of causality. Specifically, our approach is motivated by thinking of each source as independently influencing the mixing process. This gives rise to a framework which we term independent mechanism analysis. We provide theoretical and empirical evidence that our approach circumvents a number of nonidentifiability issues arising in nonlinear blind source separation.

authors:
Pierre Ablin
Jean-François Cardoso
Alexandre Gramfort

title: Faster independent component analysis by preconditioning with hessian approximations

venue: IEEE Transactions on Signal Processing,

authors:
Khaled Alyani
Marco Congedo
Maher Moakher

title: Diagonality measures of Hermitian positivedefinite matrices with application to the approximate joint diagonalization problem

venue: Linear Algebra and its Applications,

authors:
Shun-ichi Amari
Hiroshi Nagaoka

title: Methods of information geometry, volume 191

venue: American Mathematical Soc.,

authors:
Shun-ichi Amari
Andrzej Cichocki
Howard Hua Yang

title: A new learning algorithm for blind signal separation

venue: In Advances in Neural Information Processing Systems,

authors:
Martin Arjovsky
Léon Bottou
Ishaan Gulrajani
David Lopez-Paz

title: Invariant risk minimization

venue: arXiv preprint arXiv:1907.02893,

authors:
Elias Bareinboim
Andrew Forney
Judea Pearl

title: Bandits with unobserved confounders: A causal approach

venue: Advances in Neural Information Processing Systems,

authors:
Yoshua Bengio
Aaron Courville
Pascal Vincent

title: Representation learning: A review and new perspectives

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
M Besserve
N Shajarisales
B Schölkopf
D Janzing

title: Group invariance principles for causal generative models

venue: In 21st International Conference on Artificial Intelligence and Statistics (AISTATS

authors:
M Besserve
A Mehrjou
R Sun
B Schölkopf

title: Counterfactuals uncover the modular structure of deep generative models

venue: In Eighth International Conference on Learning Representations (ICLR

authors:
Patrick Blöbaum
Dominik Janzing
Takashi Washio
Shohei Shimizu
Bernhard Schölkopf

title: Causeeffect inference by comparing regression errors

venue: In International Conference on Artificial Intelligence and Statistics,

authors:
James Bradbury
Roy Frostig
Peter Hawkins
Matthew James Johnson
Chris Leary
Dougal Maclaurin
Skye Wanderman-Milne

title: JAX: composable transformations of Python+NumPy programs, 2018

authors:
Jake Bruce
David Budden
Matteo Hessel
George Papamakarios
Francisco Ruiz

title: Distrax: Probability distributions in JAX

authors:
Lars Buesing
Theophane Weber
Yori Zwols
Sebastien Racaniere
Arthur Guez
Jean-Baptiste Lespiau
Nicolas Heess

title: Woulda, coulda, shoulda: Counterfactually-guided policy search

venue: arXiv preprint arXiv:1811.06272,

title: The three easy routes to independent component analysis; contrasts and geometry

venue: In Proc. ICA,

authors:
Ricky T.Q. Chen
Jens Behrmann
David Duvenaud
Jörn-Henrik Jacobsen

title: Residual flows for invertible generative modeling

title: Independent component analysis, a new concept

venue: Signal processing,

authors:
Povilas Daniušis
Dominik Janzing
Joris Mooij
Jakob Zscheischler
Bastian Steudel
Kun Zhang
Bernhard Schölkopf

title: Inferring deterministic causal relations

venue: In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence,

title: Leçons sur les systemes orthogonaux et les coordonnées curvilignes

title: Analyse des liaisons de probabilité

venue: In Proc. Int. Stat. Conferences

title: Analyse générale des liaisons stochastiques: etude particulière de l’analyse factorielle linéaire

venue: Revue de l’Institut international de statistique,

authors:
Andrew Forney
Judea Pearl
Elias Bareinboim

title: Counterfactual data-fusion for online reinforcement learners

authors:
Mingming Gong
Kun Zhang
Tongliang Liu
Dacheng Tao
Clark Glymour
Bernhard Schölkopf

title: Domain adaptation with conditional transferable components

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Alexander N Gorban
Ivan Yu Tyukin

title: Blessing of dimensionality: mathematical foundations of the statistical physics of data

venue: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences,

authors:
Anirudh Goyal
Alex Lamb
Jordan Hoffmann
Shagun Sodhani
Sergey Levine
Yoshua Bengio
Bernhard Schölkopf

title: Recurrent independent mechanisms

venue: In 9th International Conference on Learning Representations,

authors:
Daniel Greenfeld
Uri Shalit

title: Robust learning with the hilbert-schmidt independence criterion

authors:
Luigi Gresele
Paul K Rubenstein
Arash Mehrjou
Francesco Locatello
Bernhard Schölkopf

title: The Incomplete Rosetta Stone problem: Identifiability results for multi-view nonlinear ICA

venue: In Uncertainty in Artificial Intelligence,

authors:
Luigi Gresele
Giancarlo Fissore
Adrián Javaloy
Bernhard Schölkopf
Aapo Hyvarinen

title: Relative gradient optimization of the Jacobian term in unsupervised deep learning

authors:
Hermanni Hälvä
Aapo Hyvärinen

title: Hidden Markov nonlinear ICA: Unsupervised learning from nonstationary time series

venue: In Conference on Uncertainty in Artificial Intelligence,

authors:
Christina Heinze-Deml
Nicolai Meinshausen

title: Conditional variance penalties and domain shift robustness

venue: Machine Learning,

authors:
Geoffrey E Hinton
Lawrence M Parsons

title: Frames of reference and mental imagery. Attention and performance IX, pages

authors:
Patrik Hoyer
Dominik Janzing
Joris M Mooij
Jonas Peters
Bernhard Schölkopf

title: Nonlinear causal discovery with additive noise models

authors:
Chin-Wei Huang
David Krueger
Alexandre Lacoste
Aaron C. Courville

title: Neural autoregressive flows

venue: Proceedings of the 35th International Conference on Machine Learning,

title: Fast and robust fixed-point algorithms for independent component analysis

venue: IEEE transactions on Neural Networks,

authors:
Aapo Hyvärinen
Hiroshi Morioka

title: Unsupervised feature extraction by time-contrastive learning and nonlinear ICA

title: Nonlinear ICA of temporally dependent stationary sources

venue: In Artificial Intelligence and Statistics,

authors:
Aapo Hyvärinen
Petteri Pajunen

title: Nonlinear independent component analysis: Existence and uniqueness results

venue: Neural Networks,

authors:
Aapo Hyvärinen
Juha Karhunen
Erkki Oja

title: Independent Component Analysis

authors:
Aapo Hyvärinen
Hiroaki Sasaki
Richard Turner

title: Nonlinear ICA using auxiliary variables and generalized contrastive learning

venue: In The 22nd International Conference on Artificial Intelligence and Statistics,

title: Causal version of principle of insufficient reason and maxent

venue: arXiv preprint arXiv:2102.03906,

authors:
Dominik Janzing
Bernhard Schölkopf

title: Causal inference using the algorithmic Markov condition

venue: IEEE Transactions on Information Theory,

title: Detecting confounding in multivariate linear models via spectral analysis

venue: Journal of Causal Inference,

authors:
Dominik Janzing
Patrik O Hoyer
Bernhard Schölkopf

title: Telling cause from effect based on highdimensional observations

authors:
Dominik Janzing
Joris Mooij
Kun Zhang
Jan Lemeire
Jakob Zscheischler
Povilas Daniušis
Bastian Steudel
Bernhard Schölkopf

title: Information-geometric approach to inferring causal directions

venue: Artificial Intelligence,

authors:
Dominik Janzing
Bastian Steudel
Naji Shajarisales
Bernhard Schölkopf

title: Justifying informationgeometric causal inference

venue: In Measures of complexity,

authors:
Dominik Janzing
Raphael Chaves
Bernhard Schölkopf

title: Algorithmic independence of initial condition and dynamical law in thermodynamics and causal inference

venue: New Journal of Physics,

authors:
Ilyes Khemakhem
Diederik Kingma
Ricardo Monti
Aapo Hyvärinen

title: Variational autoencoders and nonlinear ICA: A unifying framework

authors:
Ilyes Khemakhem
Ricardo Monti
Robert Leech
Aapo Hyvarinen

title: Causal autoregressive flows

title: On tables of random numbers

venue: Sankhyā: The Indian Journal of Statistics,

authors:
Sanghack Lee
Elias Bareinboim

title: Structural causal bandits: where to intervene

venue: Advances in Neural Information Processing Systems 31,

authors:
Felix Leeb
Yashas Annadani
Stefan Bauer
Bernhard Schölkopf

title: Structural autoencoders improve representations for generation and transfer

venue: arXiv preprint arXiv:2006.07796,

authors:
Erich L Lehmann
George Casella

title: Theory of point estimation

venue: Springer Science & Business Media,

authors:
Jan Lemeire
Erik Dirkx

title: Causal models as minimal descriptions of multivariate systems

authors:
Jan Lemeire
Dominik Janzing

title: Replacing causal faithfulness with algorithmic independence of conditionals

venue: Minds and Machines,

authors:
Francesco Locatello
Stefan Bauer
Mario Lucic
Gunnar Raetsch
Sylvain Gelly
Bernhard Schölkopf
Olivier Bachem

title: Challenging common assumptions in the unsupervised learning of disentangled representations

authors:
Chaochao Lu
Bernhard Schölkopf
José Miguel Hernández-Lobato

title: Deconfounding reinforcement learning in observational settings

venue: arXiv preprint arXiv:1812.10576,

authors:
Chaochao Lu
Biwei Huang
Ke Wang
José Miguel Hernández-Lobato
Kun Zhang
Bernhard Schölkopf

title: Sample-efficient reinforcement learning via counterfactual-based data augmentation

venue: arXiv preprint arXiv:2012.09092,

authors:
Sara Magliacane
Thijs van Ommen
Tom Claassen
Stephan Bongers
Philip Versteeg
Joris M Mooij

title: Domain adaptation by using causal inference to predict invariant conditional distributions

venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,

title: Beyond correlational analysis: Recent innovations in theory and method

venue: In Sociological forum,

authors:
Michael Mistry
Jonas Buchli
Stefan Schaal

title: Inverse dynamics control of floating base systems using orthogonal decomposition

venue: In 2010 IEEE International Conference on Robotics and Automation,

authors:
Ricardo Pio Monti
Kun Zhang
Aapo Hyvärinen

title: Causal discovery with general non-linear relationships using non-linear ICA

authors:
Joris M Mooij
Jonas Peters
Dominik Janzing
Jakob Zscheischler
Bernhard Schölkopf

title: Distinguishing cause from effect using observational data: methods and benchmarks

venue: The Journal of Machine Learning Research,

authors:
Parry Moon
Domina Eberle Spencer

title: Field theory handbook, including coordinate systems, differential equations and their solutions

title: Blind source separation using algorithmic information

venue: theory. Neurocomputing,

title: Blind source separation of natural signals based on approximate complexity minimization

venue: In Proc. Int. Workshop on Independent Component Analysis and Signal Separation

authors:
George Papamakarios
Eric Nalisnick
Danilo Jimenez Rezende
Shakir Mohamed
Balaji Lakshminarayanan

title: Normalizing flows for probabilistic modeling and inference

venue: Journal of Machine Learning Research,

authors:
Giambattista Parascandolo
Niki Kilbertus
Mateo Rojas-Carulla
Bernhard Schölkopf

title: Learning independent causal mechanisms

authors:
Judea Pearl
Elias Bareinboim

title: External validity: From do-calculus to transportability across populations

venue: Statistical Science,

authors:
Fabian Pedregosa
Gaël Varoquaux
Alexandre Gramfort
Vincent Michel
Bertrand Thirion
Olivier Grisel
Mathieu Blondel
Peter Prettenhofer
Ron Weiss
Vincent Dubourg

title: Scikit-learn: Machine learning in python

authors:
William S. Peebles
John Peebles
Jun-Yan Zhu
Alexei A. Efros
Antonio Torralba

title: The Hessian penalty: A weak prior for unsupervised disentanglement

venue: In ECCV,

authors:
Jonas Peters
Peter Bühlmann

title: Identifiability of Gaussian structural equation models with equal error variances

authors:
Jonas Peters
Joris M Mooij
Dominik Janzing
Bernhard Schölkopf

title: Causal discovery with continuous additive noise models

authors:
Jonas Peters
Peter Bühlmann
Nicolai Meinshausen

title: Causal inference by using invariant prediction: identification and confidence intervals

venue: Journal of the Royal Statistical Society. Series B (Statistical Methodology),

authors:
Jonas Peters
Dominik Janzing
Bernhard Schölkopf

title: Elements of causal inference: foundations and learning algorithms

authors:
Dinh-Tuan Pham
J-F Cardoso

title: Blind separation of instantaneous mixtures of nonstationary sources

authors:
Dinh Tuan Pham
Philippe Garat

title: Blind separation of mixture of independent sources through a quasi-maximum likelihood approach

title: Liouville’s theorem. Pacific

venue: Journal of Mathematics,

authors:
Aditya Ramesh
Youngduck Choi
Yann LeCun

title: A spectral regularizer for unsupervised disentanglement

venue: arXiv preprint arXiv:1812.01161,

authors:
Mateo Rojas-Carulla
Bernhard Schölkopf
Richard Turner
Jonas Peters

title: Invariant models for causal transfer learning

authors:
Dominik Rothenhäusler
Nicolai Meinshausen
Peter Bühlmann
Jonas Peters

title: Anchor regression: Heterogeneous data meet causality

venue: Journal of the Royal Statistical Society: Series B (Statistical Methodology),

title: Scientific explanation and the causal structure of the world

authors:
B Schölkopf
D Janzing
J Peters
E Sgouritsa
K Zhang
J Mooij

title: On causal and anticausal learning

venue: In 29th International Conference on Machine Learning (ICML

authors:
Bernhard Schölkopf
Francesco Locatello
Stefan Bauer
Nan Rosemary Ke
Nal Kalchbrenner
Anirudh Goyal
Yoshua Bengio

title: Toward causal representation learning

venue: Proceedings of the IEEE,

title: Listening with two ears

venue: Music perception,

authors:
Naji Shajarisales
Dominik Janzing
Bernhard Schoelkopf
Michel Besserve

title: Telling cause from effect in deterministic linear dynamical systems

authors:
Xinwei Shen
Furui Liu
Hanze Dong
Qing Lian
Zhitang Chen
Tong Zhang

title: Disentangled generative causal representation learning

venue: arXiv preprint arXiv:2010.02637,

authors:
Shohei Shimizu
Patrik O Hoyer
Aapo Hyvärinen
Antti Kerminen
Michael Jordan

title: A linear non-Gaussian acyclic model for causal discovery

title: On a property of a normal distribution

venue: In Doklady Akad. Nauk,

title: Conformal maps and the theorem of Liouville

venue: PhD thesis, Faculty of Science and Engineering,

authors:
B. Steudel
D. Janzing
B. Schölkopf

title: Causal Markov condition for submodular information measures

venue: Conference on Learning Theory (COLT),

authors:
Adarsh Subbaswamy
Peter Schulam
Suchi Saria

title: Preventing failures due to dataset shift: Learning predictive models that transport

authors:
Anisse Taleb
Christian Jutten

title: Source separation in post-nonlinear mixtures

title: Historical analysis of political processes

venue: In Handbook of sociological theory,

title: Liouville’s theorem revisited

venue: Enseignement Mathematique,

authors:
Pauli Virtanen
Ralf Gommers
Travis E Oliphant
Matt Haberland
Tyler Reddy
David Cournapeau
Evgeni Burovski
Pearu Peterson
Warren Weckesser
Jonathan Bright

title: Scipy 1.0: fundamental algorithms for scientific computing in python

venue: Nature methods,

authors:
Julius von Kügelgen
Alexander Mey
Marco Loog

title: Semi-generative modelling: Covariate-shift adaptation with cause and effect features

authors:
Julius von Kügelgen
Alexander Mey
Marco Loog
Bernhard Schölkopf

title: Semi-supervised learning, causality, and the conditional cluster assumption

authors:
Julius von Kügelgen
Ivan Ustyuzhaninov
Peter Gehler
Matthias Bethge
Bernhard Schölkopf

title: Towards causal generative scene models via competition of experts

venue: In ICLR 2020 Workshop on "Causal Learning for Decision Making",

authors:
Julius von Kügelgen
Yash Sharma
Luigi Gresele
Wieland Brendel
Bernhard Schölkopf
Michel Besserve
Francesco Locatello

title: Self-supervised learning with data augmentations provably isolates content from style

authors:
Junzhe Zhang
Elias Bareinboim

title: Near-optimal reinforcement learning in dynamic treatment regimes

authors:
K. Zhang
B. Schölkopf
K. Muandet
Z. Wang

title: Domain adaptation under target and conditional shift

venue: Proceedings of the 30th International Conference on Machine Learning,

authors:
Kun Zhang
Laiwan Chan

title: Minimal nonlinear distortion principle for nonlinear independent component analysis

authors:
Kun Zhang
Aapo Hyvärinen

title: On the identifiability of the post-nonlinear causal model

venue: In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence,

heading: 1 Introduction

text:
One of the goals of unsupervised learning is to uncover properties of the data generating process, such as latent structures giving rise to the observed data. Identifiability [55] formalises this desideratum: under suitable assumptions, a model learnt from observations should match the ground truth, up to well-defined ambiguities. Within representation learning, identifiability has been studied mostly in the context of independent component analysis (ICA) [17, 40], which assumes that the observed data x results from mixing unobserved independent random variables si referred to as sources. The aim is to recover the sources based on the observed mixtures alone, also termed blind source separation (BSS). A major obstacle to BSS is that, in the nonlinear case, independent component estimation does not necessarily correspond to recovering the true sources: it is possible to give counterexamples where the observations are transformed into components yi which are independent, yet still mixed with respect to the true sources si [20, 39, 98]. In other words, nonlinear ICA is not identifiable.
In order to achieve identifiability, a growing body of research postulates additional supervision or structure in the data generating process, often in the form of auxiliary variables [28, 30, 37, 38, 41]. In the present work, we investigate a different route to identifiability by drawing inspiration from the field of causal inference [71, 78] which has provided useful insights for a number of machine learning tasks, including semi-supervised [87, 103], transfer [6, 23, 27, 31, 61, 72, 84, 85, 97, 102, 107], reinforcement [7, 14, 22, 26, 53, 59, 60, 106], and unsupervised [9, 10, 54, 70, 88, 91, 104, 105] learning. To this end, we interpret the ICA mixing as a causal process and apply the principle of independent causal mechanisms (ICM) which postulates that the generative process consists of independent modules which do not share information [43, 78, 87]. In this context, “independent” does not refer to statistical independence of random variables, but rather to the notion that the distributions and functions composing the generative process are chosen independently by Nature [43, 48]. While a formalisation of ICM [43, 57] in terms of algorithmic (Kolmogorov) complexity [51] exists, it is not computable, and hence applying ICM in practice requires assessing such non-statistical independence
∗Equal contribution. Code available at: https://github.com/lgresele/independent-mechanism-analysis
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
with suitable domain specific criteria [96]. The goal of our work is thus to constrain the nonlinear ICA problem, in particular the mixing function, via suitable ICM measures, thereby ruling out common counterexamples to identifiability which intuitively violate the ICM principle.
Traditionally, ICM criteria have been developed for causal discovery, where both cause and effect are observed [18, 45, 46, 110]. They enforce an independence between (i) the cause (source) distribution and (ii) the conditional or mechanism (mixing function) generating the effect (observations), and thus rely on the fact that the observed cause distribution is informative. As we will show, this renders them insufficient for nonlinear ICA, since the constraints they impose are satisfied by common counterexamples to identifiability. With this in mind, we introduce a new way to characterise or refine the ICM principle for unsupervised representation learning tasks such as nonlinear ICA.
Motivating example. To build intuition, we turn to a famous example of ICA and BSS: the cocktail party problem, illustrated in Fig. 1 (Left). Here, a number of conversations are happening in parallel, and the task is to recover the individual voices si from the recorded mixtures xi. The mixing or recording process f is primarily determined by the room acoustics and the locations at which microphones are placed. Moreover, each speaker influences the recording through their positioning in the room, and we may think of this influence as ∂f/∂si. Our independence postulate then amounts to stating that the speakers’ positions are not fine-tuned to the room acoustics and microphone placement, or to each other, i.e., the contributions ∂f/∂si should be independent (in a non-statistical sense).1
Our approach. We formalise this notion of independence between the contributions ∂f/∂si of each source to the mixing process (i.e., the columns of the Jacobian matrix Jf of partial derivatives) as an orthogonality condition, see Fig. 1 (Right). Specifically, the absolute value of the determinant |Jf |, which describes the local change in infinitesimal volume induced by mixing the sources, should factorise or decompose as the product of the norms of its columns. This can be seen as a decoupling of the local influence of each partial derivative in the pushforward operation (mixing function) mapping the source distribution to the observed one, and gives rise to a novel framework which we term independent mechanism analysis (IMA). IMA can be understood as a refinement of the ICM principle that applies the idea of independence of mechanisms at the level of the mixing function.
Contributions. The structure and contributions of this paper can be summarised as follows: • we review well-known obstacles to identifiability of nonlinear ICA (§ 2.1), as well as existing ICM
criteria (§ 2.2), and show that the latter do not sufficiently constrain nonlinear ICA (§ 3); • we propose a more suitable ICM criterion for unsupervised representation learning which gives rise
to a new framework that we term independent mechanism analysis (IMA) (§ 4); we provide geometric and information-theoretic interpretations of IMA (§ 4.1), introduce an IMA contrast function which is invariant to the inherent ambiguities of nonlinear ICA (§ 4.2), and show that it rules out a large class of counterexamples and is consistent with existing identifiability results (§ 4.3);
• we experimentally validate our theoretical claims and propose a regularised maximum-likelihood learning approach based on the IMA constrast which outperforms the unregularised baseline (§ 5); additionally, we introduce a method to learn nonlinear ICA solutions with triangular Jacobian and a metric to assess BSS which can be of independent interest for the nonlinear ICA community.
1For additional intuition and possible violations in the context of the cocktail party problem, see Appendix B.4.

heading: 2 Background and preliminaries

text: Our work builds on and connects related literature from the fields of independent component analysis (§ 2.1) and causal inference (§ 2.2). We review the most important concepts below.

heading: 2.1 Independent component analysis (ICA)

text:
Assume the following data-generating process for independent component analysis (ICA) x = f(s) , ps(s) = ∏n i=1 psi(si) , (1)
where the observed mixtures x ∈ Rn result from applying a smooth and invertible mixing function f : Rn → Rn to a set of unobserved, independent signals or sources s ∈ Rn with smooth, factorised density ps with connected support (see illustration Fig. 2b). The goal of ICA is to learn an unmixing function g : Rn → Rn such that y = g(x) has independent components. Blind source separation (BSS), on the other hand, aims to recover the true unmixing f−1 and thus the true sources s (up to tolerable ambiguities, see below). Whether performing ICA corresponds to solving BSS is related to the concept of identifiability of the model class. Intuitively, identifiability is the desirable property that all models which give rise to the same mixture distribution should be “equivalent” up to certain ambiguities, formally defined as follows. Definition 2.1 (∼-identifiability). Let F be the set of all smooth, invertible functions f : Rn → Rn, and P be the set of all smooth, factorised densities ps with connected support on Rn. Let M ⊆ F×P be a subspace of models and let ∼ be an equivalence relation on M. Denote by f∗ps the push-forward density of ps via f . Then the generative process (1) is said to be ∼-identifiable on M if
∀(f , ps), (f̃ , ps̃) ∈ M : f∗ps = f̃∗ps̃ =⇒ (f , ps) ∼ (f̃ , ps̃) . (2) If the true model belongs to the model class M, then ∼-identifiability ensures that any model in M learnt from (infinite amounts of) data will be ∼-equivalent to the true one. An example is linear ICA which is identifiable up to permutation and rescaling of the sources on the subspace MLIN of pairs of (i) invertible matrices (constraint on F) and (ii) factorizing densities for which at most one si is Gaussian (constraint on P) [17, 21, 93], see Appendix A for a more detailed account. In the nonlinear case (i.e., without constraints on F), identifiability is much more challenging. If si and sj are independent, then so are hi(si) and hj(sj) for any functions hi and hj . In addition to permutation-ambiguity, such element-wise h(s) = (h1(s1), ..., hn(sn)) can therefore not be resolved either. We thus define the desired form of identifiability for nonlinear BSS as follows. Definition 2.2 (∼BSS). The equivalence relation ∼BSS on F × P defined as in Defn. 2.1 is given by
(f , ps) ∼BSS (f̃ , ps̃) ⇐⇒ ∃P,h s.t. (f , ps) = (f̃ ◦ h−1 ◦P−1, (P ◦ h)∗ps̃) (3) where P is a permutation and h(s) = (h1(s1), ..., hn(sn)) is an invertible, element-wise function.
A fundamental obstacle—and a crucial difference to the linear problem—is that in the nonlinear case, different mixtures of si and sj can be independent, i.e., solving ICA is not equivalent to solving BSS. A prominent example of this is given by the Darmois construction [20, 39]. Definition 2.3 (Darmois construction). The Darmois construction gD : Rn → (0, 1)n is obtained by recursively applying the conditional cumulative distribution function (CDF) transform:
gDi (x1:i) := P(Xi ≤ xi|x1:i−1) = ∫ xi −∞ p(x ′ i|x1:i−1)dx′i (i = 1, ..., n). (4)
The resulting estimated sources yD = gD(x) are mutually-independent uniform r.v.s by construction, see Fig. 2a for an illustration. However, they need not be meaningfully related to the true sources s, and will, in general, still be a nonlinear mixing thereof [39].2 Denoting the mixing function corresponding to (4) by fD = (gD)−1 and the uniform density on (0, 1)n by pu, the Darmois solution (fD, pu) thus allows construction of counterexamples to ∼BSS-identifiability on F × P .3 Remark 2.4. gD has lower-triangular Jacobian, i.e., ∂gDi /∂xj = 0 for i < j. Since the order of the xi is arbitrary, applying gD after a permutation yields a different Darmois solution. Moreover, (4) yields independent components yD even if the sources si were not independent to begin with.4
2Consider, e.g., a mixing f with full Jacobian which yields a contradiction to Defn. 2.2, due to Remark 2.4. 3By applying a change of variables, we can see that the transformed variables in (4) are uniformly distributed in the open unit cube, thereby corresponding to independent components [69, § 2.2]. 4This has broad implications for unsupervised learning, as it shows that, for i.i.d. observations, not only factorised priors, but any unconditional prior is insufficient for identifiability (see, e.g., [49], Appendix D.2).
Another well-known obstacle to identifiability are measure-preserving automorphisms (MPAs) of the source distribution ps: these are functions a which map the source space to itself without affecting its distribution, i.e., a∗ps = ps [39]. A particularly instructive class of MPAs is the following [49, 58]. Definition 2.5 (“Rotated-Gaussian” MPA). Let R ∈ O(n) be an orthogonal matrix, and denote by Fs(s) = (Fs1(s1), ..., Fsn(sn)) and Φ(z) = (Φ(z1), ...,Φ(zn)) the element-wise CDFs of a smooth, factorised density ps and of a Gaussian, respectively. Then the “rotated-Gaussian” MPA aR(ps) is
aR(ps) = F −1 s ◦Φ ◦R ◦Φ−1 ◦ Fs . (5)
aR(ps) first maps to the (rotationally invariant) standard isotropic Gaussian (via Φ−1 ◦ Fs), then applies a rotation, and finally maps back, without affecting the distribution of the estimated sources. Hence, if (f̃ , ps̃) is a valid solution, then so is (f̃ ◦ aR(ps̃), ps̃) for any R ∈ O(n). Unless R is a permutation, this constitutes another common counterexample to ∼BSS-identifiability on F × P . Identifiability results for nonlinear ICA have recently been established for settings where an auxiliary variable u (e.g., environment index, time stamp, class label) renders the sources conditionally independent [37, 38, 41, 49]. The assumption on ps in (1) is replaced with ps|u(s|u) = ∏n i=1 psi|u(si|u), thus restricting P in Defn. 2.1. In most cases, u is assumed to be observed, though [30] is a notable exception. Similar results exist given access to a second noisy view x̃ [28].

heading: 2.2 Causal inference and the principle of independent causal mechanisms (ICM)

text:
Rather than relying only on additional assumptions on P (e.g., via auxiliary variables), we seek to further constrain (1) by also placing assumptions on the set F of mixing functions f . To this end, we draw inspiration from the field of causal inference [71, 78]. Of central importance to our approach is the Principle of Independent Causal Mechanisms (ICM) [43, 56, 87]. Principle 2.6 (ICM principle [78]). The causal generative process of a system’s variables is composed of autonomous modules that do not inform or influence each other.
These “modules” are typically thought of as the conditional distributions of each variable given its direct causes. Intuitively, the principle then states that these causal conditionals correspond to independent mechanisms of nature which do not share information. Crucially, here “independent” does not refer to statistical independence of random variables, but rather to independence of the underlying distributions as algorithmic objects. For a bivariate system comprising a cause c and an effect e, this idea reduces to an independence of cause and mechanism, see Fig. 2c. One way to formalise ICM uses Kolmogorov complexity K(·) [51] as a measure of algorithmic information [43]. However, since Kolmogorov complexity is is not computable, using ICM in practice requires assessing Principle 2.6 with other suitable proxy criteria [9, 11, 34, 42, 45, 65, 75–78, 90, 110].5 Allowing for deterministic relations between cause (sources) and effect (observations), the criterion which is most closely related to the ICA setting in (1) is information-geometric causal inference (IGCI) [18, 46].6 IGCI assumes a nonlinear relation e = f(c) and formulates a notion of indepen-
5“This can be seen as an algorithmic analog of replacing the empirically undecidable question of statistical independence with practical independence tests that are based on assumptions on the underlying distribution” [43].
6For a similar criterion which assumes linearity [45, 110] and its relation to linear ICA, see Appendix B.1.
dence between the cause distribution pc and the deterministic mechanism f (which we think of as a degenerate conditional pe|c) via the following condition (in practice, assumed to hold approximately),
CIGCI(f , pc) := ∫ log |Jf (c)| pc(c)dc− ∫ log |Jf (c)| dc = 0 , (6)
where (Jf (c))ij = ∂fi/∂cj(c) is the Jacobian matrix and | · | the absolute value of the determinant. CIGCI can be understood as the covariance between pc and log |Jf | (viewed as r.v.s on the unit cube w.r.t. the Lebesgue measure), so that CIGCI = 0 rules out a form of fine-tuning between pc and |Jf |. As its name suggests, IGCI can, from an information-geometric perspective, also be seen as an orthogonality condition between cause and mechanism in the space of probability distributions [46], see Appendix B.2, particularly eq. (19) for further details.

heading: 3 Existing ICM measures are insufficient for nonlinear ICA

text:
Our aim is to use the ICM Principle 2.6 to further constrain the space of models M ⊆ F ×P and rule out common counterexamples to identifiability such as those presented in § 2.1. Intuitively, both the Darmois construction (4) and the rotated Gaussian MPA (5) give rise to “non-generic” solutions which should violate ICM: the former, (fD, pu), due the triangular Jacobian of fD (see Remark 2.4), meaning that each observation xi = fDi (y1:i) only depends on a subset of the inferred independent components y1:i, and the latter, (f ◦ aR(ps), ps), due to the dependence of f ◦ aR(ps) on ps (5). However, the ICM criteria described in § 2.2 were developed for the task of cause-effect inference where both variables are observed. In contrast, in this work, we consider an unsupervised representation learning task where only the effects (mixtures x) are observed, but the causes (sources s) are not. It turns out that this renders existing ICM criteria insufficient for BSS: they can easily be satisfied by spurious solutions which are not equivalent to the true one. We can show this for IGCI. Denote by MIGCI = {(f , ps) ∈ F × P : CIGCI(f , ps) = 0} ⊂ F × P the class of nonlinear ICA models satisfying IGCI (6). Then the following negative result holds. Proposition 3.1 (IGCI is insufficient for ∼BSS-identifiability). (1) is not ∼BSS-identifiable on MIGCI. Proof. IGCI (6) is satisfied when ps is uniform. However, the Darmois construction (4) yields uniform sources, see Fig. 2a. This means that (fD ◦ aR(pu), pu) ∈ MIGCI, so IGCI can be satisfied by solutions which do not separate the sources in the sense of Defn. 2.2, see footnote 2 and [39].
As illustrated in Fig. 2c, condition (6) and other similar criteria enforce a notion of “genericity” or “decoupling” of the mechanism w.r.t. the observed input distribution.7 They thus rely on the fact that the cause (source) distribution is informative, and are generally not invariant to reparametrisation of the cause variables. In the (nonlinear) ICA setting, on the other hand, the learnt source distribution may be fairly uninformative. This poses a challenge for existing ICM criteria since any mechanism is generic w.r.t. an uninformative (uniform) input distribution.

heading: 4 Independent mechanism analysis (IMA)

text: As argued in § 3, enforcing independence between the input distribution and the mechanism (Fig. 2c), as existing ICM criteria do, is insufficient for ruling out spurious solutions to nonlinear ICA. We therefore propose a new ICM-inspired framework which is more suitable for BSS and which we term independent mechanism analysis (IMA).8 All proofs are provided in Appendix C.

heading: 4.1 Intuition behind IMA

text:
As motivated using the cocktail party example in § 1 and Fig. 1 (Left), our main idea is to enforce a notion of independence between the contributions or influences of the different sources si on the observations x = f(s) as illustrated in Fig. 2d—as opposed to between the source distribution and mixing function, cf. Fig. 2c. These contributions or influences are captured by the vectors of partial derivatives ∂f/∂si. IMA can thus be understood as a refinement of ICM at the level of the mixing f : in addition to statistically independent components si, we look for a mixing with contributions ∂f/∂si which are independent, in a non-statistical sense which we formalise as follows. Principle 4.1 (IMA). The mechanisms by which each source si influences the observed distribution, as captured by the partial derivatives ∂f/∂si, are independent of each other in the sense that for all s:
log |Jf (s)| = n∑
i=1
log ∥∥∥∥ ∂f∂si (s) ∥∥∥∥ (7)
7In fact, many ICM criteria can be phrased as special cases of a unifying group-invariance framework [9]. 8The title of the present work is thus a reverence to Pierre Comon’s seminal 1994 paper [17].
Geometric interpretation. Geometrically, the IMA principle can be understood as an orthogonality condition, as illustrated for n = 2 in Fig. 1 (Right). First, the vectors of partial derivatives ∂f/∂si, for which the IMA principle postulates independence, are the columns of Jf . |Jf | thus measures the volume of the n−dimensional parallelepiped spanned by these columns, as shown on the right. The product of their norms, on the other hand, corresponds to the volume of an n-dimensional box, or rectangular parallelepiped with side lengths ∥∂f/∂si∥, as shown on the left. The two volumes are equal if and only if all columns ∂f/∂si of Jf are orthogonal. Note that (7) is trivially satisfied for n = 1, i.e., if there is no mixing, further highlighting its difference from ICM for causal discovery.
Independent influences and orthogonality. In a high dimensional setting (large n), this orthogonality can be intuitively interpreted from the ICM perspective as Nature choosing the direction of the influence of each source component in the observation space independently and from an isotropic prior. Indeed, it can be shown that the scalar product of two independent isotropic random vectors in Rn vanishes as the dimensionality n increases (equivalently: two high-dimensional isotropic vectors are typically orthogonal). This property was previously exploited in other linear ICM-based criteria (see [44, Lemma 5] and [45, Lemma 1 & Thm. 1]).9 The principle in (7) can be seen as a constraint on the function space, enforcing such orthogonality between the columns of the Jacobian of f at all points in the source domain, thus approximating the high-dimensional behavior described above.10
Information-geometric interpretation and comparison to IGCI. The additive contribution of the sources’ influences ∂f/∂si in (7) suggests their local decoupling at the level of the mechanism f . Note that IGCI (6), on the other hand, postulates a different type of decoupling: one between log |Jf | and ps. There, dependence between cause and mechanism can be conceived as a fine tuning between the derivative of the mechanism and the input density. The IMA principle leads to a complementary, non-statistical measure of independence between the influences ∂f/∂si of the individual sources on the vector of observations. Both the IGCI and IMA postulates have an information-geometric interpretation related to the influence of (“non-statistically”) independent modules on the observations: both lead to an additive decomposition of a KL-divergence between the effect distribution and a reference distribution. For IGCI, independent modules correspond to the cause distribution and the mechanism mapping the cause to the effect (see (19) in Appendix B.2). For IMA, on the other hand, these are the influences of each source component on the observations in an interventional setting (under soft interventions on individual sources), as measured by the KL-divergences between the original and intervened distributions. See Appendix B.3, and especially (22), for a more detailed account.
We finally remark that while recent work based on the ICM principle has mostly used the term “mechanism” to refer to causal Markov kernels p(Xi|PAi) or structural equations [78], we employ it in line with the broader use of this concept in the philosophical literature.11 To highlight just two examples, [86] states that “Causal processes, causal interactions, and causal laws provide the mechanisms by which the world works; to understand why certain things happen, we need to see how they are produced by these mechanisms”; and [99] states that “Mechanisms are events that alter relations among some specified set of elements”. Following this perspective, we argue that a causal mechanism can more generally denote any process that describes the way in which causes influence their effects: the partial derivative ∂f/∂si thus reflects a causal mechanism in the sense that it describes the infinitesimal changes in the observations x, when an infinitesimal perturbation is applied to si.

heading: 4.2 Definition and useful properties of the IMA contrast

text:
We now introduce a contrast function based on the IMA principle (7) and show that it possesses several desirable properties in the context of nonlinear ICA. First, we define a local contrast as the difference between the two integrands of (7) for a particular value of the sources s. Definition 4.2 (Local IMA contrast). The local IMA contrast cIMA(f , s) of f at a point s is given by
cIMA(f , s) = n∑ i=1 log ∥∥∥∥ ∂f∂si (s) ∥∥∥∥− log |Jf (s)| . (8)
Remark 4.3. This corresponds to the left KL measure of diagonality [2] for √ Jf (s)⊤Jf (s).
9This has also been used as a “leading intuition” [sic] to interpret IGCI in [46]. 10To provide additional intuition on how IMA differs from existing principles of independence of cause and
mechanism, we give examples, both technical and pictorial, of violations of both in Appendix B.4. 11See Table 1 in [62] for a long list of definitions from the literature.
The local IMA contrast cIMA(f , s) quantifies the extent to which the IMA principle is violated at a given point s. We summarise some of its properties in the following proposition. Proposition 4.4 (Properties of cIMA(f , s)). The local IMA contrast cIMA(f , s) defined in (8) satisfies:
(i) cIMA(f , s) ≥ 0, with equality if and only if all columns ∂f/∂si(s) of Jf (s) are orthogonal. (ii) cIMA(f , s) is invariant to left multiplication of Jf (s) by an orthogonal matrix and to right
multiplication by permutation and diagonal matrices.
Property (i) formalises the geometric interpretation of IMA as an orthogonality condition on the columns of the Jacobian from § 4.1, and property (ii) intuitively states that changes of orthonormal basis and permutations or rescalings of the columns of Jf do not affect their orthogonality. Next, we define a global IMA contrast w.r.t. a source distribution ps as the expected local IMA contrast. Definition 4.5 (Global IMA contrast). The global IMA contrast CIMA(f , ps) of f w.r.t. ps is given by
CIMA(f , ps) = Es∼ps [cIMA(f , s)] = ∫ cIMA(f , s)ps(s)ds . (9)
The global IMA contrast CIMA(f , ps) thus quantifies the extent to which the IMA principle is violated for a particular solution (f , ps) to the nonlinear ICA problem. We summarise its properties as follows. Proposition 4.6 (Properties of CIMA(f , ps)). The global IMA contrast CIMA(f , ps) from (9) satisfies:
(i) CIMA(f , ps) ≥ 0, with equality iff. Jf (s) = O(s)D(s) almost surely w.r.t. ps, where O(s),D(s) ∈ Rn×n are orthogonal and diagonal matrices, respectively;
(ii) CIMA(f , ps) = CIMA(f̃ , ps̃) for any f̃ = f ◦ h−1 ◦P−1 and s̃ = Ph(s), where P ∈ Rn×n is a permutation and h(s) = (h1(s1), ..., hn(sn)) an invertible element-wise function.
0.50 0.75 1.00 1.25 1.50 r
/8
0
/8
0.5 1.0 1.5 x
0.6 0.4 0.2 0.0 0.2 0.4 0.6 y
Figure 3: An example of a (non-conformal) orthogonal coordinate transformation from polar (left) to Cartesian (right) coordinates.
Property (i) is the distribution-level analogue to (i) of Prop. 4.4 and only allows for orthogonality violations on sets of measure zero w.r.t. ps. This means that CIMA can only be zero if f is an orthogonal coordinate transformation almost everywhere [19, 52, 66], see Fig. 3 for an example. We particularly stress property (ii), as it precisely matches the inherent indeterminacy of nonlinear ICA: CIMA is blind to reparametrisation of the sources by permutation and element wise transformation.
4.3 Theoretical analysis and justification of CIMA We now show that, under suitable assumptions on the generative model (1), a large class of spurious solutions—such as those based on the Darmois construction (4) or measure preserving automorphisms such as aR from (5) as described in § 2.1—exhibit nonzero IMA contrast. Denote the class of nonlinear ICA models satisfying (7) (IMA) by MIMA = {(f , ps) ∈ F × P : CIMA(f , ps) = 0} ⊂ F × P . Our first main theoretical result is that, under mild assumptions on the observations, Darmois solutions will have strictly positive CIMA, making them distinguishable from those in MIMA. Theorem 4.7. Assume the data generating process in (1) and assume that xi ⊥̸⊥ xj for some i ̸= j. Then any Darmois solution (fD, pu) based on gD as defined in (4) satisfies CIMA(fD, pu) > 0. Thus a solution satisfying CIMA(f , ps) = 0 can be distinguished from (fD, pu) based on the contrast CIMA. The proof is based on the fact that the Jacobian of gD is triangular (see Remark 2.4) and on the specific form of (4). A specific example of a mixing process satisfying the IMA assumption is the case where f is a conformal (angle-preserving) map. Definition 4.8 (Conformal map). A smooth map f : Rn → Rn is conformal if Jf (s) = O(s)λ(s) ∀s, where λ : Rn → R is a scalar field, and O ∈ O(n) is an orthogonal matrix. Corollary 4.9. Under assumptions of Thm. 4.7, if additionally f is a conformal map, then (f , ps) ∈ MIMA for any ps ∈ P due to Prop. 4.6 (i), see Defn. 4.8. Based on Thm. 4.7, (f , ps) is thus distinguishable from Darmois solutions (fD, pu). This is consistent with a result that proves identifiability of conformal maps for n = 2 and conjectures it in general [39].12 However, conformal maps are only a small subset of all maps for which CIMA = 0, as is apparent from the more flexible condition of Prop. 4.6 (i), compared to the stricter Defn. 4.8.
12Note that Corollary 4.9 holds for any dimensionality n.
Example 4.10 (Polar to Cartesian coordinate transform). Consider the non-conformal transformation from polar to Cartesian coordinates (see Fig. 3), defined as (x, y) = f(r, θ) := (r cos(θ), r sin(θ)) with independent sources s = (r, θ), with r ∼ U(0, R) and θ ∼ U(0, 2π).13 Then, CIMA(f , ps) = 0 and CIMA(fD, pu) > 0 for any Darmois solution (fD, pu) —see Appendix D for details.
Finally, for the case in which the true mixing is linear, we obtain the following result. Corollary 4.11. Consider a linear ICA model, x = As, with E[s⊤s] = I, and A ∈ O(n) an orthogonal, non-trivial mixing matrix, i.e., not the product of a diagonal and a permutation matrix DP. If at most one of the si is Gaussian, then CIMA(A, ps) = 0 and CIMA(fD, pu) > 0.
In a “blind” setting, we may not know a priori whether the true mixing is linear or not, and thus choose to learn a nonlinear unmixing. Corollary 4.11 shows that, in this case, Darmois solutions are still distinguishable from the true mixing via CIMA. Note that unlike in Corollary 4.9, the assumption that xi ⊥̸⊥ xj for some i ̸= j is not required for Corollary 4.11. In fact, due to Theorem 11 of [17], it follows from the assumed linear ICA model with non-Gaussian sources, and the fact that the mixing matrix is not the product of a diagonal and a permutation matrix (see also Appendix A).
Having shown that the IMA principle allows to distinguish a class of models (including, but not limited to conformal maps) from Darmois solutions, we next turn to a second well-known counterexample to identifiability: the “rotated-Gaussian” MPA aR(ps) (5) from Defn. 2.5. Our second main theoretical result is that, under suitable assumptions, this class of MPAs can also be ruled out for “non-trivial” R. Theorem 4.12. Let (f , ps) ∈ MIMA and assume that f is a conformal map. Given R ∈ O(n), assume additionally that ∃ at least one non-Gaussian si whose associated canonical basis vector ei is not transformed by R−1 = R⊤ into another canonical basis vector ej . Then CIMA(f ◦ aR(ps), ps) > 0.
Thm. 4.12 states that for conformal maps, applying the aR(ps) transformation at the level of the sources leads to an increase in CIMA except for very specific rotations R that are “fine-tuned” to ps in the sense that they permute all non-Gaussian sources si with another sj . Interestingly, as for the linear case, non-Gaussianity again plays an important role in the proof of Thm. 4.12.

heading: 5 Experiments

text:
Our theoretical results from § 4 suggest that CIMA is a promising contrast function for nonlinear blind source separation. We test this empirically by evaluating the CIMA of spurious nonlinear ICA solutions (§ 5.1), and using it as a learning objective to recover the true solution (§ 5.2).
We sample the ground truth sources from a uniform distribution in [0, 1]n; the reconstructed sources are also mapped to the uniform hypercube as a reference measure via the CDF transform. Unless
13For different ps, (x, y) can be made to have independent Gaussian components ([98], II.B), and CIMAidentifiability is lost; this shows that the assumption of Thm. 4.7 that xi ⊥̸⊥ xj for some i ̸= j is crucial.
otherwise specified, the ground truth mixing f is a Möbius transformation [81] (i.e., a conformal map) with randomly sampled parameters, thereby satisfying Principle 4.1. In all of our experiments, we use JAX [12] and Distrax [13]. For additional technical details, equations and plots see Appendix E. The code to reproduce our experiments is available at this link.
5.1 Numerical evaluation of the CIMA contrast for spurious nonlinear ICA solutions
Learning the Darmois construction. To learn the Darmois construction from data, we use normalising flows, see [35, 69]. Since Darmois solutions have triangular Jacobian (Remark 2.4), we use an architecture based on residual flows [16] which we constrain such that the Jacobian of the full model is triangular. This yields an expressive model which we train effectively via maximum likelihood.
CIMA of Darmois solutions. To check whether Darmois solutions (learnt from finite data) can be distinguished from the true one, as predicted by Thm. 4.7, we generate 1000 random mixing functions for n = 2, compute the CIMA values of learnt solutions, and find that all values are indeed significantly larger than zero, see Fig. 4 (a). The same holds for higher dimensions, see Fig. 4 (b) for results with 50 random mixings for n ∈ {2, 3, 5, 10}: with higher dimensionality, both the mean and variance of the CIMA distribution for the learnt Darmois solutions generally attain higher values.14 We confirmed these findings for mappings which are not conformal, while still satisfying (7), in Appendix E.5.
CIMA of MPAs. We also investigate the effect on CIMA of applying an MPA aR(·) from (5) to the true solution or a learnt Darmois solution. Results for n = 2 dim. for different rotation matrices R (parametrised by the angle θ) are shown in Fig. 4 (c). As expected, the behavior is periodic in θ, and vanishes for the true solution (blue) at multiples of π/2, i.e., when R is a permutation matrix, as predicted by Thm. 4.12. For the learnt Darmois solution (red, dashed) CIMA remains larger than zero.
CIMA values for random MLPs. Lastly, we study the behavior of spurious solutions based on the Darmois construction under deviations from our assumption of CIMA = 0 for the true mixing function. To this end, we use invertible MLPs with orthogonal weight initalisation and leaky_tanh activations [29] as mixing functions; the more layers L are added to the mixing MLP, the larger a deviation from our assumptions is expected. We compare the true mixing and learnt Darmois solutions over 20 realisations for each L ∈ {2, 3, 4}, n = 5. Results are shown in figure Fig. 4 (d): the CIMA of the mixing MLPs grows with L; still, the one of the Darmois solution is typically higher.
Summary. We verify that spurious solutions can be distinguished from the true one based on CIMA.
5.2 Learning nonlinear ICA solutions with CIMA-regularised maximum likelihood
Experimental setup. To use CIMA as a learning signal, we consider a regularised maximum-likelihood approach, with the following objective: L(g) = Ex[log pg(x)]− λCIMA(g−1, py), where g denotes the learnt unmixing, y = g(x) the reconstructed sources, and λ ≥ 0 a Lagrange multiplier. For λ = 0, this corresponds to standard maximum likelihood estimation, whereas for λ > 0, L lower-bounds the likelihood, and recovers it exactly iff. (g−1, py) ∈ MIMA. We train a residual flow g (with full Jacobian) to maximise L. For evaluation, we compute (i) the KL divergence to the true data likelihood, as a measure of goodness of fit for the learnt flow model; and (ii) the mean correlation coefficient (MCC) between ground truth and reconstructed sources [37, 49]. We also introduce (iii) a nonlinear extension of the Amari distance [5] between the true mixing and the learnt unmixing, which is larger than or equal to zero, with equality iff. the learnt model belongs to the BSS equivalence class (Defn. 2.2) of the true solution, see Appendix E.5 for details.
Results. In Fig. 4 (Top), we show an example of the distortion induced by different spurious solutions for n = 2, and contrast it with a solution learnt using our proposed objective (rightmost plot). Visually, we find that the CIMA-regularised solution (with λ = 1) recovers the true sources most faithfully. Quantitative results for 50 learnt models for each λ ∈ {0.0, 0.5, 1.0} and n ∈ {5, 7} are summarised in Fig. 5 (see Appendix E for additional plots) . As indicated by the KL divergence values (left), most trained models achieve a good fit to the data across all values of λ.15 We observe that using CIMA (i.e., λ > 0) is beneficial for BSS, both in terms of our nonlinear Amari distance (center, lower is better) and MCC (right, higher is better), though we do not observe a substantial difference between λ = 0.5 and λ = 1.16
Summary: CIMA can be a useful learning signal to recover the true solution. 14the latter possibly due to the increased difficulty of the learning task for larger n 15models with n = 7 have high outlier KL values, seemingly less pronounced for nonzero values of λ 16In Appendix E.5, we also show that our method is superior to a linear ICA baseline, FastICA [36].

heading: 6 Discussion

text:
Assumptions on the mixing function. Instead of relying on weak supervision in the form of auxiliary variables [28, 30, 37, 38, 41, 49], our IMA approach places additional constraints on the functional form of the mixing process. In a similar vein, the minimal nonlinear distortion principle [108] proposes to favor solutions that are as close to linear as possible. Another example is the post-nonlinear model [98, 109], which assumes an element-wise nonlinearity applied after a linear mixing. IMA is different in that it still allows for strongly nonlinear mixings (see, e.g., Fig. 3) provided that the columns of their Jacobians are (close to) orthogonal. In the related field of disentanglement [8, 58], a line of work that focuses on image generation with adversarial networks [24] similarly proposes to constrain the “generator” function via regularisation of its Jacobian [82] or Hessian [74], though mostly from an empirically-driven, rather than from an identifiability perspective as in the present work.
Towards identifiability with CIMA. The IMA principle rules out a large class of spurious solutions to nonlinear ICA. While we do not present a full identifiability result, our experiments show that CIMA can be used to recover the BSS equivalence class, suggesting that identifiability might indeed hold, possibly under additional assumptions—e.g., for conformal maps [39].
IMA and independence of cause and mechanism. While inspired by measures of independence of cause and mechanism as traditionally used for cause-effect inference [18, 45, 46, 110], we view the IMA principle as addressing a different question, in the sense that they evaluate independence between different elements of the causal model. Any nonlinear ICA solution that satisfies the IMA Principle 4.1 can be turned into one with uniform reconstructed sources—thus satisfying IGCI as argued in § 3— through composition with an element-wise transformation which, according to Prop. 4.6 (ii), leaves the CIMA value unchanged. Both IGCI (6) and IMA (7) can therefore be fulfilled simultaneosly, while the former on its own is inconsequential for BSS as shown in Prop. 3.1.
BSS through algorithmic information. Algorithmic information theory has previously been proposed as a unifying framework for identifiable approaches to linear BSS [67, 68], in the sense that commonly-used contrast functions could, under suitable assumptions, be interpreted as proxies for the total complexity of the mixing and the reconstructed sources. However, to the best of our knowledge, the problem of specifying suitable proxies for the complexity of nonlinear mixing functions has not yet been addressed. We conjecture that our framework could be linked to this view, based on the additional assumption of algorithmic independence of causal mechanisms [43], thus potentially representing an approach to nonlinear BSS by minimisation of algorithmic complexity.
ICA for causal inference & causality for ICA. Past advances in ICA have inspired novel causal discovery methods [50, 64, 92]. The present work constitutes, to the best of our knowledge, the first effort to use ideas from causality (specifically ICM) for BSS. An application of the IMA principle to causal discovery or causal representation learning [88] is an interesting direction for future work.
Conclusion. We introduce IMA, a path to nonlinear BSS inspired by concepts from causality. We postulate that the influences of different sources on the observed distribution should be approximately independent, and formalise this as an orthogonality condition on the columns of the Jacobian. We prove that this constraint is generally violated by well-known spurious nonlinear ICA solutions, and propose a regularised maximum likelihood approach which we empirically demonstrate to be effective in recovering the true solution. Our IMA principle holds exactly for orthogonal coordinate transformations, and is thus of potential interest for learning spatial representations [33], robot dynamics [63], or physics problems where orthogonal reference frames are common [66].

heading: Acknowledgements

text:
The authors thank Aapo Hyvärinen, Adrián Javaloy Bornás, Dominik Janzing, Giambattista Parascandolo, Giancarlo Fissore, Nasim Rahaman, Patrick Burauel, Patrik Reizinger, Paul Rubenstein, Shubhangi Ghosh, and the anonymous reviewers for helpful comments and discussions.
Funding Transparency Statement This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B; and by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645.

title: Independent mechanism analysis, a new concept?