summaries: The paper proposes applying the theory of independent causal mechanisms (ICM), from causal discovery, to nonlinear ICA approaches to the blind source separation problem (BSS).  ICM is based on the idea that variables in a system are algorithmically independent or do not share information with each other. The paper shows, however, that ICM is not sufficient for BSS as spurious solutions exist which are not equivalent to the truth.  To address these difficulties, the paper proposes a new condition for identifiability which places an orthogonality condition on the columns of the Jacobian and provides information theoretic and geometric interpretations. The paper then shows theoretically that a large class of spurious solutions are not admitted under the IMA formulation.   Experiments on toy examples demonstrate that the model is identifiable under these spurious solutions. This work proposes a new regularization scheme to improve identifiability in the _nonlinear blind source separation problem_ (nonlinear ICA) inspired by the notion of _independence of mechanism_ coming from causal inference. The idea is to consider only mixing functions which have a Jacobian matrix with orthogonal columns everywhere. They show how this extra assumption allows to exclude two classical types of degeneracies in nonlinear ICA, although without showing identifiability. The theoretical claims are validated by synthetic experiments.   The paper proposes a criterion for non-linear ICA as applied to Blind Source Separation inspired by Independence of Causal Mechanisms in the causal discovery literature. The general idea is to constrain the gradients of the output with respect to the input sources to be orthogonal to each other; this is linked to the notion that the mixing mechanism is in some sense independent (or not "too fine-tuned") to the input sources. The authors show that this constraint eliminates a number of known counterexamples to identifiability in non-linear ICA, and test it experimentally to show that helps to characterize true solutions (both as an evaluation metric, and as a regularizer). The authors conclude with a discussion putting their contributions into context. The authors propose to borrow concepts from causality, in particular "independent causal mechanisms" , to propose a novel framework termed "independent mechanism analysis" (IMA), which can provide non-spurious solutions to nonlinear blind source separation. IMA can be seen as a restricted nonlinear ICA model, where the contributions of the latent variables $z_i$ to the nonlinear mixing $\mathbf{f}$ are independent, i.e. the columns of the Jacobian $J_\mathbf{f}$ are orthogonal.  The paper aims at the identifiability of nonlinear ICA. It proposes a principle as a constraint of mixture functions based on the modified independent causal mechanism principle in the causal discovery literature; investigates the identifiability of nonlinear ICA under such an assumption; then applies it to nonlinear ICA. It also shows that the identifiability results can cover some non-identifiable cases in the existed literature.
abstractText: Independent component analysis provides a principled framework for unsupervised representation learning, with solid theory on the identifiability of the latent code that generated the data, given only observations of mixtures thereof. Unfortunately, when the mixing is nonlinear, the model is provably nonidentifiable, since statistical independence alone does not sufficiently constrain the problem. Identifiability can be recovered in settings where additional, typically observed variables are included in the generative process. We investigate an alternative path and consider instead including assumptions reflecting the principle of independent causal mechanisms exploited in the field of causality. Specifically, our approach is motivated by thinking of each source as independently influencing the mixing process. This gives rise to a framework which we term independent mechanism analysis. We provide theoretical and empirical evidence that our approach circumvents a number of nonidentifiability issues arising in nonlinear blind source separation.
authors.name: Luigi Gresele
authors.name: Julius von Kügelgen
authors.name: Vincent Stimper
authors.name: Bernhard Schölkopf
authors.name: Michel Besserve
references.authors: Pierre Ablin Jean-François Cardoso Alexandre Gramfort
references.title: Faster independent component analysis by preconditioning with hessian approximations
references.venue: IEEE Transactions on Signal Processing,
references.authors: Khaled Alyani Marco Congedo Maher Moakher
references.title: Diagonality measures of Hermitian positivedefinite matrices with application to the approximate joint diagonalization problem
references.venue: Linear Algebra and its Applications,
references.authors: Shun-ichi Amari Hiroshi Nagaoka
references.title: Methods of information geometry, volume 191
references.venue: American Mathematical Soc.,
references.authors: Shun-ichi Amari Andrzej Cichocki Howard Hua Yang
references.title: A new learning algorithm for blind signal separation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Martin Arjovsky Léon Bottou Ishaan Gulrajani David Lopez-Paz
references.title: Invariant risk minimization
references.venue: arXiv preprint arXiv:1907.02893,
references.authors: Elias Bareinboim Andrew Forney Judea Pearl
references.title: Bandits with unobserved confounders: A causal approach
references.venue: Advances in Neural Information Processing Systems,
references.authors: Yoshua Bengio Aaron Courville Pascal Vincent
references.title: Representation learning: A review and new perspectives
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: M Besserve N Shajarisales B Schölkopf D Janzing
references.title: Group invariance principles for causal generative models
references.venue: In 21st International Conference on Artificial Intelligence and Statistics (AISTATS
references.authors: M Besserve A Mehrjou R Sun B Schölkopf
references.title: Counterfactuals uncover the modular structure of deep generative models
references.venue: In Eighth International Conference on Learning Representations (ICLR
references.authors: Patrick Blöbaum Dominik Janzing Takashi Washio Shohei Shimizu Bernhard Schölkopf
references.title: Causeeffect inference by comparing regression errors
references.venue: In International Conference on Artificial Intelligence and Statistics,
references.authors: James Bradbury Roy Frostig Peter Hawkins Matthew James Johnson Chris Leary Dougal Maclaurin Skye Wanderman-Milne
references.title: JAX: composable transformations of Python+NumPy programs, 2018
references.authors: Jake Bruce David Budden Matteo Hessel George Papamakarios Francisco Ruiz
references.title: Distrax: Probability distributions in JAX
references.authors: Lars Buesing Theophane Weber Yori Zwols Sebastien Racaniere Arthur Guez Jean-Baptiste Lespiau Nicolas Heess
references.title: Woulda, coulda, shoulda: Counterfactually-guided policy search
references.venue: arXiv preprint arXiv:1811.06272,
references.authors: Jean-François Cardoso
references.title: The three easy routes to independent component analysis; contrasts and geometry
references.venue: In Proc. ICA,
references.authors: Ricky T.Q. Chen Jens Behrmann David Duvenaud Jörn-Henrik Jacobsen
references.title: Residual flows for invertible generative modeling
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Pierre Comon
references.title: Independent component analysis, a new concept
references.venue: Signal processing,
references.authors: Povilas Daniušis Dominik Janzing Joris Mooij Jakob Zscheischler Bastian Steudel Kun Zhang Bernhard Schölkopf
references.title: Inferring deterministic causal relations
references.venue: In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence,
references.authors: Gaston Darboux
references.title: Leçons sur les systemes orthogonaux et les coordonnées curvilignes
references.authors: George Darmois
references.title: Analyse des liaisons de probabilité
references.venue: In Proc. Int. Stat. Conferences
references.authors: George Darmois
references.title: Analyse générale des liaisons stochastiques: etude particulière de l’analyse factorielle linéaire
references.venue: Revue de l’Institut international de statistique,
references.authors: Andrew Forney Judea Pearl Elias Bareinboim
references.title: Counterfactual data-fusion for online reinforcement learners
references.venue: In International Conference on Machine Learning,
references.authors: Mingming Gong Kun Zhang Tongliang Liu Dacheng Tao Clark Glymour Bernhard Schölkopf
references.title: Domain adaptation with conditional transferable components
references.venue: In International Conference on Machine Learning,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Alexander N Gorban Ivan Yu Tyukin
references.title: Blessing of dimensionality: mathematical foundations of the statistical physics of data
references.venue: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences,
references.authors: Anirudh Goyal Alex Lamb Jordan Hoffmann Shagun Sodhani Sergey Levine Yoshua Bengio Bernhard Schölkopf
references.title: Recurrent independent mechanisms
references.venue: In 9th International Conference on Learning Representations,
references.authors: Daniel Greenfeld Uri Shalit
references.title: Robust learning with the hilbert-schmidt independence criterion
references.venue: In International Conference on Machine Learning,
references.authors: Luigi Gresele Paul K Rubenstein Arash Mehrjou Francesco Locatello Bernhard Schölkopf
references.title: The Incomplete Rosetta Stone problem: Identifiability results for multi-view nonlinear ICA
references.venue: In Uncertainty in Artificial Intelligence,
references.authors: Luigi Gresele Giancarlo Fissore Adrián Javaloy Bernhard Schölkopf Aapo Hyvarinen
references.title: Relative gradient optimization of the Jacobian term in unsupervised deep learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Hermanni Hälvä Aapo Hyvärinen
references.title: Hidden Markov nonlinear ICA: Unsupervised learning from nonstationary time series
references.venue: In Conference on Uncertainty in Artificial Intelligence,
references.authors: Christina Heinze-Deml Nicolai Meinshausen
references.title: Conditional variance penalties and domain shift robustness
references.venue: Machine Learning,
references.authors: Geoffrey E Hinton Lawrence M Parsons
references.title: Frames of reference and mental imagery. Attention and performance IX, pages
references.authors: Patrik Hoyer Dominik Janzing Joris M Mooij Jonas Peters Bernhard Schölkopf
references.title: Nonlinear causal discovery with additive noise models
references.venue: Advances in Neural Information Processing Systems,
references.authors: Chin-Wei Huang David Krueger Alexandre Lacoste Aaron C. Courville
references.title: Neural autoregressive flows
references.venue: Proceedings of the 35th International Conference on Machine Learning,
references.authors: Aapo Hyvärinen
references.title: Fast and robust fixed-point algorithms for independent component analysis
references.venue: IEEE transactions on Neural Networks,
references.authors: Aapo Hyvärinen Hiroshi Morioka
references.title: Unsupervised feature extraction by time-contrastive learning and nonlinear ICA
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Aapo Hyvärinen Hiroshi Morioka
references.title: Nonlinear ICA of temporally dependent stationary sources
references.venue: In Artificial Intelligence and Statistics,
references.authors: Aapo Hyvärinen Petteri Pajunen
references.title: Nonlinear independent component analysis: Existence and uniqueness results
references.venue: Neural Networks,
references.authors: Aapo Hyvärinen Juha Karhunen Erkki Oja
references.title: Independent Component Analysis
references.authors: Aapo Hyvärinen Hiroaki Sasaki Richard Turner
references.title: Nonlinear ICA using auxiliary variables and generalized contrastive learning
references.venue: In The 22nd International Conference on Artificial Intelligence and Statistics,
references.authors: Dominik Janzing
references.title: Causal version of principle of insufficient reason and maxent
references.venue: arXiv preprint arXiv:2102.03906,
references.authors: Dominik Janzing Bernhard Schölkopf
references.title: Causal inference using the algorithmic Markov condition
references.venue: IEEE Transactions on Information Theory,
references.authors: Dominik Janzing Bernhard Schölkopf
references.title: Detecting confounding in multivariate linear models via spectral analysis
references.venue: Journal of Causal Inference,
references.authors: Dominik Janzing Patrik O Hoyer Bernhard Schölkopf
references.title: Telling cause from effect based on highdimensional observations
references.venue: In International Conference on Machine Learning,
references.authors: Dominik Janzing Joris Mooij Kun Zhang Jan Lemeire Jakob Zscheischler Povilas Daniušis Bastian Steudel Bernhard Schölkopf
references.title: Information-geometric approach to inferring causal directions
references.venue: Artificial Intelligence,
references.authors: Dominik Janzing Bastian Steudel Naji Shajarisales Bernhard Schölkopf
references.title: Justifying informationgeometric causal inference
references.venue: In Measures of complexity,
references.authors: Dominik Janzing Raphael Chaves Bernhard Schölkopf
references.title: Algorithmic independence of initial condition and dynamical law in thermodynamics and causal inference
references.venue: New Journal of Physics,
references.authors: Ilyes Khemakhem Diederik Kingma Ricardo Monti Aapo Hyvärinen
references.title: Variational autoencoders and nonlinear ICA: A unifying framework
references.venue: In International Conference on Artificial Intelligence and Statistics,
references.authors: Ilyes Khemakhem Ricardo Monti Robert Leech Aapo Hyvarinen
references.title: Causal autoregressive flows
references.venue: In International Conference on Artificial Intelligence and Statistics,
references.authors: Andrei N Kolmogorov
references.title: On tables of random numbers
references.venue: Sankhyā: The Indian Journal of Statistics,
references.authors: Sanghack Lee Elias Bareinboim
references.title: Structural causal bandits: where to intervene
references.venue: Advances in Neural Information Processing Systems 31,
references.authors: Felix Leeb Yashas Annadani Stefan Bauer Bernhard Schölkopf
references.title: Structural autoencoders improve representations for generation and transfer
references.venue: arXiv preprint arXiv:2006.07796,
references.authors: Erich L Lehmann George Casella
references.title: Theory of point estimation
references.venue: Springer Science & Business Media,
references.authors: Jan Lemeire Erik Dirkx
references.title: Causal models as minimal descriptions of multivariate systems
references.authors: Jan Lemeire Dominik Janzing
references.title: Replacing causal faithfulness with algorithmic independence of conditionals
references.venue: Minds and Machines,
references.authors: Francesco Locatello Stefan Bauer Mario Lucic Gunnar Raetsch Sylvain Gelly Bernhard Schölkopf Olivier Bachem
references.title: Challenging common assumptions in the unsupervised learning of disentangled representations
references.venue: In International Conference on Machine Learning,
references.authors: Chaochao Lu Bernhard Schölkopf José Miguel Hernández-Lobato
references.title: Deconfounding reinforcement learning in observational settings
references.venue: arXiv preprint arXiv:1812.10576,
references.authors: Chaochao Lu Biwei Huang Ke Wang José Miguel Hernández-Lobato Kun Zhang Bernhard Schölkopf
references.title: Sample-efficient reinforcement learning via counterfactual-based data augmentation
references.venue: arXiv preprint arXiv:2012.09092,
references.authors: Sara Magliacane Thijs van Ommen Tom Claassen Stephan Bongers Philip Versteeg Joris M Mooij
references.title: Domain adaptation by using causal inference to predict invariant conditional distributions
references.venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,
references.authors: James Mahoney
references.title: Beyond correlational analysis: Recent innovations in theory and method
references.venue: In Sociological forum,
references.authors: Michael Mistry Jonas Buchli Stefan Schaal
references.title: Inverse dynamics control of floating base systems using orthogonal decomposition
references.venue: In 2010 IEEE International Conference on Robotics and Automation,
references.authors: Ricardo Pio Monti Kun Zhang Aapo Hyvärinen
references.title: Causal discovery with general non-linear relationships using non-linear ICA
references.venue: In Uncertainty in Artificial Intelligence,
references.authors: Joris M Mooij Jonas Peters Dominik Janzing Jakob Zscheischler Bernhard Schölkopf
references.title: Distinguishing cause from effect using observational data: methods and benchmarks
references.venue: The Journal of Machine Learning Research,
references.authors: Parry Moon Domina Eberle Spencer
references.title: Field theory handbook, including coordinate systems, differential equations and their solutions
references.authors: Petteri Pajunen
references.title: Blind source separation using algorithmic information
references.venue: theory. Neurocomputing,
references.authors: Petteri Pajunen
references.title: Blind source separation of natural signals based on approximate complexity minimization
references.venue: In Proc. Int. Workshop on Independent Component Analysis and Signal Separation
references.authors: George Papamakarios Eric Nalisnick Danilo Jimenez Rezende Shakir Mohamed Balaji Lakshminarayanan
references.title: Normalizing flows for probabilistic modeling and inference
references.venue: Journal of Machine Learning Research,
references.authors: Giambattista Parascandolo Niki Kilbertus Mateo Rojas-Carulla Bernhard Schölkopf
references.title: Learning independent causal mechanisms
references.venue: In International Conference on Machine Learning,
references.authors: Judea Pearl Elias Bareinboim
references.title: External validity: From do-calculus to transportability across populations
references.venue: Statistical Science,
references.authors: Fabian Pedregosa Gaël Varoquaux Alexandre Gramfort Vincent Michel Bertrand Thirion Olivier Grisel Mathieu Blondel Peter Prettenhofer Ron Weiss Vincent Dubourg
references.title: Scikit-learn: Machine learning in python
references.venue: The Journal of Machine Learning Research,
references.authors: William S. Peebles John Peebles Jun-Yan Zhu Alexei A. Efros Antonio Torralba
references.title: The Hessian penalty: A weak prior for unsupervised disentanglement
references.venue: In ECCV,
references.authors: Jonas Peters Peter Bühlmann
references.title: Identifiability of Gaussian structural equation models with equal error variances
references.authors: Jonas Peters Joris M Mooij Dominik Janzing Bernhard Schölkopf
references.title: Causal discovery with continuous additive noise models
references.venue: Journal of Machine Learning Research,
references.authors: Jonas Peters Peter Bühlmann Nicolai Meinshausen
references.title: Causal inference by using invariant prediction: identification and confidence intervals
references.venue: Journal of the Royal Statistical Society. Series B (Statistical Methodology),
references.authors: Jonas Peters Dominik Janzing Bernhard Schölkopf
references.title: Elements of causal inference: foundations and learning algorithms
references.authors: Dinh-Tuan Pham J-F Cardoso
references.title: Blind separation of instantaneous mixtures of nonstationary sources
references.venue: IEEE Transactions on Signal Processing,
references.authors: Dinh Tuan Pham Philippe Garat
references.title: Blind separation of mixture of independent sources through a quasi-maximum likelihood approach
references.venue: IEEE Transactions on Signal Processing,
references.authors: Robert Phillips
references.title: Liouville’s theorem. Pacific
references.venue: Journal of Mathematics,
references.authors: Aditya Ramesh Youngduck Choi Yann LeCun
references.title: A spectral regularizer for unsupervised disentanglement
references.venue: arXiv preprint arXiv:1812.01161,
references.authors: Mateo Rojas-Carulla Bernhard Schölkopf Richard Turner Jonas Peters
references.title: Invariant models for causal transfer learning
references.venue: The Journal of Machine Learning Research,
references.authors: Dominik Rothenhäusler Nicolai Meinshausen Peter Bühlmann Jonas Peters
references.title: Anchor regression: Heterogeneous data meet causality
references.venue: Journal of the Royal Statistical Society: Series B (Statistical Methodology),
references.authors: Wesley C Salmon
references.title: Scientific explanation and the causal structure of the world
references.authors: B Schölkopf D Janzing J Peters E Sgouritsa K Zhang J Mooij
references.title: On causal and anticausal learning
references.venue: In 29th International Conference on Machine Learning (ICML
references.authors: Bernhard Schölkopf Francesco Locatello Stefan Bauer Nan Rosemary Ke Nal Kalchbrenner Anirudh Goyal Yoshua Bengio
references.title: Toward causal representation learning
references.venue: Proceedings of the IEEE,
references.authors: Manfred R Schroeder
references.title: Listening with two ears
references.venue: Music perception,
references.authors: Naji Shajarisales Dominik Janzing Bernhard Schoelkopf Michel Besserve
references.title: Telling cause from effect in deterministic linear dynamical systems
references.venue: In International Conference on Machine Learning,
references.authors: Xinwei Shen Furui Liu Hanze Dong Qing Lian Zhitang Chen Tong Zhang
references.title: Disentangled generative causal representation learning
references.venue: arXiv preprint arXiv:2010.02637,
references.authors: Shohei Shimizu Patrik O Hoyer Aapo Hyvärinen Antti Kerminen Michael Jordan
references.title: A linear non-Gaussian acyclic model for causal discovery
references.venue: Journal of Machine Learning Research,
references.authors: VP Skitović
references.title: On a property of a normal distribution
references.venue: In Doklady Akad. Nauk,
references.authors: Mirjam Soeten
references.title: Conformal maps and the theorem of Liouville
references.venue: PhD thesis, Faculty of Science and Engineering,
references.authors: B. Steudel D. Janzing B. Schölkopf
references.title: Causal Markov condition for submodular information measures
references.venue: Conference on Learning Theory (COLT),
references.authors: Adarsh Subbaswamy Peter Schulam Suchi Saria
references.title: Preventing failures due to dataset shift: Learning predictive models that transport
references.venue: In The 22nd International Conference on Artificial Intelligence and Statistics,
references.authors: Anisse Taleb Christian Jutten
references.title: Source separation in post-nonlinear mixtures
references.venue: IEEE Transactions on Signal Processing,
references.authors: Charles Tilly
references.title: Historical analysis of political processes
references.venue: In Handbook of sociological theory,
references.authors: Ruy Tojeiro
references.title: Liouville’s theorem revisited
references.venue: Enseignement Mathematique,
references.authors: Pauli Virtanen Ralf Gommers Travis E Oliphant Matt Haberland Tyler Reddy David Cournapeau Evgeni Burovski Pearu Peterson Warren Weckesser Jonathan Bright
references.title: Scipy 1.0: fundamental algorithms for scientific computing in python
references.venue: Nature methods,
references.authors: Julius von Kügelgen Alexander Mey Marco Loog
references.title: Semi-generative modelling: Covariate-shift adaptation with cause and effect features
references.venue: In The 22nd International Conference on Artificial Intelligence and Statistics,
references.authors: Julius von Kügelgen Alexander Mey Marco Loog Bernhard Schölkopf
references.title: Semi-supervised learning, causality, and the conditional cluster assumption
references.venue: In Conference on Uncertainty in Artificial Intelligence,
references.authors: Julius von Kügelgen Ivan Ustyuzhaninov Peter Gehler Matthias Bethge Bernhard Schölkopf
references.title: Towards causal generative scene models via competition of experts
references.venue: In ICLR 2020 Workshop on "Causal Learning for Decision Making",
references.authors: Julius von Kügelgen Yash Sharma Luigi Gresele Wieland Brendel Bernhard Schölkopf Michel Besserve Francesco Locatello
references.title: Self-supervised learning with data augmentations provably isolates content from style
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Junzhe Zhang Elias Bareinboim
references.title: Near-optimal reinforcement learning in dynamic treatment regimes
references.venue: Advances in Neural Information Processing Systems,
references.authors: K. Zhang B. Schölkopf K. Muandet Z. Wang
references.title: Domain adaptation under target and conditional shift
references.venue: Proceedings of the 30th International Conference on Machine Learning,
references.authors: Kun Zhang Laiwan Chan
references.title: Minimal nonlinear distortion principle for nonlinear independent component analysis
references.venue: Journal of Machine Learning Research,
references.authors: Kun Zhang Aapo Hyvärinen
references.title: On the identifiability of the post-nonlinear causal model
references.venue: In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence,
sections.heading: 1 Introduction
sections.text: One of the goals of unsupervised learning is to uncover properties of the data generating process, such as latent structures giving rise to the observed data. Identifiability [55] formalises this desideratum: under suitable assumptions, a model learnt from observations should match the ground truth, up to well-defined ambiguities. Within representation learning, identifiability has been studied mostly in the context of independent component analysis (ICA) [17, 40], which assumes that the observed data x results from mixing unobserved independent random variables si referred to as sources. The aim is to recover the sources based on the observed mixtures alone, also termed blind source separation (BSS). A major obstacle to BSS is that, in the nonlinear case, independent component estimation does not necessarily correspond to recovering the true sources: it is possible to give counterexamples where the observations are transformed into components yi which are independent, yet still mixed with respect to the true sources si [20, 39, 98]. In other words, nonlinear ICA is not identifiable.
  In order to achieve identifiability, a growing body of research postulates additional supervision or structure in the data generating process, often in the form of auxiliary variables [28, 30, 37, 38, 41]. In the present work, we investigate a different route to identifiability by drawing inspiration from the field of causal inference [71, 78] which has provided useful insights for a number of machine learning tasks, including semi-supervised [87, 103], transfer [6, 23, 27, 31, 61, 72, 84, 85, 97, 102, 107], reinforcement [7, 14, 22, 26, 53, 59, 60, 106], and unsupervised [9, 10, 54, 70, 88, 91, 104, 105] learning. To this end, we interpret the ICA mixing as a causal process and apply the principle of independent causal mechanisms (ICM) which postulates that the generative process consists of independent modules which do not share information [43, 78, 87]. In this context, “independent” does not refer to statistical independence of random variables, but rather to the notion that the distributions and functions composing the generative process are chosen independently by Nature [43, 48]. While a formalisation of ICM [43, 57] in terms of algorithmic (Kolmogorov) complexity [51] exists, it is not computable, and hence applying ICM in practice requires assessing such non-statistical independence
  ∗Equal contribution. Code available at: https://github.com/lgresele/independent-mechanism-analysis
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  with suitable domain specific criteria [96]. The goal of our work is thus to constrain the nonlinear ICA problem, in particular the mixing function, via suitable ICM measures, thereby ruling out common counterexamples to identifiability which intuitively violate the ICM principle.
  Traditionally, ICM criteria have been developed for causal discovery, where both cause and effect are observed [18, 45, 46, 110]. They enforce an independence between (i) the cause (source) distribution and (ii) the conditional or mechanism (mixing function) generating the effect (observations), and thus rely on the fact that the observed cause distribution is informative. As we will show, this renders them insufficient for nonlinear ICA, since the constraints they impose are satisfied by common counterexamples to identifiability. With this in mind, we introduce a new way to characterise or refine the ICM principle for unsupervised representation learning tasks such as nonlinear ICA.
  Motivating example. To build intuition, we turn to a famous example of ICA and BSS: the cocktail party problem, illustrated in Fig. 1 (Left). Here, a number of conversations are happening in parallel, and the task is to recover the individual voices si from the recorded mixtures xi. The mixing or recording process f is primarily determined by the room acoustics and the locations at which microphones are placed. Moreover, each speaker influences the recording through their positioning in the room, and we may think of this influence as ∂f/∂si. Our independence postulate then amounts to stating that the speakers’ positions are not fine-tuned to the room acoustics and microphone placement, or to each other, i.e., the contributions ∂f/∂si should be independent (in a non-statistical sense).1
  Our approach. We formalise this notion of independence between the contributions ∂f/∂si of each source to the mixing process (i.e., the columns of the Jacobian matrix Jf of partial derivatives) as an orthogonality condition, see Fig. 1 (Right). Specifically, the absolute value of the determinant |Jf |, which describes the local change in infinitesimal volume induced by mixing the sources, should factorise or decompose as the product of the norms of its columns. This can be seen as a decoupling of the local influence of each partial derivative in the pushforward operation (mixing function) mapping the source distribution to the observed one, and gives rise to a novel framework which we term independent mechanism analysis (IMA). IMA can be understood as a refinement of the ICM principle that applies the idea of independence of mechanisms at the level of the mixing function.
  Contributions. The structure and contributions of this paper can be summarised as follows: • we review well-known obstacles to identifiability of nonlinear ICA (§ 2.1), as well as existing ICM
  criteria (§ 2.2), and show that the latter do not sufficiently constrain nonlinear ICA (§ 3); • we propose a more suitable ICM criterion for unsupervised representation learning which gives rise
  to a new framework that we term independent mechanism analysis (IMA) (§ 4); we provide geometric and information-theoretic interpretations of IMA (§ 4.1), introduce an IMA contrast function which is invariant to the inherent ambiguities of nonlinear ICA (§ 4.2), and show that it rules out a large class of counterexamples and is consistent with existing identifiability results (§ 4.3);
  • we experimentally validate our theoretical claims and propose a regularised maximum-likelihood learning approach based on the IMA constrast which outperforms the unregularised baseline (§ 5); additionally, we introduce a method to learn nonlinear ICA solutions with triangular Jacobian and a metric to assess BSS which can be of independent interest for the nonlinear ICA community.
  1For additional intuition and possible violations in the context of the cocktail party problem, see Appendix B.4.
sections.heading: 2 Background and preliminaries
sections.text: Our work builds on and connects related literature from the fields of independent component analysis (§ 2.1) and causal inference (§ 2.2). We review the most important concepts below.
sections.heading: 2.1 Independent component analysis (ICA)
sections.text: Assume the following data-generating process for independent component analysis (ICA) x = f(s) , ps(s) = ∏n i=1 psi(si) , (1)
  where the observed mixtures x ∈ Rn result from applying a smooth and invertible mixing function f : Rn → Rn to a set of unobserved, independent signals or sources s ∈ Rn with smooth, factorised density ps with connected support (see illustration Fig. 2b). The goal of ICA is to learn an unmixing function g : Rn → Rn such that y = g(x) has independent components. Blind source separation (BSS), on the other hand, aims to recover the true unmixing f−1 and thus the true sources s (up to tolerable ambiguities, see below). Whether performing ICA corresponds to solving BSS is related to the concept of identifiability of the model class. Intuitively, identifiability is the desirable property that all models which give rise to the same mixture distribution should be “equivalent” up to certain ambiguities, formally defined as follows. Definition 2.1 (∼-identifiability). Let F be the set of all smooth, invertible functions f : Rn → Rn, and P be the set of all smooth, factorised densities ps with connected support on Rn. Let M ⊆ F×P be a subspace of models and let ∼ be an equivalence relation on M. Denote by f∗ps the push-forward density of ps via f . Then the generative process (1) is said to be ∼-identifiable on M if
  ∀(f , ps), (f̃ , ps̃) ∈ M : f∗ps = f̃∗ps̃ =⇒ (f , ps) ∼ (f̃ , ps̃) . (2) If the true model belongs to the model class M, then ∼-identifiability ensures that any model in M learnt from (infinite amounts of) data will be ∼-equivalent to the true one. An example is linear ICA which is identifiable up to permutation and rescaling of the sources on the subspace MLIN of pairs of (i) invertible matrices (constraint on F) and (ii) factorizing densities for which at most one si is Gaussian (constraint on P) [17, 21, 93], see Appendix A for a more detailed account. In the nonlinear case (i.e., without constraints on F), identifiability is much more challenging. If si and sj are independent, then so are hi(si) and hj(sj) for any functions hi and hj . In addition to permutation-ambiguity, such element-wise h(s) = (h1(s1), ..., hn(sn)) can therefore not be resolved either. We thus define the desired form of identifiability for nonlinear BSS as follows. Definition 2.2 (∼BSS). The equivalence relation ∼BSS on F × P defined as in Defn. 2.1 is given by
  (f , ps) ∼BSS (f̃ , ps̃) ⇐⇒ ∃P,h s.t. (f , ps) = (f̃ ◦ h−1 ◦P−1, (P ◦ h)∗ps̃) (3) where P is a permutation and h(s) = (h1(s1), ..., hn(sn)) is an invertible, element-wise function.
  A fundamental obstacle—and a crucial difference to the linear problem—is that in the nonlinear case, different mixtures of si and sj can be independent, i.e., solving ICA is not equivalent to solving BSS. A prominent example of this is given by the Darmois construction [20, 39]. Definition 2.3 (Darmois construction). The Darmois construction gD : Rn → (0, 1)n is obtained by recursively applying the conditional cumulative distribution function (CDF) transform:
  gDi (x1:i) := P(Xi ≤ xi|x1:i−1) = ∫ xi −∞ p(x ′ i|x1:i−1)dx′i (i = 1, ..., n). (4)
  The resulting estimated sources yD = gD(x) are mutually-independent uniform r.v.s by construction, see Fig. 2a for an illustration. However, they need not be meaningfully related to the true sources s, and will, in general, still be a nonlinear mixing thereof [39].2 Denoting the mixing function corresponding to (4) by fD = (gD)−1 and the uniform density on (0, 1)n by pu, the Darmois solution (fD, pu) thus allows construction of counterexamples to ∼BSS-identifiability on F × P .3 Remark 2.4. gD has lower-triangular Jacobian, i.e., ∂gDi /∂xj = 0 for i < j. Since the order of the xi is arbitrary, applying gD after a permutation yields a different Darmois solution. Moreover, (4) yields independent components yD even if the sources si were not independent to begin with.4
  2Consider, e.g., a mixing f with full Jacobian which yields a contradiction to Defn. 2.2, due to Remark 2.4. 3By applying a change of variables, we can see that the transformed variables in (4) are uniformly distributed in the open unit cube, thereby corresponding to independent components [69, § 2.2]. 4This has broad implications for unsupervised learning, as it shows that, for i.i.d. observations, not only factorised priors, but any unconditional prior is insufficient for identifiability (see, e.g., [49], Appendix D.2).
  Another well-known obstacle to identifiability are measure-preserving automorphisms (MPAs) of the source distribution ps: these are functions a which map the source space to itself without affecting its distribution, i.e., a∗ps = ps [39]. A particularly instructive class of MPAs is the following [49, 58]. Definition 2.5 (“Rotated-Gaussian” MPA). Let R ∈ O(n) be an orthogonal matrix, and denote by Fs(s) = (Fs1(s1), ..., Fsn(sn)) and Φ(z) = (Φ(z1), ...,Φ(zn)) the element-wise CDFs of a smooth, factorised density ps and of a Gaussian, respectively. Then the “rotated-Gaussian” MPA aR(ps) is
  aR(ps) = F −1 s ◦Φ ◦R ◦Φ−1 ◦ Fs . (5)
  aR(ps) first maps to the (rotationally invariant) standard isotropic Gaussian (via Φ−1 ◦ Fs), then applies a rotation, and finally maps back, without affecting the distribution of the estimated sources. Hence, if (f̃ , ps̃) is a valid solution, then so is (f̃ ◦ aR(ps̃), ps̃) for any R ∈ O(n). Unless R is a permutation, this constitutes another common counterexample to ∼BSS-identifiability on F × P . Identifiability results for nonlinear ICA have recently been established for settings where an auxiliary variable u (e.g., environment index, time stamp, class label) renders the sources conditionally independent [37, 38, 41, 49]. The assumption on ps in (1) is replaced with ps|u(s|u) = ∏n i=1 psi|u(si|u), thus restricting P in Defn. 2.1. In most cases, u is assumed to be observed, though [30] is a notable exception. Similar results exist given access to a second noisy view x̃ [28].
sections.heading: 2.2 Causal inference and the principle of independent causal mechanisms (ICM)
sections.text: Rather than relying only on additional assumptions on P (e.g., via auxiliary variables), we seek to further constrain (1) by also placing assumptions on the set F of mixing functions f . To this end, we draw inspiration from the field of causal inference [71, 78]. Of central importance to our approach is the Principle of Independent Causal Mechanisms (ICM) [43, 56, 87]. Principle 2.6 (ICM principle [78]). The causal generative process of a system’s variables is composed of autonomous modules that do not inform or influence each other.
  These “modules” are typically thought of as the conditional distributions of each variable given its direct causes. Intuitively, the principle then states that these causal conditionals correspond to independent mechanisms of nature which do not share information. Crucially, here “independent” does not refer to statistical independence of random variables, but rather to independence of the underlying distributions as algorithmic objects. For a bivariate system comprising a cause c and an effect e, this idea reduces to an independence of cause and mechanism, see Fig. 2c. One way to formalise ICM uses Kolmogorov complexity K(·) [51] as a measure of algorithmic information [43]. However, since Kolmogorov complexity is is not computable, using ICM in practice requires assessing Principle 2.6 with other suitable proxy criteria [9, 11, 34, 42, 45, 65, 75–78, 90, 110].5 Allowing for deterministic relations between cause (sources) and effect (observations), the criterion which is most closely related to the ICA setting in (1) is information-geometric causal inference (IGCI) [18, 46].6 IGCI assumes a nonlinear relation e = f(c) and formulates a notion of indepen-
  5“This can be seen as an algorithmic analog of replacing the empirically undecidable question of statistical independence with practical independence tests that are based on assumptions on the underlying distribution” [43].
  6For a similar criterion which assumes linearity [45, 110] and its relation to linear ICA, see Appendix B.1.
  dence between the cause distribution pc and the deterministic mechanism f (which we think of as a degenerate conditional pe|c) via the following condition (in practice, assumed to hold approximately),
  CIGCI(f , pc) := ∫ log |Jf (c)| pc(c)dc− ∫ log |Jf (c)| dc = 0 , (6)
  where (Jf (c))ij = ∂fi/∂cj(c) is the Jacobian matrix and | · | the absolute value of the determinant. CIGCI can be understood as the covariance between pc and log |Jf | (viewed as r.v.s on the unit cube w.r.t. the Lebesgue measure), so that CIGCI = 0 rules out a form of fine-tuning between pc and |Jf |. As its name suggests, IGCI can, from an information-geometric perspective, also be seen as an orthogonality condition between cause and mechanism in the space of probability distributions [46], see Appendix B.2, particularly eq. (19) for further details.
sections.heading: 3 Existing ICM measures are insufficient for nonlinear ICA
sections.text: Our aim is to use the ICM Principle 2.6 to further constrain the space of models M ⊆ F ×P and rule out common counterexamples to identifiability such as those presented in § 2.1. Intuitively, both the Darmois construction (4) and the rotated Gaussian MPA (5) give rise to “non-generic” solutions which should violate ICM: the former, (fD, pu), due the triangular Jacobian of fD (see Remark 2.4), meaning that each observation xi = fDi (y1:i) only depends on a subset of the inferred independent components y1:i, and the latter, (f ◦ aR(ps), ps), due to the dependence of f ◦ aR(ps) on ps (5). However, the ICM criteria described in § 2.2 were developed for the task of cause-effect inference where both variables are observed. In contrast, in this work, we consider an unsupervised representation learning task where only the effects (mixtures x) are observed, but the causes (sources s) are not. It turns out that this renders existing ICM criteria insufficient for BSS: they can easily be satisfied by spurious solutions which are not equivalent to the true one. We can show this for IGCI. Denote by MIGCI = {(f , ps) ∈ F × P : CIGCI(f , ps) = 0} ⊂ F × P the class of nonlinear ICA models satisfying IGCI (6). Then the following negative result holds. Proposition 3.1 (IGCI is insufficient for ∼BSS-identifiability). (1) is not ∼BSS-identifiable on MIGCI. Proof. IGCI (6) is satisfied when ps is uniform. However, the Darmois construction (4) yields uniform sources, see Fig. 2a. This means that (fD ◦ aR(pu), pu) ∈ MIGCI, so IGCI can be satisfied by solutions which do not separate the sources in the sense of Defn. 2.2, see footnote 2 and [39].
  As illustrated in Fig. 2c, condition (6) and other similar criteria enforce a notion of “genericity” or “decoupling” of the mechanism w.r.t. the observed input distribution.7 They thus rely on the fact that the cause (source) distribution is informative, and are generally not invariant to reparametrisation of the cause variables. In the (nonlinear) ICA setting, on the other hand, the learnt source distribution may be fairly uninformative. This poses a challenge for existing ICM criteria since any mechanism is generic w.r.t. an uninformative (uniform) input distribution.
sections.heading: 4 Independent mechanism analysis (IMA)
sections.text: As argued in § 3, enforcing independence between the input distribution and the mechanism (Fig. 2c), as existing ICM criteria do, is insufficient for ruling out spurious solutions to nonlinear ICA. We therefore propose a new ICM-inspired framework which is more suitable for BSS and which we term independent mechanism analysis (IMA).8 All proofs are provided in Appendix C.
sections.heading: 4.1 Intuition behind IMA
sections.text: As motivated using the cocktail party example in § 1 and Fig. 1 (Left), our main idea is to enforce a notion of independence between the contributions or influences of the different sources si on the observations x = f(s) as illustrated in Fig. 2d—as opposed to between the source distribution and mixing function, cf. Fig. 2c. These contributions or influences are captured by the vectors of partial derivatives ∂f/∂si. IMA can thus be understood as a refinement of ICM at the level of the mixing f : in addition to statistically independent components si, we look for a mixing with contributions ∂f/∂si which are independent, in a non-statistical sense which we formalise as follows. Principle 4.1 (IMA). The mechanisms by which each source si influences the observed distribution, as captured by the partial derivatives ∂f/∂si, are independent of each other in the sense that for all s:
  log |Jf (s)| = n∑
  i=1
  log ∥∥∥∥ ∂f∂si (s) ∥∥∥∥ (7)
  7In fact, many ICM criteria can be phrased as special cases of a unifying group-invariance framework [9]. 8The title of the present work is thus a reverence to Pierre Comon’s seminal 1994 paper [17].
  Geometric interpretation. Geometrically, the IMA principle can be understood as an orthogonality condition, as illustrated for n = 2 in Fig. 1 (Right). First, the vectors of partial derivatives ∂f/∂si, for which the IMA principle postulates independence, are the columns of Jf . |Jf | thus measures the volume of the n−dimensional parallelepiped spanned by these columns, as shown on the right. The product of their norms, on the other hand, corresponds to the volume of an n-dimensional box, or rectangular parallelepiped with side lengths ∥∂f/∂si∥, as shown on the left. The two volumes are equal if and only if all columns ∂f/∂si of Jf are orthogonal. Note that (7) is trivially satisfied for n = 1, i.e., if there is no mixing, further highlighting its difference from ICM for causal discovery.
  Independent influences and orthogonality. In a high dimensional setting (large n), this orthogonality can be intuitively interpreted from the ICM perspective as Nature choosing the direction of the influence of each source component in the observation space independently and from an isotropic prior. Indeed, it can be shown that the scalar product of two independent isotropic random vectors in Rn vanishes as the dimensionality n increases (equivalently: two high-dimensional isotropic vectors are typically orthogonal). This property was previously exploited in other linear ICM-based criteria (see [44, Lemma 5] and [45, Lemma 1 & Thm. 1]).9 The principle in (7) can be seen as a constraint on the function space, enforcing such orthogonality between the columns of the Jacobian of f at all points in the source domain, thus approximating the high-dimensional behavior described above.10
  Information-geometric interpretation and comparison to IGCI. The additive contribution of the sources’ influences ∂f/∂si in (7) suggests their local decoupling at the level of the mechanism f . Note that IGCI (6), on the other hand, postulates a different type of decoupling: one between log |Jf | and ps. There, dependence between cause and mechanism can be conceived as a fine tuning between the derivative of the mechanism and the input density. The IMA principle leads to a complementary, non-statistical measure of independence between the influences ∂f/∂si of the individual sources on the vector of observations. Both the IGCI and IMA postulates have an information-geometric interpretation related to the influence of (“non-statistically”) independent modules on the observations: both lead to an additive decomposition of a KL-divergence between the effect distribution and a reference distribution. For IGCI, independent modules correspond to the cause distribution and the mechanism mapping the cause to the effect (see (19) in Appendix B.2). For IMA, on the other hand, these are the influences of each source component on the observations in an interventional setting (under soft interventions on individual sources), as measured by the KL-divergences between the original and intervened distributions. See Appendix B.3, and especially (22), for a more detailed account.
  We finally remark that while recent work based on the ICM principle has mostly used the term “mechanism” to refer to causal Markov kernels p(Xi|PAi) or structural equations [78], we employ it in line with the broader use of this concept in the philosophical literature.11 To highlight just two examples, [86] states that “Causal processes, causal interactions, and causal laws provide the mechanisms by which the world works; to understand why certain things happen, we need to see how they are produced by these mechanisms”; and [99] states that “Mechanisms are events that alter relations among some specified set of elements”. Following this perspective, we argue that a causal mechanism can more generally denote any process that describes the way in which causes influence their effects: the partial derivative ∂f/∂si thus reflects a causal mechanism in the sense that it describes the infinitesimal changes in the observations x, when an infinitesimal perturbation is applied to si.
sections.heading: 4.2 Definition and useful properties of the IMA contrast
sections.text: We now introduce a contrast function based on the IMA principle (7) and show that it possesses several desirable properties in the context of nonlinear ICA. First, we define a local contrast as the difference between the two integrands of (7) for a particular value of the sources s. Definition 4.2 (Local IMA contrast). The local IMA contrast cIMA(f , s) of f at a point s is given by
  cIMA(f , s) = n∑ i=1 log ∥∥∥∥ ∂f∂si (s) ∥∥∥∥− log |Jf (s)| . (8)
  Remark 4.3. This corresponds to the left KL measure of diagonality [2] for √ Jf (s)⊤Jf (s).
  9This has also been used as a “leading intuition” [sic] to interpret IGCI in [46]. 10To provide additional intuition on how IMA differs from existing principles of independence of cause and
  mechanism, we give examples, both technical and pictorial, of violations of both in Appendix B.4. 11See Table 1 in [62] for a long list of definitions from the literature.
  The local IMA contrast cIMA(f , s) quantifies the extent to which the IMA principle is violated at a given point s. We summarise some of its properties in the following proposition. Proposition 4.4 (Properties of cIMA(f , s)). The local IMA contrast cIMA(f , s) defined in (8) satisfies:
  (i) cIMA(f , s) ≥ 0, with equality if and only if all columns ∂f/∂si(s) of Jf (s) are orthogonal. (ii) cIMA(f , s) is invariant to left multiplication of Jf (s) by an orthogonal matrix and to right
  multiplication by permutation and diagonal matrices.
  Property (i) formalises the geometric interpretation of IMA as an orthogonality condition on the columns of the Jacobian from § 4.1, and property (ii) intuitively states that changes of orthonormal basis and permutations or rescalings of the columns of Jf do not affect their orthogonality. Next, we define a global IMA contrast w.r.t. a source distribution ps as the expected local IMA contrast. Definition 4.5 (Global IMA contrast). The global IMA contrast CIMA(f , ps) of f w.r.t. ps is given by
  CIMA(f , ps) = Es∼ps [cIMA(f , s)] = ∫ cIMA(f , s)ps(s)ds . (9)
  The global IMA contrast CIMA(f , ps) thus quantifies the extent to which the IMA principle is violated for a particular solution (f , ps) to the nonlinear ICA problem. We summarise its properties as follows. Proposition 4.6 (Properties of CIMA(f , ps)). The global IMA contrast CIMA(f , ps) from (9) satisfies:
  (i) CIMA(f , ps) ≥ 0, with equality iff. Jf (s) = O(s)D(s) almost surely w.r.t. ps, where O(s),D(s) ∈ Rn×n are orthogonal and diagonal matrices, respectively;
  (ii) CIMA(f , ps) = CIMA(f̃ , ps̃) for any f̃ = f ◦ h−1 ◦P−1 and s̃ = Ph(s), where P ∈ Rn×n is a permutation and h(s) = (h1(s1), ..., hn(sn)) an invertible element-wise function.
  0.50 0.75 1.00 1.25 1.50 r
  /8
  0
  /8
  0.5 1.0 1.5 x
  0.6 0.4 0.2 0.0 0.2 0.4 0.6 y
  Figure 3: An example of a (non-conformal) orthogonal coordinate transformation from polar (left) to Cartesian (right) coordinates.
  Property (i) is the distribution-level analogue to (i) of Prop. 4.4 and only allows for orthogonality violations on sets of measure zero w.r.t. ps. This means that CIMA can only be zero if f is an orthogonal coordinate transformation almost everywhere [19, 52, 66], see Fig. 3 for an example. We particularly stress property (ii), as it precisely matches the inherent indeterminacy of nonlinear ICA: CIMA is blind to reparametrisation of the sources by permutation and element wise transformation.
  4.3 Theoretical analysis and justification of CIMA We now show that, under suitable assumptions on the generative model (1), a large class of spurious solutions—such as those based on the Darmois construction (4) or measure preserving automorphisms such as aR from (5) as described in § 2.1—exhibit nonzero IMA contrast. Denote the class of nonlinear ICA models satisfying (7) (IMA) by MIMA = {(f , ps) ∈ F × P : CIMA(f , ps) = 0} ⊂ F × P . Our first main theoretical result is that, under mild assumptions on the observations, Darmois solutions will have strictly positive CIMA, making them distinguishable from those in MIMA. Theorem 4.7. Assume the data generating process in (1) and assume that xi ⊥̸⊥ xj for some i ̸= j. Then any Darmois solution (fD, pu) based on gD as defined in (4) satisfies CIMA(fD, pu) > 0. Thus a solution satisfying CIMA(f , ps) = 0 can be distinguished from (fD, pu) based on the contrast CIMA. The proof is based on the fact that the Jacobian of gD is triangular (see Remark 2.4) and on the specific form of (4). A specific example of a mixing process satisfying the IMA assumption is the case where f is a conformal (angle-preserving) map. Definition 4.8 (Conformal map). A smooth map f : Rn → Rn is conformal if Jf (s) = O(s)λ(s) ∀s, where λ : Rn → R is a scalar field, and O ∈ O(n) is an orthogonal matrix. Corollary 4.9. Under assumptions of Thm. 4.7, if additionally f is a conformal map, then (f , ps) ∈ MIMA for any ps ∈ P due to Prop. 4.6 (i), see Defn. 4.8. Based on Thm. 4.7, (f , ps) is thus distinguishable from Darmois solutions (fD, pu). This is consistent with a result that proves identifiability of conformal maps for n = 2 and conjectures it in general [39].12 However, conformal maps are only a small subset of all maps for which CIMA = 0, as is apparent from the more flexible condition of Prop. 4.6 (i), compared to the stricter Defn. 4.8.
  12Note that Corollary 4.9 holds for any dimensionality n.
  Example 4.10 (Polar to Cartesian coordinate transform). Consider the non-conformal transformation from polar to Cartesian coordinates (see Fig. 3), defined as (x, y) = f(r, θ) := (r cos(θ), r sin(θ)) with independent sources s = (r, θ), with r ∼ U(0, R) and θ ∼ U(0, 2π).13 Then, CIMA(f , ps) = 0 and CIMA(fD, pu) > 0 for any Darmois solution (fD, pu) —see Appendix D for details.
  Finally, for the case in which the true mixing is linear, we obtain the following result. Corollary 4.11. Consider a linear ICA model, x = As, with E[s⊤s] = I, and A ∈ O(n) an orthogonal, non-trivial mixing matrix, i.e., not the product of a diagonal and a permutation matrix DP. If at most one of the si is Gaussian, then CIMA(A, ps) = 0 and CIMA(fD, pu) > 0.
  In a “blind” setting, we may not know a priori whether the true mixing is linear or not, and thus choose to learn a nonlinear unmixing. Corollary 4.11 shows that, in this case, Darmois solutions are still distinguishable from the true mixing via CIMA. Note that unlike in Corollary 4.9, the assumption that xi ⊥̸⊥ xj for some i ̸= j is not required for Corollary 4.11. In fact, due to Theorem 11 of [17], it follows from the assumed linear ICA model with non-Gaussian sources, and the fact that the mixing matrix is not the product of a diagonal and a permutation matrix (see also Appendix A).
  Having shown that the IMA principle allows to distinguish a class of models (including, but not limited to conformal maps) from Darmois solutions, we next turn to a second well-known counterexample to identifiability: the “rotated-Gaussian” MPA aR(ps) (5) from Defn. 2.5. Our second main theoretical result is that, under suitable assumptions, this class of MPAs can also be ruled out for “non-trivial” R. Theorem 4.12. Let (f , ps) ∈ MIMA and assume that f is a conformal map. Given R ∈ O(n), assume additionally that ∃ at least one non-Gaussian si whose associated canonical basis vector ei is not transformed by R−1 = R⊤ into another canonical basis vector ej . Then CIMA(f ◦ aR(ps), ps) > 0.
  Thm. 4.12 states that for conformal maps, applying the aR(ps) transformation at the level of the sources leads to an increase in CIMA except for very specific rotations R that are “fine-tuned” to ps in the sense that they permute all non-Gaussian sources si with another sj . Interestingly, as for the linear case, non-Gaussianity again plays an important role in the proof of Thm. 4.12.
sections.heading: 5 Experiments
sections.text: Our theoretical results from § 4 suggest that CIMA is a promising contrast function for nonlinear blind source separation. We test this empirically by evaluating the CIMA of spurious nonlinear ICA solutions (§ 5.1), and using it as a learning objective to recover the true solution (§ 5.2).
  We sample the ground truth sources from a uniform distribution in [0, 1]n; the reconstructed sources are also mapped to the uniform hypercube as a reference measure via the CDF transform. Unless
  13For different ps, (x, y) can be made to have independent Gaussian components ([98], II.B), and CIMAidentifiability is lost; this shows that the assumption of Thm. 4.7 that xi ⊥̸⊥ xj for some i ̸= j is crucial.
  otherwise specified, the ground truth mixing f is a Möbius transformation [81] (i.e., a conformal map) with randomly sampled parameters, thereby satisfying Principle 4.1. In all of our experiments, we use JAX [12] and Distrax [13]. For additional technical details, equations and plots see Appendix E. The code to reproduce our experiments is available at this link.
  5.1 Numerical evaluation of the CIMA contrast for spurious nonlinear ICA solutions
  Learning the Darmois construction. To learn the Darmois construction from data, we use normalising flows, see [35, 69]. Since Darmois solutions have triangular Jacobian (Remark 2.4), we use an architecture based on residual flows [16] which we constrain such that the Jacobian of the full model is triangular. This yields an expressive model which we train effectively via maximum likelihood.
  CIMA of Darmois solutions. To check whether Darmois solutions (learnt from finite data) can be distinguished from the true one, as predicted by Thm. 4.7, we generate 1000 random mixing functions for n = 2, compute the CIMA values of learnt solutions, and find that all values are indeed significantly larger than zero, see Fig. 4 (a). The same holds for higher dimensions, see Fig. 4 (b) for results with 50 random mixings for n ∈ {2, 3, 5, 10}: with higher dimensionality, both the mean and variance of the CIMA distribution for the learnt Darmois solutions generally attain higher values.14 We confirmed these findings for mappings which are not conformal, while still satisfying (7), in Appendix E.5.
  CIMA of MPAs. We also investigate the effect on CIMA of applying an MPA aR(·) from (5) to the true solution or a learnt Darmois solution. Results for n = 2 dim. for different rotation matrices R (parametrised by the angle θ) are shown in Fig. 4 (c). As expected, the behavior is periodic in θ, and vanishes for the true solution (blue) at multiples of π/2, i.e., when R is a permutation matrix, as predicted by Thm. 4.12. For the learnt Darmois solution (red, dashed) CIMA remains larger than zero.
  CIMA values for random MLPs. Lastly, we study the behavior of spurious solutions based on the Darmois construction under deviations from our assumption of CIMA = 0 for the true mixing function. To this end, we use invertible MLPs with orthogonal weight initalisation and leaky_tanh activations [29] as mixing functions; the more layers L are added to the mixing MLP, the larger a deviation from our assumptions is expected. We compare the true mixing and learnt Darmois solutions over 20 realisations for each L ∈ {2, 3, 4}, n = 5. Results are shown in figure Fig. 4 (d): the CIMA of the mixing MLPs grows with L; still, the one of the Darmois solution is typically higher.
  Summary. We verify that spurious solutions can be distinguished from the true one based on CIMA.
  5.2 Learning nonlinear ICA solutions with CIMA-regularised maximum likelihood
  Experimental setup. To use CIMA as a learning signal, we consider a regularised maximum-likelihood approach, with the following objective: L(g) = Ex[log pg(x)]− λCIMA(g−1, py), where g denotes the learnt unmixing, y = g(x) the reconstructed sources, and λ ≥ 0 a Lagrange multiplier. For λ = 0, this corresponds to standard maximum likelihood estimation, whereas for λ > 0, L lower-bounds the likelihood, and recovers it exactly iff. (g−1, py) ∈ MIMA. We train a residual flow g (with full Jacobian) to maximise L. For evaluation, we compute (i) the KL divergence to the true data likelihood, as a measure of goodness of fit for the learnt flow model; and (ii) the mean correlation coefficient (MCC) between ground truth and reconstructed sources [37, 49]. We also introduce (iii) a nonlinear extension of the Amari distance [5] between the true mixing and the learnt unmixing, which is larger than or equal to zero, with equality iff. the learnt model belongs to the BSS equivalence class (Defn. 2.2) of the true solution, see Appendix E.5 for details.
  Results. In Fig. 4 (Top), we show an example of the distortion induced by different spurious solutions for n = 2, and contrast it with a solution learnt using our proposed objective (rightmost plot). Visually, we find that the CIMA-regularised solution (with λ = 1) recovers the true sources most faithfully. Quantitative results for 50 learnt models for each λ ∈ {0.0, 0.5, 1.0} and n ∈ {5, 7} are summarised in Fig. 5 (see Appendix E for additional plots) . As indicated by the KL divergence values (left), most trained models achieve a good fit to the data across all values of λ.15 We observe that using CIMA (i.e., λ > 0) is beneficial for BSS, both in terms of our nonlinear Amari distance (center, lower is better) and MCC (right, higher is better), though we do not observe a substantial difference between λ = 0.5 and λ = 1.16
  Summary: CIMA can be a useful learning signal to recover the true solution. 14the latter possibly due to the increased difficulty of the learning task for larger n 15models with n = 7 have high outlier KL values, seemingly less pronounced for nonzero values of λ 16In Appendix E.5, we also show that our method is superior to a linear ICA baseline, FastICA [36].
sections.heading: 6 Discussion
sections.text: Assumptions on the mixing function. Instead of relying on weak supervision in the form of auxiliary variables [28, 30, 37, 38, 41, 49], our IMA approach places additional constraints on the functional form of the mixing process. In a similar vein, the minimal nonlinear distortion principle [108] proposes to favor solutions that are as close to linear as possible. Another example is the post-nonlinear model [98, 109], which assumes an element-wise nonlinearity applied after a linear mixing. IMA is different in that it still allows for strongly nonlinear mixings (see, e.g., Fig. 3) provided that the columns of their Jacobians are (close to) orthogonal. In the related field of disentanglement [8, 58], a line of work that focuses on image generation with adversarial networks [24] similarly proposes to constrain the “generator” function via regularisation of its Jacobian [82] or Hessian [74], though mostly from an empirically-driven, rather than from an identifiability perspective as in the present work.
  Towards identifiability with CIMA. The IMA principle rules out a large class of spurious solutions to nonlinear ICA. While we do not present a full identifiability result, our experiments show that CIMA can be used to recover the BSS equivalence class, suggesting that identifiability might indeed hold, possibly under additional assumptions—e.g., for conformal maps [39].
  IMA and independence of cause and mechanism. While inspired by measures of independence of cause and mechanism as traditionally used for cause-effect inference [18, 45, 46, 110], we view the IMA principle as addressing a different question, in the sense that they evaluate independence between different elements of the causal model. Any nonlinear ICA solution that satisfies the IMA Principle 4.1 can be turned into one with uniform reconstructed sources—thus satisfying IGCI as argued in § 3— through composition with an element-wise transformation which, according to Prop. 4.6 (ii), leaves the CIMA value unchanged. Both IGCI (6) and IMA (7) can therefore be fulfilled simultaneosly, while the former on its own is inconsequential for BSS as shown in Prop. 3.1.
  BSS through algorithmic information. Algorithmic information theory has previously been proposed as a unifying framework for identifiable approaches to linear BSS [67, 68], in the sense that commonly-used contrast functions could, under suitable assumptions, be interpreted as proxies for the total complexity of the mixing and the reconstructed sources. However, to the best of our knowledge, the problem of specifying suitable proxies for the complexity of nonlinear mixing functions has not yet been addressed. We conjecture that our framework could be linked to this view, based on the additional assumption of algorithmic independence of causal mechanisms [43], thus potentially representing an approach to nonlinear BSS by minimisation of algorithmic complexity.
  ICA for causal inference & causality for ICA. Past advances in ICA have inspired novel causal discovery methods [50, 64, 92]. The present work constitutes, to the best of our knowledge, the first effort to use ideas from causality (specifically ICM) for BSS. An application of the IMA principle to causal discovery or causal representation learning [88] is an interesting direction for future work.
  Conclusion. We introduce IMA, a path to nonlinear BSS inspired by concepts from causality. We postulate that the influences of different sources on the observed distribution should be approximately independent, and formalise this as an orthogonality condition on the columns of the Jacobian. We prove that this constraint is generally violated by well-known spurious nonlinear ICA solutions, and propose a regularised maximum likelihood approach which we empirically demonstrate to be effective in recovering the true solution. Our IMA principle holds exactly for orthogonal coordinate transformations, and is thus of potential interest for learning spatial representations [33], robot dynamics [63], or physics problems where orthogonal reference frames are common [66].
sections.heading: Acknowledgements
sections.text: The authors thank Aapo Hyvärinen, Adrián Javaloy Bornás, Dominik Janzing, Giambattista Parascandolo, Giancarlo Fissore, Nasim Rahaman, Patrick Burauel, Patrik Reizinger, Paul Rubenstein, Shubhangi Ghosh, and the anonymous reviewers for helpful comments and discussions.
  Funding Transparency Statement This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B; and by the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645.
title: Independent mechanism analysis, a new concept?