Independent causal mechanisms are a central concept in the study of causality with implications for machine learning tasks. In this work we develop an algorithm to recover a set of (inverse) independent mechanisms relating a distribution transformed by the mechanisms to a reference distribution. The approach is fully unsupervised and based on a set of experts that compete for data to specialize and extract the mechanisms. We test and analyze the proposed method on a series of experiments based on image transformations. Each expert successfully maps a subset of the transformed data to the original domain, and the learned mechanisms generalize to other domains. We discuss implications for domain transfer and links to recent trends in generative modeling.
This paper describes a setting in which a system learns collections of inverse-mapping functions that transform altered inputs to their unaltered "canonical" counterparts, while only needing unassociated and separate sets of examples of each at training time.  Each inverse map is an "expert" E akin to a MoE expert, but instead of using a feed-forward gating on the input, an expert is selected (for training or inference) based on the value of a distribution-modeling function c applied to the output of all experts:  The expert with maximum value c(E(x)) is selected.  When c is an adversarially trained discriminator network, the experts learn to model the different transformations that map altered images back to unaltered ones.  This is demonstrated using MNIST with a small set of synthetic translations and noise.
The fact that these different inverse maps arise under these conditions is interesting --- and Figure 5 is quite convincing in showing how each expert generalizes.  However, I think the experimental conditions are very limited:  Only one collection of transformations is studied, and on MNIST digits only.  In particular, I found the fact that only one of ten transformations can be applied at a time (as opposed to a series of multiple transforms) to be restrictive.  This is touched on in the conclusion, but to me it seems fundamental, as any real-world new example will undergo significantly more complex processes with many different variables all applied at once.
Another direction I think would be interesting, is how few examples are needed in the canonical distribution?  For example, in MNIST, could the canonical distribution P be limited to just one example per digit (or just one example per mode / style of digit, e.g. "2" with loop, and without loop)?  The different handwriters of the digits, and sampling and scanning process, may themselves constitute in-the-wild transformations that might be inverted to single (or few) canonical examples --- Is this possible with this mechanism?
Overall, it is nice to see the different inverse maps arise naturally in this setting.  But I find the single setting limiting, and think the investigation could be pushed further into less restricted settings, a couple of which I mention above.
Other comments:
- c is first described to be any distribution model, e.g. the autoencoder described on p.5.  But it seems that using such a fixed, predefined c like the autoencoder may lead to collapse:  What is preventing an expert from learning a single constant mode that has high c value?  The adversarially trained c doesn't suffer from this, because presumably the discriminator will be able to learn the difference between a single constant mode output and the distribution P.  But if this is the case, it seems a critical part of the system, not a simple implementation choice as the text seems to say.
- The single-net baseline is good, but I'd like to get a clearer picture of its results.  p.8 says this didn't manage to "learn more than one inverse mechanism" --- Does that mean it learns to invert a single mechanism (that is always translates up, for example, when presented an image)?  Or that it learned some mix of transforms that didn't seem to generalize as well?  Or does it have some other behavior?  Also, I'm not entirely clear on how it was trained wrt c --- is argmax(c(E(x)) always just the single expert?  Is c also trained adversarially?  And if so, is the approximate identity initialization used?