We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision. Many prior approaches to this problem have focused on learning "disentangled" representations so that as individual factors vary in a new domain, only a portion of the representation need be updated. In this work, we seek the generalization power of disentangled representations, but relax the requirement of explicit latent disentanglement and instead encourage linearity of individual factors of variation by requiring them to be manipulable by learned linear transformations. We dub these transformations latent canonicalizers, as they aim to modify the value of a factor to a pre-determined (but arbitrary) canonical value (e.g., recoloring the image foreground to black). Assuming a source domain with access to meta-labels specifying the factors of variation within an image, we demonstrate experimentally that our method helps reduce the number of observations needed to generalize to a similar target domain when compared to a number of supervised baselines.
This paper proposes a method for unsupervised learning  of data representations that can be manipulated to remove factors of variation via linear transformations. These transformations are called canonicalizations in the paper. The canonicalizations are trained such that images for arbitrary values of the corresponding factor of variation are transformed into images with a fixed, canonical, value for that factor. The paper proposes a model architecture based on a denoising autoencoder, where the canonicalizations are applied to the encoded representation. It also proposes a loss function and sampling scheme for training the model. The paper demonstrates the method on the dSprites dataset, showing that it can effectively learn linear canonicalizations, and that multiple of these canonicalizations can be applied to the same image representation. The paper goes on to test the method on a digit classification task, where the model is trained to learn a representation in a simulator for SVHN data where the transformations to be canonicalized can be controlled, and  used to train a classifier on unseen real data from the SVHN test set.
I think this paper should be accepted as it proposes a novel idea, which does not seem too difficult to reproduce, describes a simulator for synthetic data for digit recognition, and proposes it as a benchmark for learning representations, and provides experimental results that help in better understanding the representation learned by the model.
A couple things I thought were missing in the paper:
Did you try applying the classification loss to both the encoded representation and the canonicalized representation at the same time?
For Figure 2, Did you try applying canonicalizations in different orders? Do they give the same results?
Instead of trying to learn idempotency by gradient descent, you could try to parametrize the canonicalizations with a matrix X, such that C =  X (X^T X)^{-1} X^T. C will be idempotent (although restricted to be symmetric). There might be other constructions that are more efficient and less restrictive.
I'm not sure I understand the PCA figures. Can you please explain how the first principal component was used to generate them?
Minor comments:
* "data  tripets" on page 2
* Figure 5 should appear after Figure 4.