Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In particular, we show that TCL combined with linear ICA estimates the nonlinear ICA model up to point-wise transformations of the sources, and this solution is unique -thus providing the first identifiability result for nonlinear ICA which is rigorous, constructive, as well as very general.
This paper proposes a new method for unsupervised learning of nonlinear features from non-stationary data. The features are tuned to be optimally discriminative between different time segments in the data. Interestingly, they show that this model can uniquely recover nonlinear independent components under certain conditions. 
There has been a lot of progress taking the successes of supervised techniques and extending them to unsupervised domains by defining an interesting label to predict, and this paper continues this trend by highlighting the task of discriminating between time windows. Furthermore, I think the emphasis on the importance of identifiability for nonlinear ICA and showing how it is solved in this case is a timely contribution. However, these contributions were somewhat muted by other shortcomings of the approach which make me doubt the robustness and generality of the method.   1) Temporal "Artifacts", a systematic deficiency in the setup In the MRI results, the authors use a single MLP across all data but a separate MLR for each subject and session to avoid learning "artifactual differences". This is a significant source of prior knowledge to include in the experiments and makes the comparisons unfair. In particular the comment that "none of the hidden units seem to represent artefacts, in contrast to ICA" rings a bit hollow since it seems that the elimination of artifacts was really achieved through hand-picked choice in the model.  The idea of trivial temporal artifacts that distinguish time periods is not, I think, an unlucky result of the MRI experimental setup either. I've noticed this occurring, e.g., in financial data where historical periods are trivially distinguished by the overall size of the market. If the method as described typically fails in the presence of these artifacts, I think its overall usefulness will be limited. The bright side is that putting in additional information about which types of temporal differences to learn really seems to help, so systematically exploring this effect could really strengthen the results.   2) Generality of the identifiability result unclear  As the authors point out, independent components suffer from a bad identifiability problem, so I was most intrigued by Theorem 1. However, I immediately began to wonder about the generality of the result. Does it rely on the assumption that the discriminator has "universal approximation capacity"? To me this seems like a large and poorly defined assumption.  Also in A1: "for simplicity we assume only a single function". The proof also assumes this, so it seems to be necessary for the proof? It seems like a large restriction, though it covers the non-stationary variance problem which seems to have been studied previously.  Extension 2 suggests that the extension is straightforward, but if this is true, why not include this proof in the supplementary material? To me, it looks significantly harder.   Minor point:  - In some cases, there might be a natural way to split up time windows, or you can simply split them finely enough that the stationarity holds within each window. I guess you need enough samples in each window to estimate the parameters of a totally new model. Is it easy to estimate exactly how many this should be?   edit: Thanks for the clarification about artifacts. I am still worried about it, but I think it is a general problem for unsupervised learning that could be addressed in future work. I.e., what if the best predictors of time slices are not good predictors for different tasks? Are there perhaps more systematic ways to force the method to learn interesting features (tying weights across subjects was certainly effective in this case)?