We propose Power Slow Feature Analysis, a gradient-based method to extract temporally slow features from a high-dimensional input stream that varies on a faster time-scale, as a variant of Slow Feature Analysis (SFA). While displaying performance comparable to hierarchical extensions to the SFA algorithm, such as Hierarchical Slow Feature Analysis, for a small number of output-features, our algorithm allows fully differentiable end-to-end training of arbitrary differentiable approximators (e.g., deep neural networks). We provide experimental evidence that PowerSFA is able to extract meaningful and informative low-dimensional features in the case of (a) synthetic low-dimensional data, (b) visual data, and also for (c) a general dataset for which symmetric non-temporal relations between points can be defined.
Summary
The manuscript proposes to use power iterations in an approximate "whitening layer" to optimize the slowness objective of SFA in a very general setting. A set of experiments illustrates that this way of doing nonlinear SFA is meaningful.
Quality
Although the idea is pretty straight forward and the paper shows qualitative results on a number of datasets, the relative merit of the approach is empirically not well characterized.
Clarity
The manuscript is in general well written and the technical content is well accessible. However the description of the whitening layer implementation needs some more details.
Originality
The idea of using a whitening layer together with the slowness objective has not been explored before. There is a second ICLR 2019 submission (Pfau et al.) with a very similar idea, though.
Empirical Evaluation
The approximate whitening should lead to a trade-off between whitening and slowness optimization. I miss an experiment illustrating that trade-off. Also the comparison to nonlinear SFA using expansion or kernelization of hierarchical SFA is empirically not properly characterized. In the end, if one takes the slowness objective seriously, one would use the method yielding slower results.
Significance
The manuscript introduces a way of running nonlinear SFA with approximate constraints in a general deep learning setting with a differentiable implementation using a dedicated whitening layer based on power iterations.
Reproducibility
The data is either synthetic or publicly available. The Keras implementation of the PowerWhitening layer as well as the entire neural network along with its optimization schedule is not shared. Hence, there should be some effort involved to reproduce the experiments.
Pros and Cons
1+) The idea of an approximate whitening layer is conceptually simple and clear.
2-) The description of the practical implementation of the power iteration is slightly imprecise.
3-) The algorithm scales badly in the number of output dimensions. This scaling is bad in a computational sense and also in a statistical sense.
Details
a) Section 6.1: Why do you need to add the noise term? What is the statistical meaning of this added noise?
b) Section 6.1: the solutions if comparable -> the solutions is comparable
c) References: Shaham -> ICLR 2018 paper
d) References: nyström -> Nyström
e) The name for the algorithm "Power SFA" is a little bit bold.