SPECTRAL INFERENCE NETWORKS: UNIFYING DEEP AND SPECTRAL LEARNING
We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization. Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics. As such, they can be a powerful tool for unsupervised representation learning from video or graph-structured data. We cast training Spectral Inference Networks as a bilevel optimization problem, which allows for online learning of multiple eigenfunctions. We show results of training Spectral Inference Networks on problems in quantum mechanics and feature learning for videos on synthetic datasets. Our results demonstrate that Spectral Inference Networks accurately recover eigenfunctions of linear operators and can discover interpretable representations from video in a fully unsupervised manner.
Spectral Inference Networks, Unifying Deep and Spectral Learning
This paper presents a framework to learn eigenfunctions via a stochastic process. They are exploited in an unsupervised setting to learn representation of video data. Computing eigenfunctions can be computationally challenging in large-scale context. This paper proposes to tackle this challenge b y approximating then using a two-phase stochastic optimization process. The fundamental motivation is to merge approaches from spectral decomposition via stochastic approximation and learning an implicit representation. This is achievement with a clever use of masked gradients, Cholesky decomposition and explicit orthogonalization of resulting eigenvectors. A bilevel optimization process finds local minima as approximate eigenfunction, mimicking Borkar’97. Results are shown to correctly recover known 2d- schrodinger eigenfunctions and interpretable latent representation a video dataset, with a practical promising results using the arcade learning environment.
Positive
+ Computation of eigenfunctions on very large settings, without relying on Nystrom approximation
+ Unifying spectral decomposition within a neural net framework
Specific comments
- Accuracy issue - Shape of eigenfunctions are said to be correctly recovered, but no words indicates their accuracy. If eigenfunction values are wrong, this may be critical to the generalization of the method.
- Clarity could be improved in the neural network implementation, what is exactly done and why, when building the network
- Algorithm requires computing the jacobian of the covariance, which can be large and computationally expensive - how to scale it to large settings?
- Fundamentally, a local minimum is reached - any future work on tackling a global solution?  Perhaps by exploring varying learning rates?
- Practically, eigenfunction have an ambiguity to rotation - how is this enforced and checked during validation? (e.g., rotating eigenfunctions in Fig 1c)
- Eigenfunction of transition matrix should, if not mistaken, be smooth, whereas Fig 2a shows granularity in the eigenfunctions values (noisy red-blue maps) - Is this regularization issue, and can this be explicitly correctly?
- Perhaps a word on computational time/complexity?