Neural Networks for Principal Component Analysis: A New Loss Function Provably Yields Ordered Exact Eigenvectors
In this paper, we propose a new loss function for performing principal component analysis (PCA) using linear autoencoders (LAEs). Optimizing the standard L 2 loss results in a decoder matrix that spans the principal subspace of the sample covariance of the data, but fails to identify the exact eigenvectors. This downside originates from an invariance that cancels out in the global map. Here, we prove that our loss function eliminates this issue, i.e. the decoder converges to the exact ordered unnormalized eigenvectors of the sample covariance matrix. For this new loss, we establish that all local minima are global optima and also show that computing the new loss (and also its gradients) has the same order of complexity as the classical loss. We report numerical results on both synthetic simulations, and a real-data PCA experiment on MNIST (i.e., a 60, 000×784 matrix), demonstrating our approach to be practically applicable and rectify previous LAEs' downsides.
This paper proposes and analyzes a new loss function for linear autoencoders (LAEs) whose minima directly recover the principal components of the data. The core idea is to simultaneously solve a set of MSE LAE problems with tied weights and increasingly stringent masks on the encoder/decoder matrices. My intuition is that the weights that touch every subproblem are the most motivated to find the largest principal component, the weights that touch all but one find the next largest, and so forth; I found this idea clever and elegant.
That said, I lean towards rejection, because the paper does not do a very good job of demonstrating the practical or theoretical utility of this approach. As I see it, there are two main claims that one could make to motivate this work:
1. This is a practical algorithm for doing PCA.
2. This is a step towards better understanding (and perhaps improving) nonlinear autoencoders, which do things that PCA can't.
Claim (2) might be compelling, but the authors do not make it, and it isn't self evident.
I do not find claim (1) convincing on the basis of the evidence presented. PCA is an extremely well studied problem, with lots of good solutions such as randomized SVD (Halko et al., 2009). A possible advantage of using LAEs to address the PCA problem is that they play nicely with SGD, but again, the claim that the SGD-LAE approach is superior to, say, randomized SVD on a data subsample requires evidence. Also, even if one buys the claim that LAEs are a good way to solve PCA, one can always recover the eigenvectors/eigenvalues by a final decomposition step; the authors claim that an advantage of their approach is that it does not require such "bells and whistles", but this seems like a pretty minor consideration; implementing the proposed loss function seems at least as complicated as making a call to an SVD solver, and it's hard for me to imagine a situation where the cost of that final SVD isn't dominated by the cost of solving the MSE LAE problem.
In summary, I think this paper proposes a clever and elegant solution to a problem that doesn't seem to be very important. I can't recommend acceptance unless the authors can come up with a stronger argument for why it's not just interesting but also useful.