A Statistical Recurrent Model on the Manifold of Symmetric Positive Definite Matrices *
In a number of disciplines, the data (e.g., graphs, manifolds) to be analyzed are non-Euclidean in nature. Geometric deep learning corresponds to techniques that generalize deep neural network models to such non-Euclidean spaces. Several recent papers have shown how convolutional neural networks (CNNs) can be extended to learn with graph-based data. In this work, we study the setting where the data (or measurements) are ordered, longitudinal or temporal in nature and live on a Riemannian manifold -this setting is common in a variety of problems in statistical machine learning, vision and medical imaging. We show how recurrent statistical network models can be defined in such spaces. Then, we present an efficient algorithm and conduct a rigorous analysis of its statistical properties. We perform numerical experiments demonstrating competitive performance with state of the art methods but with significantly fewer parameters. We also show applications to a statistical analysis task in brain imaging, a regime where deep neural network models have only been utilized in limited ways.
OVERVIEW: At a high level, the paper extends a RNN, the Statistical Recurrent Unit (SRU) to the manifold of Symmetric Positive Definite (SPD) matrices to propose a new RNN called SPD-SRU. The idea is motivated by current trends of Geometric Deep Learning for graphs and manifolds and is of interest to the community. The main contributions of the authors lie in the technical implementation of this idea. They extend the update equations for the SRU (Eqns. 2-5), defined for a Euclidean metric space, to the SPD manifold (Eqns. 6-9). This involves (i) posing the linear or fully connected layer WX, as multiple weighted Frechet Mean (wFM) problems (ii) the bias term of the linear layer is captured a group translation operation and (iii) a ReLU non-linear activation is used in the parameter space. The trick that makes this work efficiently is a new proposed algorithm to compute the wFM efficiently using recursive closed form updates. The novelty of the paper is in the derivation of this computation with proofs of consistency and convergence. They evaluate their proposed SRU-RNN on three experiments: (1) Moving MNIST, (2) UCF11 and (3) permutation testing to detect group differences between patients with and without Parkinson's disease.   STRENGTHS: 1. The idea is well motivated and the background material, both Riemannian geometry on the SPD manifold and the SRU is covered nicely.  2. The authors have 65 citations in their submission which is indicative of the depth in which they review the literature.  3. The math is clear at a high level in motivating what they want to do, the technical challenges it involves and their solutions to these challenges.  4. They provide proofs for different mathematical properties of their proposed solution to the wFM problem for SPD matrices. 5. Their experimental evaluation shows that they achieve comparable (or better in some cases) performance for a fraction of the number of parameters to other RNN units like LSTM, SRU, TT-GRU, TT-LSTM.  WEAKNESSES: 1. The single biggest critique I have with the submission is with regards to how it is written. In two locations: (1) the proofs provided in the supplementary material and (2) the experimental evaluation, the authors can do a better job at making the paper more accessible. They skip steps or leave parts to the reader which lead to incomplete or confusing proofs. In the experimental evaluation, they provide some details but these details are probably sufficient for the authors to redo their experiments but not for a new researcher trying out their models. If the aim is to make the SPD-SRU as common and popular as the RNN/LSTM, the authors need to make these details more accessible. I agree that page restrictions play a role in how the paper is presented but all these details can and should be provided as part of the supplementary material. I specify the exact problems I refer to above: (i) In line 93, "One can easily see that, with the Stein metric, G is the set of nxn orthogonal matrices, denoted by O(n)". Should it be "a choice of G is the set"? "G is the set" seems to indicate an if and only if statement. The if direction is clear but the only if is not. (ii) All the discussions of network architectures in Sections 5.1 and 5.2 especially with regard to the number of parameters. A table outlining the final chosen network architectures (in the supplementary if needed) with specific choices of layer dimensions is required. This is important to firstly understand how you get the #params column in your tables and secondly to see if the comparison is fair. You want the reader to trust you but you also want him/her to be able to verify these statements made as facts. (iii) Proposition 1: For the \beta function, you define co-domain as S^\infty and range as \mathcal{H} = \beta (\mathcal{N}). The way I see it you have shown that for any \tilde{f} in \mathcal{H}, there exists an f and for any f there exists a \tilde{f}. What I'm missing is the connection between co-domain S^\infty and range \mathcal{H}. I'm also missing a one-to-one mapping from \mathcal{N} to \mathcal{H} to show that many f do not map to a single \tilde{f}. I think a lot of this goes away if you explicitly define what \beta is ? or what the function norm \| f \| is ? (iv) Proposition 2 is correct but Line 11, "we have used the fact that  = ..." is left as homework to the reader. It checks out with Linear algebra properties (A^-1 + B^-1)^-1 = A (A+B)^-1 B and det(AB) = det(A) det(B) used in the middle but adding this proof is a couple of extra lines of work that make the proposition complete. A secondary question is why is this proposition needed ? I do not see it explicitly being called in any future proof.  (v) Proposition 3: Line 15, I-yy^t for functional y \in \mathcal{H} is not clear. Also, how do you use it in the proof ? How do you get   \leq  ? (vi) Proposition 7: This is the most important proof of the paper and it's also the most confusing. You start by defining \alpha and \theta. Then you proceed to define g(\alpha) which I assume has some connection to Eqn.11 (not mentioned explicitly and not clear, especially the \log(\cos^2(\theta-\alpha)) term). Then you basically solve for \alpha as a function of \theta and finally write Line 31. The equation for the closed form update in Line 31 checks out but again, it's not clear how you get there. (vii) Proposition 5: Please add the approximations in the Taylor expansion \cos \theta_k = 1 - \frac{\theta_k^2}{2} + \frac{\theta_k^4}{4} + ... and \log(1-x) = -x -\frac{x^2}{2} + ... and the couple of extra steps between equality 2 and 3. (viii) Proposition 6: Not clear what inequality is used in \sum_{k=n+1}^m \sqrt(-2\log \cos \alpha_k) \leq (m-n-1) \sqrt{-\frac{2}{m-n-1} \sum... }. Not clear what happens to the 1/2c term in the approximation above line 42.  Most of the math presented here checks out and I trust the authors but complete proofs are important to verify them and for future readers. 2. A key detail that is only briefly mentioned is the implementation of the requirement that the weights are convex \geq 0 and \sum = 1. To me the biggest jump from Eqn.2 to Eqn.6 is this requirement and there is no discussion about changing the Linear layer in this manner, why this is not a sub-optimal step and why this still makes sense. 3. Experimental evaluation: The authors report time taken per epoch during training in Tables 1 and 2 for their experiments on Moving MNIST and UCF11, which is unusual. Standard practice is to report time taken at inference. It is good to be able to train a model in reasonable amount of time but for the time scales they report (a few seconds to maybe 100s per epoch) it does not seem to be a bottleneck. I would be more interested in inference times across models. Also, the permutation testing experiment in Sec.5.2 makes sense but I would propose a modification. As a machine learning practitioner, I would suggest a leave-one-out experiment where you remove one subject from the data, train your with-disease and without-disease models and assign the subject to whichever model it is closest to. This might also be a future diagnostic tool based on your work ?  MINOR COMMENTS: 1. Data is singular uncountable ? Lines 1 and 21. 2. Move references to per property in Lines 30-31. 3. Before Sec.2 motivate why SPD is important. I think current literature review is good but it is missing a survey of SPD based models especially deep learning ones. 4. Repetition in Lines 291-295 and Lines 300-305 5. Citation needed for PSRN in Line 354 6. Line 29 of Supplementary, x = \tan(\alpha)  DECISION: I have marked the overall rating as 8.  Quality: 8 Clarity: 7 Originality: 7 Significance: 8  Update post rebuttal: I have read the author response and other reviews. Based on the response and discussions with other reviewers, I have downgraded by review from 8 to 7. This is due to two reasons: (1) The authors have oversold their claim of generalizing over multiple manifolds. This should either be backed up with technical details or the text modified to reflect the current SPD + SRU setting, which might still be sufficient if convincing arguments are made for it. (2) Scalability is not clear. I am not in complete agreement with R1 that the algorithm is restricted to toy examples but I agree with his/her opinion that this has not been dealt with sufficiently by the authors.