RIEMANNIAN ADAPTIVE OPTIMIZATION METHODS
Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent (SGD), accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings. However, some of the most popular of these optimization tools − namely ADAM, ADAGRAD and the more recent AMSGRAD − remain to be generalized to Riemannian manifolds. We discuss the difficulty of generalizing such adaptive schemes to the most agnostic Riemannian setting, and then provide algorithms and convergence proofs for geodesically convex objectives in the particular case of a product of Riemannian manifolds, in which adaptivity is implemented across manifolds in the cartesian product. Our generalization is tight in the sense that choosing the Euclidean space as Riemannian manifold yields the same algorithms and regret bounds as those that were already known for the standard algorithms. Experimentally, we show faster convergence and to a lower train loss value for Riemannian adaptive methods over their corresponding baselines on the realistic task of embedding the WordNet taxonomy in the Poincaré ball.
I have enjoyed reading this paper. The paper is accessible in most cases and provides a novel optimization technique. Having said this, I have a few concerns here,
- I am not sure why the notion of product manifolds is required in developing the technique. To me, all the arguments follow without that. Even if the authors are only interested in manifolds that can be constructed in a product manner (say R^n from R),  the development can be done without explicitly going along that path. Nevertheless I may have missed something so please elaborate why product manifolds. I have to add that in many cases, the underlying Riemannian geometry cannot be derived as a product  space. For example, the SPD manifold cannot be constructed as a product space of lower dimensional geometries. 
- I have a feeling that finding the operator \Pi in many interesting cases is not easy. Given the dependency of the developments on this operator, I am wondering if the method can be used to address problems on other manifolds such as SPD, Grassmannian or Stiefel. Please provide the form of this operator for the aforementioned manifolds and comment on how the method can be used if such an operator is not at our disposal.
- While I appreciate the experiments done in the paper,  common tests (e.g., Frechet means) are not presented in the paper (see my comment below as well).  
- last but not least, the authors missed the work of   Roy et. al., "Geometry Aware Constrained Optimization Techniques for Deep Learning", CVPR'18 where RSGC with momentum and Riemannian version of RMSProp are developed. This reference should be considered and compared.
Aside from the above, please
- define v and \hat{v} for Eq.(5) 
- provide a reference for the claim at l3-p4 (claim about the gradient and Hessian)
- maybe you want to mention that \alpha -> 0 for |g_t^i| at the bottom of p4
- what does [.] mean in the last step of the algorithm presented in p7
- what is the dimensionality of the Hn in the experiments