Optimization on manifold has been widely used in machine learning, to handle optimization problems with constraint. Most previous works focus on the case with a single manifold. However, in practice it is quite common that the optimization problem involves more than one constraints, (each constraint corresponding to one manifold). It is not clear in general how to optimize on multiple manifolds effectively and provably especially when the intersection of multiple manifolds is not a manifold or cannot be easily calculated. We propose a unified algorithm framework to handle the optimization on multiple manifolds. Specifically, we integrate information from multiple manifolds and move along an ensemble direction by viewing the information from each manifold as a drift and adding them together. We prove the convergence properties of the proposed algorithms. We also apply the algorithms into training neural network with batch normalization layers and achieve preferable empirical results.
The title is misleading, since only two particular manifolds are studied in this work. In addition, the proposed methods cannot be applied to a larger or a general class of manifolds. Therefore, you should update the title.
There are multiple problem definitions proposed in the paper. They are not compatible with each other and also with the proposed methods. In addition, some of the proposed problem definitions are incorrect, as explained below:
You should be more precise about the definition of the manifold you consider in this paper. For example, in equation (1), please define your manifold of interest more precisely checking some standard textbooks.
Please define intersection of manifolds, what do you mean by which intersection of which type of manifolds?
In the contribution (1); the paper does not introduce an algorithm to deal with optimization on with multiple manifolds, but for a particular type of individual manifolds.
In the contribution (2): It is not clear why and how the proposed method can be applied to optimization on manifolds with momentum (what do you mean by use of momentum here?), and regularization (what do you mean by regularization?). There are many problems with this claim, but you can simply consider that applying momentum and regularization will affect the geometry of loss landscape.
Definition of retraction is not precise, please fix it.
What is L in equation (2)?
Please define neighborhood in U_x in Lemma 2.1.
What is || ||in Lemma 2.1?
As you also noticed on page 3, an intersection of manifolds may not be a manifold. Then, your proposed first problem (1) fails. Therefore, you should completely change your claims on your problem definitions and contributions.
What do you mean by “We add a drift which contains information from the other manifold to the original gradient descent on manifold”? What is “the information from the manifold”? In equations (3) and (4), you just apply optimization on manifolds individually. 
How do you compute/determine a_k^(1) and a_k^(2)? How do they affect the theoretical and experimental results?
In your claim “From the construction of bk, we can see that the smaller the correlation between gradf(xk) and hk is, the smaller effect the information from M2 brings”, it is not clear how “the information from M2” affects? First, again, what is “the information”? Second, b_k^(1) and b_k^(2) are computed for individual manifolds separately. Then, how “the information” make an effect?
 In Theorem 2.2, what do you mean by “then xk convergence to a local minimizer“?
What is <,> in Theorem 2.2?
What is ^ in Theorem 2.3?
What is v in proof 6?
What is an engine value?
What does P (1) xk gradf(yk) denote in computation of h_k? For example, gradf(yk) is a vector on tangent space of the second manifold at yk. Then, how do you project orthogonally this projected vector to the tangent space of the first manifold at xk? 
They may be completely different geometries, and such an “orthogonal projection” may not exist in general. Then, how do you compute and calculate that projection?
All the theoretical results given in the paper are not about convergence of parameters on a manifold at the intersection or product of manifolds but for an individual manifold. For example, x and y belong to manifolds M1 and M2, and convergence results is about x. How are they related to parameters at the intersection or product of manifolds?
The statements regarding batch normalization are confusing and also sound incorrect:
Do you apply batch normalization on weights on BN(w)?
Please explain what you mean by “BN(w) has same image space on G(1, n) and St(n, 1)“. There are not such results in the papers Cho & Lee (2017); Huang et al. (2017) you cited for these results.
What do you mean by “applying optimization on manifold to batch normalization problem”?
In your statement “However, the property of these two manifold implies that we can actually optimize on the intersection of two manifolds”. Please explain how does this property imply this result more precisely?
Please define “Grassmann manifold G(1, n)“ more precisely. In your notation, together with explanation of the notation for St(n,p), G(1,n) is like a set of 1xn dimensional row vectors, while St(n,1) is an nx1 dimensional column vector, Then, their intersection is an empty set and your proposal for optimization on a vector on their intersection is wrong. 
Notation and definitions used in (9) are wrong and confusing. Please check and revise them.
In the whole paper, the problem, method, solutions, theorems, and contributions are proposed for optimization using parameters which belong to intersection of some manifolds. Then, suddenly, you start considering optimization on product manifolds, and give the results for that;
What does the statement “Then we apply Algorithm 1 to update parameters, which means we optimize on a product manifold” mean?
What do “G(1, k1) × · · · G(1, kn)” and “St(k1, 1) × · · · St(kn, 1)” denote?
Don’t you perform optimization on intersection of manifolds? Why do you ignore your original problem and methods, and consider this problem? 
In addition, how do you use your Algorithm 1 for optimization on product manifolds? Optimization on intersection on manifolds and product manifolds are completely different problems. If they are same or related to each in particular cases in your specific definitions, then you should provide these definitions more precisely.
What do you mean by optimization on product manifold of weights of all layers? If you compute a product manifold for spaces of all layers, then you simply perform a shallow optimization on a huge matrix containing millions of dimensions according to this definition. First, how do you do that? Second, how can you train a large network using this approach?
In the experiments, please first give variance of errors. These results are statistically insignificant.
Which problem is solved to perform these experiments is not also clear (see above).
The results reported in the paper are also not good, may be due to the mathematical and algorithmic  problems and errors mentioned above. Please clarify them, and provide additional results, especially using other datasets (small scale mnist and large scale imagenet), and networks (mlp, vgg, resnet etc.)
Related work is also incomplete, such that many traditional and recent work on optimization on multiple manifolds are omitted.