Randomly initialized first-order optimization algorithms are the method of choice for solving many high-dimensional nonconvex problems in machine learning, yet general theoretical guarantees cannot rule out convergence to critical points of poor objective value. For some highly structured nonconvex problems however, the success of gradient descent can be understood by studying the geometry of the objective. We study one such problemcomplete orthogonal dictionary learning, and provide converge guarantees for randomly initialized gradient descent to the neighborhood of a global optimum. The resulting rates scale as low order polynomials in the dimension even though the objective possesses an exponential number of saddle points. This efficient convergence can be viewed as a consequence of negative curvature normal to the stable manifolds associated with saddle points, and we provide evidence that this feature is shared by other nonconvex problems of importance as well.
The authors analyze the convergence performance of Riemannian gradient descent based algorithm for the dictionary learning problem with orthogonal dictionary and sparse factors. They demonstrate a polynomial time convergence from a random initialization for a smooth surrogate objective for the original non-smooth one. The problem and the analysis are of interest, but I have several questions regarding the paper as follows.
My first concern is that the analysis is on a smooth surrogate of the non-smooth sparse minimization for solving the dictionary learning problem, so it is not clear what is relationship between the global minimizer of the smooth problem to the underlying true dictionary. More specifically, how far is the global minimizer of problem (1) or (2) to the true dictionary parameter, and whether they share (approximately) the space or components regarding the sparse factors. Without clarifying this, it is not well motivated why we are interested in studying the problem considered in this paper at the beginning. Intuitively, since the recovered factors are not sparse anymore, it will impact the dictionary accordingly due to the linear mapping, which may lead to a very different set of dictionary components. Thus, explicit explanation is necessary here to avoid such degenerate case.
My second concern is the eligibility of assuming the dictionary A to be an identity matrix and extending it to the general orthogonal matrix case. The analysis uses the fact that rows of A are canonical basis, i.e., each row only has one non-zero entry. I do not see a straightforward extension by replacing A to be an orthogonal matrix as the authors claimed on page 3, since then the inner product of one row of A and one column of X is not just the corresponding entry of the column of X. It will be helpful if the authors can explain this explicitly or adjust the analysis accordingly to make this valid.
Another issue is the clarity of the paper. Some statements in the paper are not very clear. Form example, on page 3, third paragraph of Section 3, what does row(Y) = row(X_0) mean? Also, in eqn (1), y_k means k-th column of Y, and in eqn (2), q_i means i-th entry of q? Since both are bold lower case letters, clear distinction will help. Moreover, the reference use (. ) instead [ .], which can be confusing sometimes.