SUBGRADIENT DESCENT LEARNS ORTHOGONAL DIC- TIONARIES
This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem. We show that a subgradient descent algorithm, with random initialization, can recover orthogonal dictionaries on a natural nonsmooth, nonconvex 1 minimization formulation of the problem, under mild statistical assumption on the data. This is in contrast to previous provable methods that require either expensive computation or delicate initialization schemes. Our analysis develops several tools for characterizing landscapes of nonsmooth functions, which might be of independent interest for provable training of deep networks with nonsmooth activations (e.g., ReLU), among other applications. Preliminary synthetic and real experiments corroborate our analysis and show that our algorithm works well empirically in recovering orthogonal dictionaries.
This paper is a direct follow-up on the Sun-Qu-Wright non-convex optimization view on the Spielman-Wang-Wright complete dictionary learning approach. In the latter paper the idea is to simply realize that with Y=AX, X being nxm sparse and A a nxn rotation, one has the property that for m large enough, the rows of X will be the sparsest element of the subspace in R^m generated by the rows of Y. This leads to a natural non-convex optimization problem, whose local optimum are hopefully the rows of X. This was proved in SWW for *very* sparse X, and then later improved in SQW to the linear sparsity scenario. The present paper refines this approach, and obtain slightly better sample complexity by studying the most natural non-convex problem (ell_1 regularization on the sphere).
I am not an expert on SQW so it is hard to evaluate how difficult it was to extend their approach to the non-smooth case (which seems to be the main issue with ell_1 regularization compared to the surrogate loss of SQW).
Overall I think this is a solid theoretical contribution, at least from the point of view of non-smooth non-convex optimization. I have some concerns about the model itself. Indeed *complete* dictionary learning seemed like an important first step in 2012 towards more general and realistic scenario. It is unclear to this reviewer whether the insights gained for this complete scenario are actually useful more generally.