Alternating minimization for dictionary learning with random initialization
We present theoretical guarantees for an alternating minimization algorithm for the dictionary learning/sparse coding problem. The dictionary learning problem is to factorize vector samples y 1 , y 2 , . . . , y n into an appropriate basis (dictionary) A * and sparse vectors x 1 * , . . . , x n * . Our algorithm is a simple alternating minimization procedure that switches between 1 minimization and gradient descent in alternate steps. Dictionary learning and specifically alternating minimization algorithms for dictionary learning are well studied both theoretically and empirically. However, in contrast to previous theoretical analyses for this problem, we replace a condition on the operator norm (that is, the largest magnitude singular value) of the true underlying dictionary A * with a condition on the matrix infinity norm (that is, the largest magnitude term). Our guarantees are under a reasonable generative model that allows for dictionaries with growing operator norms, and can handle an arbitrary level of overcompleteness, while having sparsity that is information theoretically optimal. We also establish upper bounds on the sample complexity of our algorithm.
Erratum, August 7, 2019: An earlier version of this paper appeared in NIPS 2017 which had an erroneous claim about convergence guarantees with random initialization. The main resultTheorem 3 -has been corrected by adding an assumption about the initialization (Assumption B1).
This paper proposes and analyzes an alternating minimization-based algorithm to recover the dictionary matrix and sparse coefficient matrix in a dictionary learning setting.  A primary component of the contribution here comes in the form of an alternate analysis of the matrix uncertainty (MU) selector of Belloni, Rosenbaum, and Tsybakov, to account for worst-case rather than probabilistic corruptions.
Pros:
+ The flavor of the contribution here seems to improve (i.e., relax) the conditions under which methods like this will succeed, relative to existing works.  Specifically, the motivation and result of this work amounts to specifying sufficient conditions on the vectorized infinity norm of the unknown dictionary matrix, rather than its operator norm, under which provable recovery is possible.  This has the effect of making the method potentially less dependent on ambient dimensions, especially for "typical" constructions of the (incoherent) dictionaries such as certain random generations.
+ The alternate analysis of the MU selector is independently interesting.
Cons:
- It would be interesting to see some experimental validation of the proposed method, especially ones that investigate the claimed improvements in the conditions on the unknown dictionary relative to prior efforts.  In other words, do the other efforts that state results in terms of the operator norm fail in settings where this method succeeds?  Or are the methods all viable, but just a more refined analysis here?  This is hard to determine here, and should be explored a bit, I think.
- The paper is hard to digest, partly because of notation, and partly because of some pervasive grammatical and formatting issues:
    - Line 8 of algorithm 1, as written, seems to require knowledge of the true A,x quantities to compute.  In reality, it seems this should be related somehow to the samples {y} themselves.  Can this be written a bit more clearly?
    - Condition (c4) on page 6, line 206 is confusing as written.  The dimension of x is r, and it is s-sparse, so there are more than r options for *sets* of size s; this should be r-choose-s, I guess.  The subsequent conditions are apparently based on this kind of model, and seem to be correct.
    - Why include the under brace in the first equation of line 251 on page 7?  Also, repeating the LHS is a little non-standard.
    - The "infinite samples" analysis is a little strange to me, too.  Why not simply present and analyze the algorithm (in the main body of the paper) in terms of the finite sample case?  The infinite case seems to be an analytical intermediate step, not a main contribution in itself.
    - There are many sentence fragments that are hard to parse, e.g., "Whereas..." on line 36 page 2, "While..." on line 77 page 2, and "Given..." on line 131 page 4.