Efficient Supervised Sparse Analysis and Synthesis Operators
In this paper, we propose a new computationally efficient framework for learning sparse models. We formulate a unified approach that contains as particular cases models promoting sparse synthesis and analysis type of priors, and mixtures thereof. The supervised training of the proposed model is formulated as a bilevel optimization problem, in which the operators are optimized to achieve the best possible performance on a specific task, e.g., reconstruction or classification. By restricting the operators to be shift invariant, our approach can be thought as a way of learning sparsity-promoting convolutional operators. Leveraging recent ideas on fast trainable regressors designed to approximate exact sparse codes, we propose a way of constructing feed-forward networks capable of approximating the learned models at a fraction of the computational cost of exact solvers. In the shift-invariant case, this leads to a principled way of constructing a form of taskspecific convolutional networks. We illustrate the proposed models on several experiments in music analysis and image processing applications.
Summary of paper: This is a technical paper on 
      supervised sparse coding. It defines a generalized form of Lasso problem 
      that equates some input variables x (e.g. the input image) to some hidden 
      variables y (e.g. the underlying, analyzed or denoised form of the image) 
      in least squares via two arbitrary weight matrices -- ||M1*x-M2*y||^2 -- 
      adding an L1 penalty on a linear mapping Omega*y and an optional L2 
      penalty on y itself. This formulation is general enough to cover both 
      analysis and synthesis (generative) problems. The paper solves this system 
      for y(x) via a proximal iteration on auxilliary variables z = Omega*y -- 
      an ADMM style method -- and it also offers fast approximation of y(x) via 
      a network that contains an unwound, truncated ADMM with re-learned 
      parameters along the lines of Gregor & LeCun [11]. Finally, it 
      proposes supervised parameter (mapping matrix) learning using stochastic 
      gradient descent over an arbitrary problem-specific loss function on y(x), 
      and to this end it derives some explicit formulae for cost gradients in 
      terms of sign(Omega*y) and the matrices and auxilliary vectors involved. 
      The method is illustrated on an image super-resolution problem and a 
      polyphonic music transcription one. It seems to give quite good results on 
      both, although I found this aspect difficult to judge and would bow to 
      more expert opinions on this. Review: This is a reasonable but 
      somewhat pedestrian technical paper. AFAIK the precise model studied is 
      new, and I agree that it is useful to have very general models of this 
      kind as references and tutorial examples, especially as there are now so 
      many variant models in L1 learning that it is difficult to keep track of 
      all of them. However the techniques used -- proximal projection, ADMM, 
      Gregor-LeCun approximation network, SGD, L2 smoothing or limiting 
      arguments for the local gradient/subgradient estimates -- are all 
      straightforward and fairly standard for recent methods, and the 
      derivations themselves contain no surprises. Also, for many of the most 
      interesting special cases, more specialized algorithms are likely to be 
      preferred: the iterations for a model this general are necessarily rather 
      expensive (equations of size = number of image pixels need to be resolved, 
      at least in principle) and depending on the exact M1, M2, Omega, 
      potentially also very ill-conditioned. Still, on balance it will be useful 
      to have this study recorded, so I am favourable to acceptance if space 
      permits. Further points: - In the first equation of 
      proposition 1, "A" should presumably be "P_{Lambda^c}". The derivation of 
      the gradient could perhaps be relegated to an appendix. Fig.1 defines 
      different A,U,B matrices - consider changing notation there to avoid 
      confusion. - More information on the effect of K in SI-NN, on the 
      (especially ambiguous?) cases where SI-NN fails but full SI-ADNN doesn't, 
      and on the degree to which the specific loss influences the learned 
      encoders would be useful. - Just writing "matrix inverse" in Q, B, 
      etc, is not very satisfying. These very large sets of equations are 
      presumably solved by conjugate gradients or similar, and this will fail if 
      the matrices become too ill-conditioned. Comment on the methods actually 
      used.