Understanding Trainable Sparse Coding with Matrix Factorization
Sparse coding is a core building block in many data analysis and machine learning pipelines. Typically it is solved by relying on generic optimization techniques, such as the Iterative Soft Thresholding Algorithm and its accelerated version (ISTA, FISTA). These methods are optimal in the class of first-order methods for non-smooth, convex functions. However, they do not exploit the particular structure of the problem at hand nor the input data distribution. An acceleration using neural networks, coined LISTA, was proposed in cite{Gregor10}, which showed empirically that one could achieve high quality estimates with few iterations by modifying the parameters of the proximal splitting appropriately. In this paper we study the reasons for such acceleration. Our mathematical analysis reveals that it is related to a specific matrix factorization of the Gram kernel of the dictionary, which attempts to nearly diagonalise the kernel with a basis that produces a small perturbation of the ball. When this factorization succeeds, we prove that the resulting splitting algorithm enjoys an improved convergence bound with respect to the non-adaptive version. Moreover, our analysis also shows that conditions for acceleration occur mostly at the beginning of the iterative process, consistent with numerical experiments. We further validate our analysis by showing that on dictionaries where this factorization does not exist, adaptive acceleration fails.
This work presents an analysis of LISTA, which originally proposes to accelerate sparse coding algorithms with some prior on the structure of the problem. The authors here propose a solid analysis of the acceleration performance of LISTA, using a specific matrix factorisation of the dictionary. The analysis is well structured, and provides interesting insights. It would have been good to tie more closely these insights to specific properties of data or input distributions. The learned dictionary results in Section 3.3 are not very clear: is the dictionary learned with a sort of alternating minimisation strategy that would include LISTA as sparse coding step? Or is it only the sparse coding that is studied, with a dictionary that has been learned a priori? Overall, the paper does not propose a new algorithm and representation, but provides key insights on a well-known and interesting acceleration method on sparse coding. This is quite a nice work. The title seems however a bit confusing as *neural sparse coding* is actually rather *LISTA*, or *neural network acceleration of sparse coding* - basically, it is not immediate to understand what *neural sparse coding* means...
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Understanding Trainable Sparse Coding with Matrix Factorization
Sparse coding is a core building block in many data analysis and machine learning pipelines. Typically it is solved by relying on generic optimization techniques, such as the Iterative Soft Thresholding Algorithm and its accelerated version (ISTA, FISTA). These methods are optimal in the class of first-order methods for non-smooth, convex functions. However, they do not exploit the particular structure of the problem at hand nor the input data distribution. An acceleration using neural networks, coined LISTA, was proposed in cite{Gregor10}, which showed empirically that one could achieve high quality estimates with few iterations by modifying the parameters of the proximal splitting appropriately. In this paper we study the reasons for such acceleration. Our mathematical analysis reveals that it is related to a specific matrix factorization of the Gram kernel of the dictionary, which attempts to nearly diagonalise the kernel with a basis that produces a small perturbation of the ball. When this factorization succeeds, we prove that the resulting splitting algorithm enjoys an improved convergence bound with respect to the non-adaptive version. Moreover, our analysis also shows that conditions for acceleration occur mostly at the beginning of the iterative process, consistent with numerical experiments. We further validate our analysis by showing that on dictionaries where this factorization does not exist, adaptive acceleration fails.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work presents an analysis of LISTA, which originally proposes to accelerate sparse coding algorithms with some prior on the structure of the problem. The authors here propose a solid analysis of the acceleration performance of LISTA, using a specific matrix factorisation of the dictionary. The analysis is well structured, and provides interesting insights. It would have been good to tie more closely these insights to specific properties of data or input distributions. The learned dictionary results in Section 3.3 are not very clear: is the dictionary learned with a sort of alternating minimisation strategy that would include LISTA as sparse coding step? Or is it only the sparse coding that is studied, with a dictionary that has been learned a priori? Overall, the paper does not propose a new algorithm and representation, but provides key insights on a well-known and interesting acceleration method on sparse coding. This is quite a nice work. The title seems however a bit confusing as *neural sparse coding* is actually rather *LISTA*, or *neural network acceleration of sparse coding* - basically, it is not immediate to understand what *neural sparse coding* means...
assistant
This work presents an analysis of LISTA, which originally proposes to accelerate sparse coding algorithms with some prior on the structure of the problem. The authors here propose a solid analysis of the acceleration performance of LISTA, using a specific matrix factorisation of the dictionary. The analysis is well structured, and provides interesting insights. It would have been good to tie more closely these insights to specific properties of data or input distributions. The learned dictionary results in Section 3.3 are not very clear: is the dictionary learned with a sort of alternating minimisation strategy that would include LISTA as sparse coding step? Or is it only the sparse coding that is studied, with a dictionary that has been learned a priori? Overall, the paper does not propose a new algorithm and representation, but provides key insights on a well-known and interesting acceleration method on sparse coding. This is quite a nice work. The title seems however a bit confusing as *neural sparse coding* is actually rather *LISTA*, or *neural network acceleration of sparse coding* - basically, it is not immediate to understand what *neural sparse coding* means...