In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems. Following assumptions made by prior works, we first discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced. Specific design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed. In addition to the gain gates, we further introduce overshoot gates for compensating insufficient step size in LISTA. Extensive empirical results confirm our theoretical findings and verify the effectiveness of our method.
1. Summary
The authors propose extensions to LISTA with the goal of addressing underestimation (by introducing “gain gates”) and including momentum (by introducing “overshoot gates”). The authors provide theoretical analysis for each step of their LISTA augmentations, showing that it improves convergence rate. Their theoretical statements are empirically validated and then a numerical comparison is performed between the most interesting LISTA variants. Their proposed GLISTA performs favorably, especially for networks of depth greater than 10.
2. Decision and arguments
The theory given is quite comprehensive and seems solid. Moreover it’s motivated by and helps real problems, namely L1’s well-known underestimation and LISTA’s lack of momentum. On top of that, the numerical results not only show your GLISTA outperforming others, but that by adding gates to other LISTA variants you can get better results. On the other hand (see details in the Questions section), there is some critical information missing from the empirical section which makes the results non-repeatable. 
Training is not described, i.e. hyperparameter searches and stopping criteria. A couple of the plots validating theory are difficult to understand. The datasets used are synthesized and tiny, only 1000 samples (training/testing/validation sets are not described) and 250x500 dimensions. That means your network has many more parameters than data samples unless I am mistaken.
So I have given weak reject because the paper has strong theory but weak experiments making it hard to trust the conclusions. I really want to hear back about the questions raised below.
3. Questions 
a) Just before eqn 12 you say that gains greater than 1 can be more appropriate. Am I understanding this correctly: *before* shrinkage you want to apply a gain on code elements that are “truly” nonzero, in order to cancel out the imminent L1 penalization? And you will learn to predict those code elements via parameters Lambda?
b) Just before Section 4, you describe the difference between yours and Moreau & Bruna’s momentum taps. If I understand correctly they have a matrix called W_m^(k) which is multiplied onto the previous iterate—but for each layer / time unit (k), the matrix may vary by learning (note that in the main body of their paper they don’t explicitly use ^(k) notation, but it is explained to be the same as the other LISTA parameter matrices which vary with time, and is made explicit in the appendix). So I think their momentum is in fact time-varying. Moreover it is certainly dependent on the previous iterate, using the function f(z^(k-1)) = W_m^(k) * z^(k-1). Did you mean something else by that? In what sense do you have a higher capacity—do you have more parameters?
c) If I understand correctly, in Figure 5a, 5b, for each point you have trained independently an N-layer GLISTA. Then you observe properties of your learned parameters.
For 5a: how do you compute the “output” of the gain gates? What is the input that gives this output?
For 5b: which ‘t’ is used to calculate ||W(t)D-I+U(t)A||? Or did you just train a single 15-layer GLISTA, and the x-axis is just ‘t’?
d) Could you tell us more about the hyperparameter tests for every method? How do we know it was a fair comparison? There are no error bars, but from experience I know that training LISTA type networks can be a pain. What algorithms did you use? Stopping criteria? Some plots start at zero layers (4c, 5b, 6a-c) and some start at 1 layer… 
e) Interesting that in your real-data example, the sparse vector e has more non-zeros than zeros, is that really sparse? What dictionary A did you use, then? By your own description this task does not fit the sparse coding model you have analyzed…. Am I misunderstanding?
4. Additional feedback/ minor comments
a) Add reference for DOA estimation application
b) You should put something like “we prove this in the supplement” for props and theorems. Otherwise to readers less familiar with the literature it will seem like you forgot to put a reference.
c) It seems unnecessary to put ||epsilon||<= 0 , instead of epsilon=0.
d) Basically I think you should explain what your gate is before providing proofs about it. It would be useful to see a plot of the function g and/or kappa so provide some intuition about what you are doing to the architecture (nonlinear? Linear? Threshold? Etc.). Capital “Lambda” is not defined until after theorems are provided about the functions g, kappa, f. In fact it seems like kappa is only useful for the sake of proofs. The first time through, I over-read the statement “All the learnable parameters are thus collected as Lambda…” just before Section 3.1.1 because I was thinking “Why isn’t Lambda in the definitions of f/g/kappa??”. Anyway it makes perfect sense when you define it clearly as after Eqn 18.
e) At the beginning of Section 3.2 you have a clause “the over-shoot gates act more like on the output”, which doesn’t make sense.
f) Figures should be approximately self-explanatory—but “Ratio of lower components” is not explained in the caption of Fig 4b. Although I greatly appreciate that you have made the actual plot lines/markers very easy to see! Should these results be averaged over many training attempts with error bars?