Auto-Encoders are unsupervised models that aim to learn patterns from observed data by minimizing a reconstruction cost. The useful representations learned are often found to be sparse and distributed. On the other hand, compressed sensing and sparse coding assume a data generating process, where the observed data is generated from some true latent signal source, and try to recover the corresponding signal from measurements. Looking at auto-encoders from this signal recovery perspective enables us to have a more coherent view of these techniques. In this paper, in particular, we show that the true hidden representation can be approximately recovered if the weight matrices are highly incoherent with unit 2 row length and the bias vectors takes the value (approximately) equal to the negative of the data mean. The recovery also becomes more and more accurate as the sparsity in hidden signals increases. Additionally, we empirically also demonstrate that auto-encoders are capable of recovering the data generating dictionary when only data samples are given.
This papers proposes to analyze auto-encoders under sparsity constraints of an underlying signal to be recovered.
Based on concentration inequality, the reconstruction provided for a simple class of functions is guaranteed to be accurate in l1 norm with high probability.
The proof techniques are classical, but the results seem novel as far as I know.
As an open question, could the results be given for other lp norms, in particular for infinity-norm? Indeed, this is a privileged norm for support recovery.
Presentation issues:
- section should be Section when stating for instance "section 1". Idem for eq, equation, assumption...
- bold fonts for vectors are randomly used: some care should be given to harmonizing symbols fonts.
- equations should be cited with brackets
References issues:
- harmonize citations: if you add first name for some authors add it for all references: why writing Roland Makhzani and J. Wright?
- Candes -> Cand\`es
- Consider citing "Sparse approximate solutions to linear systems", Natarajan 1995 when mentioning Amaldi and Kann 1998.
Specific comments:
page 1:
- hasn't -> has not.
page 2:
- "activation function": at this stage s_e and s_d are just functions. What is the "activation" refers to? Also a clarification on the space they act on should be stated. Idem for b_e and b_d.
- "the identity of h in eq. 1 is only well defined in the presence of l1 regularization due to the over-completeness of the dictionary" : this is implicitly stating the uniqueness of the Lasso. Not that it is well known that there are cases where the Lasso is non-unique. Please, clarify your statement accordingly.
- for simplicity b_d could be removed here.
- in (4) it would be more natural to write f_j(h_j) instead of f(h_j)
- "has is that to be bounded"-> is boundedness?
- what is l_max_j here? moreover the bold letters seem to represent vectors but this should be state explicitly somewhere.
page 3:
- what is the precise meaning of "distributed" when referring to representation
- In remark 1: the font has changed weirdly for W and h.
- "two class"->two classes
- Definition 1: again what is a precise definition of activation function?
- "if we set": bold issue.
- b should b_e in Theorem 1, right? Also, please recall the definition of the sigmoid function here. Moreover l_max and mu_h seem useless in this theorem... why referring to them?
- "if the rows of the weight matrix is"-> if the rows of the weight matrix are
page 4:
- Proposition 1 could be stated as a theorem and Th.1 as a corollary (with e=0). The same is true for proposition 2 I suspect.
- Again the influence of l_max and mu_h are none here...
- Please, provide the definition of the ReLu function here. Is this just x->x_+ ?
page 6:
- R^+m -> font issue again.
- "are maximally incoherent": what is the precise meaning of this statement?
- what the motivation for Theorem 3? This should be discussed.
- De-noising -> de-noising
- the discussion after (15) should be made more precise.
page 7:
- Figure 1 and 2 should be postponed to page 8.
- in Eq. (16) one needs to known E_h(x) and E_h_i(h_i), but I suspect this quantity are usually unknown to the practitioner. Can the author comment on that?
page 8:
- "the recovery is denoised through thresholding": where is this step analyzed?
page 9:
- figure 3: sparseness-> sparsity; also what is the activation function used here?
- "are then generate"->are then generated
- "by greedily select"->by greedily selecting
- "the the"
- "and thus pre-process"-> and thus pre-processing
Supplementary:
page 1:
- please define \sigma, and its simple properties used along the proof.
page 2:
- g should be g_j (in eq 27 - > 31)
- overall this proof relies on ingredients such as the one used for Hoeffding's inequality.
Most ingredients could be taken from standard tools on concentration (see for instance Boucheron, Lugosi, Massart: "Concentration Inequalities: A Nonasymptotic Theory of Independence", 2013).
Moreover, some elements should be factorized as they are shared among the next proofs. This should reduce the size of the supplementary dramatically.
page 7:
- Eq. (99): it should be reminded that W_ii=1 here.
- the upper bound used on \mu to get equation 105 seems to be in the wrong order.