Designing neural networks for continuous-time stochastic processes is challenging, especially when observations are made irregularly. In this article, we analyze neural networks from a frame theoretic perspective to identify the sufficient conditions that enable smoothly recoverable representations of signals in L 2 (R). Moreover, we show that, under certain assumptions, these properties hold even when signals are irregularly observed. As we obtain a family of (convolutional) neural networks that satisfy these conditions, we show that we can optimize our convolution filters while constraining them so that they effectively compute a Discrete Wavelet Transform. Such a neural network can efficiently divide the time-axis of a signal into orthogonal sub-spaces of different temporal scale and localization. We evaluate the resulting neural network on an assortment of synthetic and real-world tasks: parsimonious auto-encoding, video classification, and financial forecasting.
Summary
This article considers neural networks over time-series, defined as a succession of convolutions and fully-connected layers with Leaky ReLU activations. The authors provide relatively general conditions for transformations described by such networks to admit a Lipschitz-continuous inverse. They extend these results to the case where the first layer is a convolution with irregular sampling. Finally, they show that the first convolutional filters can be chosen so as to represent a discrete wavelet transform, and provide some numerical experiments.
Main remarks
While the introduction seemed promising, and I enjoyed the writing style, I was disappointed with this article.
(1) There are many mistakes in the mathematical statements. First, in Theorem 1.1, I do not think that phi_L \circ ... \circ phi_1 \circ F is a non-linear frame, because I do not see why it should be of the form of Definition 1.2 (what would be the functions psi_n?). For the same reason, I also do not understand Theorem 1.2. In Proof 1.4, the line of equalities after « Also with the Plancherel formula » is, in my opinion, not true, because the L^2 norm of a product of functions is not the product of the L^2 norms of the functions. It also seems to me that Theorem 1.3, from [Benedetto, 1992], is incorrect: it is not the limit of t_n/n that must be larger than 2R, but the limit of N_n/n (with N_n the number of t_i's that belong to the interval [-n;n]), and there must probably be a compatibility condition between (t_n)_n and R_1, not only between (t_n)_n and R. In Proposition 1.6, I think that the equality should be a strict inequality. Additionally, I do not say that Proof 2.1 is not true, but the fact that the undersampling by a factor 2 does not prevent the operator from being a frame should be justified.
(2) The authors do not justify, in the introduction, why admitting a continuous inverse should be a crucial criterion of quality for the representation described by a neural network. Additionally, the existence of this continous inverse relies on the fact that the non-linearity that is used is a Leaky ReLU, which looks a bit like "cheating" to me, because the Lipschitz constant of the inverse of a Leaky ReLU, although finite, is large, so it seems to me that cascading several layers with Leaky ReLUs could encode a transformation with strictly positive, but still very poor frame bounds.
(3) I also do not understand why having "orthogonal outputs", as in Section 2, is really desirable; I think that it should be better justified. Also, there are probably other ways to achieve orthogonality than using wavelets in the first layer, so the fact that wavelets achieve orthogonality does not really justify why using wavelets in the first layer is a good choice, compared to other filters.
(4) I had understood in the introduction that the authors would explain how to define a (good) deep representation for data of the form (x_n)_{n\in\N}, where each x_n would be the value of a time series at instant t_n, with the t_n non-uniformly spaced. But all the representations considered in the article seem to be applicable to functions in L^2(\R) only (like in Theorem 1.4 and Theorem 2.2), and not to sequences (x_n)_{n\in\N}. There is something that I did not get here.
Minor remarks
- Fourth paragraph, third line: "this generalization frames"?
- Last paragraph before "Contributions & Organization": "that that".
- Paragraph about notations: it seems to me that what is defined as l^2(R) is denoted as l^2(Z) after the introduction.
- Last line of this paragraph: R^d_1 should be R^{d_1}, and R^d_2 R^{d_2}.
- I think "smooth" could be replaced by "continuous" (smoothness implies a notion of differentiability).
- Paragraph before Proposition 1.1: \sqrt{s} is not defined, and "is supported" should be "are supported".
- Theorem 1.1: the f_k should be phi_k.
- Definition 1.4: "piece-linear" -> "piecewise linear"?
- Lemma 1.2 and Proof 1.4: there are indices missing to \tilde h and \tilde g.
- Proof 1.4: "and finally" -> "And finally".
- Proof 1.5: I do not understand the grammatical structure of the second sentence.
- Proposition 1.4: the definition of a RNN is the same as definition 1.2 (except for the frame bounds); I do not see why such transformations should model RNNs.
- Paragraph before Proposition 1.5: "in,formation".
- Proposition 1.6: it should be said on which space the frame is injective.
- On page 8, "Lipschitz" is erroneously written (twice).
- Proposition 1.7: "ProjW,l"?
- Definition 2.1: in the "nested" property, I think that the inclusion should be the other way around.
- Before Theorem 2.1, the sentence "Such Riesz basis is proven" is unclear to me.
- Theorem 2.1: "filters convolution filters".
- I think the architecture described in Theorem 2.2 could be clarified; I am not exactly sure where all the arrows start from.
- First line of Subsection 2.3: ". is always" -> "is always".
- First paragraph of Subsection 3.2: "the the".
- Paragraph 3.2: could the previous algorithms developed for this dataset be described in slightly more detail? I also do not understand the meaning of "must solely leverage the temporal structure".
- I think that the section about numerical experiments could be slightly rewritten, so that the architecture used in each experiment is clearer. In Paragraph 3.2 in particular, I did not get why the architecture presented in Figure 6 has far fewer parameters than the one in Figure 5; it would help if the authors clearly precised how many parameters each layer contains.
- Conclusion: "we can to" -> "we can".
- Definition 4.1: p_v(s) -> p_v(t).