We propose and investigate the design of a convolutional layer where kernels are parameterized functions. This layer aims at being the input layer of convolutional neural networks for audio applications. The kernels are defined as functions having a band-pass filter shape, with a limited number of trainable parameters. Building on the literature on this topic, we confirm that networks having such an input layer can achieve state-of-the-art accuracy on several audio classification tasks. We propose new filter functions and explore the effect of different parameters on the network accuracy and learning ability. This approach reduces the number of weights to be trained and enables larger kernel sizes, an advantage for audio applications. Furthermore, the learned filters bring additional interpretability and a better understanding of the data properties exploited by the network.
This paper proposes to specify the first layer of a CNN for audio applications with predefined filterbanks from the signal processing community. Those latter are only specified by a limited number of parameters, such as the bandwidth or the central frequency of the filter, and those parameters are then optimized through the standard back-propagation algorithm. Some accuracy improvements are obtained on non trivial datasets.
I think there are a lot of interesting ideas and the numerical improvements seem consistent with the method. However, I find that this study would benefit of more careful comparisons to understand which particular component is responsible for some of their success! Also, I think some relevant papers are missing in the introduction.
Pros : 
- Good numerical performances.
- Interesting study of the impact of predefined filters; an analysis at the end of the paper(bandwidth, principal frequency chosen by the algorithm) is shown, which is a positive aspect of the paper.
Cons :
- Several attempts to employ hybrid architectures (as defined in the text) have been already proposed. References to hybrid architecture from Mallat's group are missing, e.g.:  https://arxiv.org/abs/1809.06367/ https://arxiv.org/abs/1605.06644 . Another line of work concerns the steerable filters, which is another manner to parametrized the filters and learn them (e.g.: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Jacobsen_Structured_Receptive_Fields_CVPR_2016_paper.pdf ) Another manner could be to directly learn the filters as wavelets: https://arxiv.org/pdf/1811.06115.pdf . I agree some of those references are only considering images, but those methods are definitely not specific to them.
- Comparing the number of parameters of hybrid and non hybrid architectures is meaningless in this setting, as in all the experiments, the number of parameters of the layers above the first layer are kept identical: one only sees the difference due to the first layer, whose kernel is indeed relatively high-dimensional.
- Also, my understanding is that the general pipeline is slower: indeed, a parameter update aims to compute @f/@w=@f/@x*@x/@w. The computation of the term @f/@x is unavoidable and is identical to its non-hybrid counter-part. However, @x/@w might be sometimes higher because the computations can involve potentially more complex functions(e.g., exponential, cos, sin contrary to linear functions). Would you mind to clarify this thought?(A small fair timing comparison would be welcome!)
- Furthermore, the improvement in performances is clearly thanks to those a-priori incorporated. As stated in the text, many works propose to initialize the CNN with a specific filter bank. Have the authors tried to compare their performances if the first layer is simply initialized with those filters and then freely evolve? I feel this is missing and would make the claim of the paper stronger. If this has been already done, please highlight it in the text.
- Abstract: it is claimed that this technique leads to a training speedup (i.e., less epochs) but I do not understand where this is shown.
- Section 3: Sometimes(e.g., AudioMnist), the hybrid training pipeline is quite different from the original implementation, for instance, because of the use of ADAM when the original implementation was using SGD. Did the use of a different optimizer affect the performances?(e.g., SGD?)
- Section 3.4: Why not comparing with data augmented settings?
Suggestions of improvement:
- I would have liked to see a littlewood-paley plot (eg, the sum of the modulus of the filters in the frequency domain) to understand better the distribution of the filters in the Fourier domain, in particular w.r.t. the high-frequency.
- "the output maybe too redundant"(page 5) - I don't understand why this would be an issue? In this case, the network should decide which coefficients to discard if the classifier is good enough, shouldn't it?
Post-discussion:
R1 made several relevant comments about the technical novelty and my concerns weren't fully solved. Thus I decided to maintain my score.