Prior work on speech and audio processing has demonstrated the ability to obtain excellent performance when learning directly from raw audio waveforms using convolutional neural networks (CNNs). However, the exact inner workings of a CNN remain unclear, which hinders further developments and improvements into this direction. In this paper, we theoretically analyze and explain how deep CNNs learn from raw audio waveforms and identify potential limitations of existing network structures. Based on this analysis, we further propose a new network architecture (called SimpleNet), which offers a very simple but concise structure and high model interpretability.
The paper proposes a CNN-based based approach for speech processing using raw waveforms as input. An analysis of convolution and pooling layers applied on waveforms is first presented. An architecture called SimpleNet is then presented and evaluated on two speech tasks: emotion recognition and gender classification. 
This paper propose a theoretical analysis of convolution and pooling layers to motivate the SimpleNet architecture. To my understanding, the analysis is flawed (see comments below). The SimpleNet approach is interesting but not sufficiently backed with experimental results. The network analysis is minimal and provides almost no insights. I therefore recommend to reject the paper. 
Detailed comments:
Section 1:
* “Therefore, it remains unknown what actual features CNNs learn from waveform”. This is not true, several works on speech recognition have shown that a convolution layer taking raw speech as input can be seen as a bank of learned filters. For instance in the context of speech recognition, [9] showed that the filters learn phoneme-specific responses, [10] showed that the learned filters are close to Mel filter banks and [7] showed that the learned filters are related to MRASTA features and Gabor filters. The authors should discuss these previous works in the paper.
Section 2:
* Section 2.1 seems unnecessary, I think it’s safe to assume that the Shannon-Nyquist theorem and the definition of convolution are known by the reader.
* Section 2.2.2 & 2.2.3: I don't follow the justification that stacking convolutions are not needed: the example provided is correct if two convolutions are directly stacked without non-linearity, but the conclusion does not hold with a non-linearity and/or a pooling layer between the convolutions: two stacked convolutions with non-linearities are not equivalent to a single convolution. To my understanding, the same problem is present for the pooling layer: the presented conclusion that pooling introduces aliasing is only valid for two directly stacked pooling layers and is not correct for stacked blocks of convolution/pooling/non-linearity.
* Section 2.2.5: The ReLU can be seen as a half-wave rectifier if it is applied directly to the waveform. However, it is usually not the case as it is applied on the output of the convolution and/or pooling layers. Therefore I don’t see the point of this section. 
* Section 2.2.6: In this section, the authors discuss the differences between spectrogram-based and waveforms-based approaches, assuming that spectrogram-based approach have fixed filters. But spectrogram can also be used as input to CNNs (i.e. using learned filters) for instance in speech recognition [1] or emotion recognition [11]. Thus the comparison could be more interesting if it was between spectrogram-based and raw waveform-based approaches when the filters are learned in both cases.  
Section 3:
* Figure 4 is very interesting, and is in my opinion a stronger motivation for SimpleNet that the analysis presented in Section 2.
* Using known filterbanks such as Mel or Gammatone filters as initialization point for the convolution layer is not novel and has been already investigated in [7,8,10] in the context of speech recognition. 
Section 4:
* On emotion recognition, the results show that the proposed approach is slightly better, but there is some issues: the average recall metric is usually used for this task due to class imbalance (see [1] for instance). Could the authors provide results with this metric ? Also IEMOCAP is a well-used corpus for this task, could the authors provide some baselines performance for comparison (e.g. [11]) ? 
* For gender classification, there is no gain from SimpleNet compared to the baselines. The authors also mention that some utterances have overlapping speech. These utterances are easy to find from the annotations provided with the corpus, so it should be easy to remove them for the train and test set. Overall, in the current form, the results are not convincing.
* Section 4.3: The analysis is minimal: it shows that filters changed after training (as already presented in Figure 4). I don't follow completely the argument that the filters should focus on low frequency. It is more informative, but one could expect that the filters will specialized, thus some of them will focus on high frequencies, to model the high frequency events such as consonants or unvoiced event. 
It could be very interesting to relate the learned filters to the labels: are some filters learned to model specific emotions ? For gender classification, are some filters focusing on the average pitch frequency of male and female speaker ?
* Finally, it would be nice to see if the claims in Section 2 about the fact that only one convolution layer is needed and that stacking pooling layers can hurt the performance are verified experimentally: for instance, experiments with more than one pair of convolution/pooling could be presented.
Minor comments:
* More references for raw waveforms-based approach for speech recognition should be added [3,4,6,7,8,9] in the introduction.
* I don’t understand the first sentence of the paper: “In the field of speech and audio processing, due to the lack of tools to directly process high dimensional data …”. Is this also true for any pattern recognition fields ?  
* For the MFCCs reference in 2.2.2, the authors should cite [12].
* Figure 6: Only half of the spectrum should be presented.
References: 
[1] H. Lee, P. Pham, Y. Largman, and A. Y. Ng. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in Neural Information Processing Systems 22, pages 1096–1104, 2009.
[2] Schuller, Björn, Stefan Steidl, and Anton Batliner. "The interspeech 2009 emotion challenge." Tenth Annual Conference of the International Speech Communication Association. 2009.
[3] N. Jaitly, G. Hinton, Learning a better representation of speech sound waves using restricted Boltzmann machines, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 5884–5887.
[4] D. Palaz, R. Collobert, and M. Magimai.-Doss. Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networks, INTERSPEECH 2013, pages 1766–1770.
[5] Van den Oord, Aaron, Sander Dieleman, and Benjamin Schrauwen. "Deep content-based music recommendation." Advances in neural information processing systems. 2013.
[6] Z.Tuske, P.Golik, R.Schluter, H.Ney, Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR,
in: Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), Singapore, 2014, pp. 890–894.
[7] P. Golik, Z. Tuske, R. Schlu ̈ter, H. Ney, Convolutional Neural Networks for Acoustic Modeling of Raw Time Signal in LVCSR, in: Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2015, pp. 26–30.
[8] Yedid Hoshen and Ron Weiss and Kevin W Wilson, Speech Acoustic Modeling from Raw Multichannel Waveforms, International Conference on Acoustics, Speech, and Signal Processing, 2015.
[9] D. Palaz, M. Magimai-Doss, and R. Collobert. Analysis of CNN-based Speech Recognition System using Raw Speech as Input, INTERSPEECH 2015, pages 11–15.
[10] T. N. Sainath, R. J. Weiss, A. Senior, K. W. Wilson, and O. Vinyals. Learning the Speech Front-end With Raw Waveform CLDNNs. Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2015.
[11] Satt, Aharon & Rozenberg, Shai & Hoory, Ron. (2017). Efficient Emotion Recognition from Speech Using Deep Learning on Spectrograms. 1089-1093. Interspeech 2017.
[12] S. Davis and P. Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech and Signal Processing, 28(4):357–366, 1980.