Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library will be made available upon paper acceptance and we encourage further contributions from the community and domain experts.
This paper presents a model for audio generation/synthesis  where the model is trained to output time-varying parameters of a vocoder/synthesiser rather than directly outputting audio samples/spectrogram frames. The model is trained by mining an L1 loss between the synthesised audio and the real training audio. Overall, I found the paper to be well written, I found the online supplementary material helpful in getting an intuition for the quality of audio samples generated and to understand the various parts of the proposed architecture. I also found the figures to be extremely  informative, especially figure 2. Although, I think the title of the paper could be updated to something more specific like Differentiable Vocoders or something similar, since the description and experiments very specifically deal with audio synthesis with vocoders, even though the components might be general DSP components. I think the paper presents a reasonable alternative to current autoregressive audio generation methods and should be accepted for publication. 
Minor Comments
1. The reference provided for RNNs, Sutskever et. al. 2014, should be supplemented with older references from the 80s when RNNs were first trained with backdrop through time. 
2. “The bias of the natural world is to vibrate.” I am not sure what exactly is meant here. Is it really a “bias” if every object in the universe vibrates? I think the term bias has been overloaded in the paper and is confused with structural/inductive priors. Bias as used in the paper, seems to have several means. It would help the description if the authors cleared up this confusing use of the term. 
3. In Section 1.3, the authors claim that one of the strengths of the proposed method is that the models are “relatively small”, however all discussion of the sizes is relegated to the appendix. It would be helpful to the reader if some model complexity comparisons are presented in the results section and a high level summary is presented at the end of Section 1. 
4. In Section 3, some additional high-level description for the Harmonic plus Noise model (Serra & Smith, 1990) should be provided to motivate the discussion and experiments in the rest of the paper. 
5. Section 4.2 should provide some description of the parameters counts for each of the components considered and how this compares with existing auto-regressive generation algorithms.