Deep convolutional neural networks are known to specialize in distilling compact and robust prior from a large amount of data. We are interested in applying deep networks in the absence of training dataset. In this paper, we introduce deep audio prior (DAP) which leverages the structure of a network and the temporal information in a single audio file. Specifically, we demonstrate that a randomly-initialized neural network can be used with carefully designed audio prior to tackle challenging audio problems such as universal blind source separation, interactive audio editing, audio texture synthesis, and audio co-separation.
To understand the robustness of the deep audio prior, we construct a benchmark dataset Universal-150 for universal sound source separation with a diverse set of sources. We show superior audio results than previous work on both qualitative and quantitative evaluations. We also perform thorough ablation study to validate our design choices. https://iclr-dap.github.io/Deep-Audio-Prior/
This paper presents a method for blind source separation relying on randomly initialized networks to decompose an input audio spectrogram into two components.
The networks are designed to promote temporal contiguity of spectral energy in the estimated signals, which are modulated (in time) by estimated masks.
The proposed method is evaluated on a collection of 150 random mixes of sounds, and performs favorably relative to some standard baseline methods (RPCA, NMF, KAM).
This seems like a promising line of work, but at this point I think the weaknesses of the paper outweigh its strengths (as detailed below).  Some of these points may be addressed during discussion, but I currently lean toward reject.
Strengths of the paper:
- The ideas are interesting, and appear to perform well on a simulated and real(ish) data.
- The authors investigate several variations and applications of source separation, including interactive editing, co-separation, and texture synthesis.
- A small (qualitative) ablation study is included to clarify the importance of different components of the loss function.
Weaknesses of the paper:
- Much of the presentation is vague or opaque.  There is little detail provided about the specific architectural parameters of the model, and the diagram (figure 1) does not appear to match the equations.  Specifically, it's unclear whether S_1^* is a function of S_mixture or not.
- The quantitative evaluation focuses entirely on one (new) dataset with unclear characteristics.  No details are provided about the evaluation protocol, and in particular, the tuning of hyperparameters for the various methods under comparison.  Aggregate statistics are included (mean? SDR, etc), but no notion of variance or error bars are included.  It's not ultimately clear how fair this evaluation is.  There should at minimum be a comparison on a standard source separation or speech enhancement dataset, in addition to the new set presented here.
- Most of the spectrogram figures appear to be upside-down, which is confusing.  The details of the audio processing are omitted: STFT parameters are stated, but not the sampling rate.
- There are numerous typos ("grounth", "spectrogram stokes", etc), indicative of the authors not running a spell-checker on their submission.
Questions for the authors:
- The model itself consists of several competing loss functions, but it seems like they may have a trivial, optimal solution at S1 = S_mix (M_1 = 1) and S_2 = 0 (M_2 = 0 or 1).  As far as I can tell, this solution would trivially minimize each term of equation 9.  (If S_1* does not depend on S_mixture, this may be less of an issue, but the trivial solution may still exist when the driving noise is of sufficiently high dimension and the optimization is run long enough.)  Am I misunderstanding the algorithm, or is there some deeper reason why this solution would not be preferred?