Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales which time-domain models have yet to achieve. We demonstrate that our model captures longer-range dependencies than time-domain models such as WaveNet across a diverse set of unconditional generation tasks, including single-speaker speech generation, multi-speaker speech generation, and music generation.
The authors introduce MelNet, an autoregressive model of Mel-frequency scaled spectrograms. They convert audio into high resolution spectrograms to reduce the audio artifacts introduced by inverting spectrograms (here they use gradient-based inversion over Griffin-Lim). To improve modeling of long term dependencies, they perform multi-scale splitting of the spectrograms and maximize the likelihood at each scale (avoiding dominance of noise at higher resolutions). They condition generation at finer scales from coarser scales, enabling sampling through an ancestral process. The authors also highlight the difference between temporal and frequency dimensions, creating different conditioning stacks for the past in time vs. the "past" in frequency (lower frequencies), and mixing conditioning between the two stacks through layers of the network. Multilayer RNNs are used throughout the network and external conditioning is incorporated at the input. 
The challenge the authors are attempting to address is modeling of audio structure on both long and short timescales. As the authors demonstrate with strong baselines, WaveNet models, while superior on fine-scale fidelity, fail to capture dynamics more than a couple hundred milliseconds. The experiments demonstrate improvements on state-of-the-art for unconditional generation on text-to-speech datasets (generating coherent words and phrases) and the MAESTRO piano dataset (generating sections with consistent dynamics/timing/motifs). The continuations of primed examples in both domains are particularly impressive qualitatively, as they maintain much of the character of the priming sample. Ablation experiments qualitatively demonstrate the importance of multi-scale modeling for unconditional generation. Human listener studies support the claims made from qualitative evaluation of long term structure.
This paper should be accepted because it represents a non-trivial adaptation of autoregressive modeling to handle multi-scale structure in audio. The baselines comparisons and strong, and experiments validate the claims of the paper. 
That said, several things could be done to improve the clarity and significance of the paper.
* While the network architecture is described in detail and some figures, the full network structure itself is non-trivial and still somewhat opaque from the plain text description. A schematic diagram of the full network architecture, even in the appendix, could help clarify how many layers are present connecting each component of the model, which would improve reproducibility.
* The paper is a bit thin on metrics. Human listening studies compare long-term structure, but not short-scale fidelity. For TTS, there are clear artifacts from the spectrogram inversion process. Mean opinion scores on conditional samples could help to quantify the importance of each element of the network for audio quality. For instance, how does MOS compare between Griffin-Lim MelNet, Gradient Inversion MelNet, and WaveNet? How does MelNet compare to Linear scaled spectrograms?
* Generating MelSpectrograms to model long-term structure is a fairly established technique, most notably employed by all of the Tacotron variants (https://google.github.io/tacotron/). These models are perhaps a more appropriate comparison for MelNet in many ways, and opt for spectrogram inversion by smaller WaveRNN models. One of the claims of the paper is that it is important to model the fine-scale structure of spectrograms, but it is not clear if that really is the case. A proper comparison to Tacotron models (where spectrograms are generated at the same resolution / the same inversion methods are used) would help clarify the importance of end-2-end training, vs. the learned inversion approach.