ENABLING FACTORIZED PIANO MUSIC MODELING AND GENERATION WITH THE MAESTRO DATASET
Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (∼0.1 ms to ∼100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (≈3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.
This paper combines state of the art models for piano transcription, symbolic music synthesis, and waveform generation all using a shared piano-roll representation.  It also introduces a new dataset of 172 hours of aligned MIDI and audio from real performances recorded on Yamaha Disklavier pianos in the context of the piano-e-competition.  
By using this shared representation and this dataset, it is able to expand the amount of time that it can coherently model music from a few seconds to a minute, necessary for truly modeling entire musical pieces.
Training an existing state of the art transcription model on this data improves performance on a standard benchmark by several percentage points (depending on the specific metric used).
Listening test results show that people still prefer the real recordings a plurality of the time, but that the syntheses are selected over them a fair amount.  One thing that is clear from the audio examples is that the different systems produce output with different equalization levels, which may lead to some of the listening results.  If some sort of automatic mastering were done to the outputs this might be avoided.
While the novelty of the individual algorithms is relatively meager, their combination is very synergistic and makes a significant contribution to the field.  Piano music modeling is a long-standing problem that the current paper has made significant progress towards solving.
The paper is very well written, but there are a few minor issues:
* Eq (1) this is really the joint distribution between audio and notes, not the marginal of audio
* Table 4: What do precision, recall, and f1 score mean for notes with velocity?  How close does the system have to be to the velocity to get it right?
* Table 6: NLL presumably stands for Negative Log Likelihood, but this should be made explicity
* Figure 2: Are the error bars the standard deviation of the mean or the standard error of the mean?