Machine learning models of music typically break down the task of composition into a chronological process, composing a piece of music in a single pass from beginning to end. On the contrary, human composers write music in a nonlinear fashion, scribbling motifs here and there, often revisiting choices previously made. We explore the use of blocked Gibbs sampling as an analogue to the human approach, and introduce COCONET, a convolutional neural network in the NADE family of generative models (Uria et al., 2016). Despite ostensibly sampling from the same distribution as the NADE ancestral sampling procedure, we find that a blocked Gibbs approach significantly improves sample quality. We provide evidence that this is due to some conditional distributions being poorly modeled. Moreover, we show that even the cheap approximate blocked Gibbs procedure from Yao et al. (2014) yields better samples than ancestral sampling. We demonstrate the versatility of our method on unconditioned polyphonic music generation. Step 0 Step 1 Step 4 Step 16 Step 64 Ground Truth Figure 1: Ancestral inpainting of a corrupted Bach chorale by COCONET. Colors are used to distinguish the four voices. Grayscale heatmaps show predictions p(xi | xC). The pitch sampled in the current step is indicated by a rectangular outline. The original Bach chorale is shown in the bottom right. Step 0 shows the corrupted Bach chorale. Step 64 shows the result. ∗Work done while at Google Brain. †Work done while at Google Brain.
The paper tackles the task of music generation. They use an orderless NADE model for the task of "fill in the notes". Given a roll of T timesteps of pitches, they randomly mask out some pitches, and the model is trained to predict the missing notes. This follows how the orderless NADE model can be trained. During sampling, one normally follows an ancestral sampling procedure. For this, an ordering is defined over outputs, and one runs the model on the current input, samples one of the outputs according to the order, adds this output to the next input, and continues this procedure until all outputs have been sampled. The key point of the paper is that this is a bad sampling strategy. Instead, they suggest the strategy of Yao et al. 2014, which uses a blocked Gibbs sampling approach. The blocked Gibbs strategy instead masks N inputs randomly and independently, samples them, and repeats this procedure. The point of this strategy is the make sure the sampling chain mixes well, which will happen for large N. However, since the samples are independent, having a large N gives incoherent samples. Thus, the authors follow an annealed schedule for N, making it smaller over time, which will eventually reduce to ancestral sampling (giving global structure to the sample). They conduct a variety of experiments involving both normal metrics and human evaluations, and find that this blocked Gibbs sampling outperforms other sampling procedures.
This is a well written paper - great job.
My main problem with the paper is that having read Uria and Yao, I don't know how much I have learned from this work in the context of this being an ICLR submission. If this was submitted to some computational music / art conference, this paper would be a clear accept. However, for ICLR, I don't see enough novelty compared with previous works this builds upon. Orderless NADE is an established model. The blocked Gibbs sampling and annealing scheme are basically the exact same one used in Yao. Thus, the main novelty of this paper is its application to the music domain, and finding that Yao's method works better for sampling music. This is a good contribution, but more tailored to those working in the music domain. If the authors found that these results also hold for other domains like images (e.g. on CIFAR / tiny Imagenet) and text (e.g. document generation), then I would change my mind and accept this paper for ICLR. Even just trying musical domains other than Bach chorales would be useful. However, as it stands, the experiments are not convincing enough.