Generative models for singing voice have been mostly concerned with the task of "singing voice synthesis," i.e., to produce singing voice waveforms given musical scores and text lyrics. In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time. In particular, we propose three either unconditioned or weakly conditioned singing voice generation schemes. We outline the associated challenges and propose a pipeline to tackle these new tasks. This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metrics for evaluation.
This paper claims to be the first to tackle unconditional singing voice generation. It is noted that previous singing voice generation approaches leverage explicit pitch information (either of an accompaniment via a score or for the voice itself), and/or specified lyrics the voice should sing. The authors first create their own dataset of singing voice data with accompaniments, then use a GAN to generate singing voice waveforms in three different settings:
1) Free singer - only noise as input, completely unconditional singing sampling
2) Accompanied singer - Providing the accompaniment *waveform* (not symbolic data like a score - the model needs to learn how to transcribe to use this information) as a condition for the singing voice
3) Solo singer - The same setting as 1 but the model first generates an accompaniment then, from that, generates singing voice
Firstly, the authors have done a lot of work - first making their own data, then designing their tasks and evaluating them. The motivation is slightly lacking - it is not clear why we are interested in these three task settings i.e. what we will learn from a difference in their performance, and there is a lack of discussion about which setting makes for better singing voice generation. Also, there is no comparison with other methods: whilst score data is not available it could be estimated, then used for existing models, providing a nice baseline e.g. first a score is extracted with a state of the art AMT method, then a state of the art score to singing voice generation method could be used.
There are existing datasets of clean singing voice and accompaniment, for example MIR-1k (unfortunately I think iKala, another dataset, is now unavailable). It is true that this dataset is small in comparison to the training data the authors generate, but it will certainly be cleaner. I would have liked to see an evaluation performed on this data as opposed to another dataset which was the result of source separation (the authors generate a held out test set on Jazz from Jamendo, on which they perform singing voice separation).
I also had questions about the training data - there is very little information about it included other than it is in-house and covers diverse musical genres (page 6 under 4.1), a second set of 4.5 hours of solo piano, and a third set (?) of jazz singers. This was a bit confusing and could do with clarification. At minimum, I would like to know what genre we are restricting ourselves to - is everything Jazz? Are the accompaniments exclusively piano (it's alluded that the answer to this is no, but it's not clear to me)? Is there any difference between the training and test domain.
On page 6, second to last paragraph when discussing the validation set, I would like the sampling method to be specified - it makes a difference whether the same piece of music will be contained within both the training and validation split, or whether the source piece (from which the 10 second clips are extracted) are in separate splits <- I'd recommend that setting.
The data used to train the model will greatly affect my qualitative assessment of the provided audio samples so, without a clear statement on the training data used, I can't really assess this.
However, with respect to the provided audio samples, I'd first note that these are explicitly specified as randomly sampled, and not cherry picked, which is great, thank you. However, whilst I would admit that the domain is different, when the singing samples are compared with the piano generation unconditional samples of MelNet (https://audio-samples.github.io/#section-3), which I would argue is just as hard to make harmonically valid, they are not as harmonically consistent, even when an accompaniment has been provided. However, samples do sound like human voice, and the pitch is relatively good. The words are unintelligible, but this is explicitly specified as out of scope for this paper, and I agree that this is much harder to achieve.
As an aside, MelNet is not cited in this paper and, given the similarity and relevance, I think it probably should be - https://arxiv.org/abs/1906.01083. It was published this year however so it would be a little harsh to expect it to be there. I would invite the authors to rebut this claim if they think the methods are not comparable.
My main criticism is in relation to the evaluation. For Table 2, without a baseline or the raw data (which would have required no further effort) included in the MOS study, it's very difficult to judge success. If the authors think that comparison with raw data is unfair (as it is an embryonic task) they could include a model which has an unfair advantage from the literature - e.g. uses extracted score information. 
For Table 1, I appreciate the effort that went into the design of 'Vocalness' and 'Matchness' which are 'Inception Score' type metrics leaning on other learned models to return scores. I would like to see discussion which explains the differences in scores for the different model settings (there is a short sentence at the bottom of page 7, but nothing on vocalness).
In summary - this is a hard problem and the authors are the first to tackle it. The different approaches to solve the problem are not well motivated. However, the models are detailed, and well explained. Code is even provided, but data for training is not. If the authors were able to compare with a baseline (like that I describe above), it would go a long way to convincing me that this was good work. As it stands, Tables 1 and 2, and the provided audio samples have no context, so I cant make a conclusion. If this issue and motivation was addressed I would likely vote to accept the paper.
Things to improve the paper that did not impact the score:
1. p2 "we hardly provide any labelled data" specify whether you do or not (I think it's entirely unsupervised since you extract chord progressions and pitch curves using learned models...)
2. p2 "...may suffer from the artifact" -> the artefacts
3. p2 "for the scenario addressed by the accompanied singer" a bit clumsy, may be worth naming your tasks 1, 2 and 3 such that you can easily refer to them
4. p2 "We investigate using conditional GAN ... to address this issue" - which issue do you mean? If it is the issue specified at the top of the paragraph, i.e. that there are many valid melodies for a given harmony (no single ground truth), I don't think using a GAN is a *solution* to this per se. It is a valid model to use, and the solution would be enough varied data (and evaluation to show you're covering your data space and haven't collapsed to a few modes)
5. p2 "the is no established ways" -> there are no established ways
6. p3 "Discriminators in GAN" -> in the GAN
7. p6 "piano playing audio on our own..." -> piano playing on its own (or even just rephrase the sentence - collect 4.5 hours of audio of solo piano)
8. p7 "We apply source separation to the audios divide them into ..." -> we apply source separation to the audio data then divide each track into 20 second...
9. p7 If your piano transcription model was worse than Hawthorne, why didn't you use it? It would have been fine to say you can't reproduce their model if it is not available, but instead you say that 'according to out observation [it] is strong enough' which comes across quite weakly.
10. p8 "in a quiet environment with proper microphone volume" -> headphone volume?
11. p8 "improv" - I think this sentence trailed off prematurely!