summaries: This paper proposes a domain adaptation type of task via proposing fine-tuning of pre-trained models such as BERT on data from financial domains. The paper starts off with a good motivation about requiring some kind of domain adaptation particularly when performing tasks such as sentiment analysis on data sets from the financial domain. However, there is not much novelty in this paper.

summaries: This paper described the application of BERT in the field of financial sentiment analysis. Authors find that when fine-tuned with in-domain data, BERT outperforms the state-of-the-art, demonstrating that language model pre-training can transfer knowledge learned from unsupervised large corpus to new domain with minimum effort. Experiments are conducted to explore 1) the utility of different in-domain dataset for further pre-training; 2) strategies to avoid catastrophic forgetting, and 3) effectiveness of fine-tuning a subset of the full model.

abstractText: While many sentiment classification solutions report high accuracy scores in product or movie review datasets, the performance of the methods in niche domains such as finance still largely falls behind. The reason of this gap is the domainspecific language, which decreases the applicability of existing models, and lack of quality labeled data to learn the new context of positive and negative in the specific domain. Transfer learning has been shown to be successful in adapting to new domains without large training data sets. In this paper, we explore the effectiveness of NLP transfer learning in financial sentiment classification. We introduce FinBERT, a language model based on BERT, which improved the state-of-the-art performance by 14 percentage points for a financial sentiment classification task in FinancialPhrasebank dataset.

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018

venue: ISSN 0140-525X. doi: arXiv: 1811.03600v2. URL https://github.com/tensorflow/tensor2tensorhttp:// arxiv.org/abs/1810.04805

authors:
Jeremy Howard
Sebastian Ruder

title: Universal Language Model Fine-tuning for Text Classification. jan 2018

venue: URL http://arxiv.org/abs/1801.06146

title: Sentiment analysis of financial news articles using performance indicators. Knowledge and Information Systems, 56(2):373–394, aug 2018

authors:
Macedo Maia
Andr Freitas
Siegfried Handschuh

title: FinSSLx: A Sentiment Analysis Model for the Financial Domain Using Text Simplification

venue: IEEE 12th International Conference on Semantic Computing (ICSC),

authors:
Pekka Malo
Ankur Sinha
Pekka Korhonen
Jyrki Wallenius
Pyry Takala

title: Good debt or bad debt: Detecting semantic orientations in economic texts

venue: Journal of the Association for Information Science and Technology,

authors: Luke Zettlemoyer. Deep contextualized word representations.

title: ISSN 9781941643327

venue: doi: 10.18653/v1/N18-1202. URL http://allennlp.org/elmohttp://arxiv.org/ abs/1802.05365.

authors:
Guangyuan Piao
John G Breslin

title: Financial Aspect and Sentiment Predictions with Deep Neural Networks

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N. Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention Is All You Need

venue: (Nips),

authors:
Steve Yang
Jason Rosenfeld
Jacques Makutonin

title: Financial Aspect-Based Sentiment Analysis using Deep Representations. 2018

venue: URL https://arxiv.org/pdf/1808.07931v1

heading: 1 INTRODUCTION

text:
With unprecedented amount of textual data being created every day, analyzing large bodies of text from distinct domains like medicine or finance is of the utmost importance. Yet it is more difficult to apply supervised NLP methods, like text classification, in these domains than it is for more general language. The difficulty comes from two factors: 1) The most sophisticated classification methods that make use of neural nets require vast amounts of labeled data and labeling domain-specific text snippets requires costly expertise. 2) The NLP models trained on general corpora are not well-suited to supervised tasks since domain-specific texts have a specialized language with unique vocabulary and expressions.
NLP transfer learning methods look like a promising solution to both of the challenges mentioned above, and are the focus of this paper. The core idea behind these models is first training a language model on a very large corpus and then initializing down-stream models with the weights learned from the language modeling task. The initialized layers can range from the single word embedding layer Peters et al. (2018) to the whole model Howard & Ruder (2018). This approach should reduce the size of the required labeled data since language models learn the language syntax and semantic in an unsupervised way on a very large unlabeled corpora by predicting the next word. By further pretraining a language model on a domain specific unlabeled corpus, the model can learn the semantic relations in the text of the target domain, which is likely to have a different distribution than a general corpus.
In this paper, we explore the effectiveness of using and fine-tuning a pre-trained language model, BERT Devlin et al. (2018), in financial sentiment classification using the Financial PhraseBank created by Malo et al. (2014) and FiQA Task-1 sentiment scoring dataset in Maia et al. (2018b).
The main contributions of this paper are the following:
• We introduce FinBERT, which is a language model based on BERT for financial NLP tasks. We evaluate FinBERT on two financial sentiment analysis datasets, where we achieve the state-of-the-art on FiQA sentiment scoring and Financial PhraseBank.
• We implement two other pre-trained language models, ULMFit and ELMo for financial sentiment analysis and compare these with FinBERT.
• We conduct experiments to investigate several aspects of the model, including: effects of further pre-training on financial corpus, training strategies to prevent catastrophic forgetting
and fine-tuning only a small subset of model layers for decreasing training time without a significant drop in performance.

heading: 2 FINBERT

text:
BERT (Devlin et al., 2018) is a language model that consists of a set of transformer encoders (Vaswani et al., 2017) stacked on top of each other. It defines the language modeling in a novel way. Instead of predicting the next word given previous ones, BERT ”masks” a randomly selected 15% of all tokens. With a softmax layer over vocabulary on top of the last encoder layer the masked tokens are predicted. A second task BERT is trained on is ”next sentence prediction”. Given two sentences, the model predicts whether or not these two actually follow each other.
Following the previous work (Howard & Ruder, 2018) on the effectiveness of further pre-training a language model on a target domain, we experimented with two approaches: The first is pre-training the model on a relatively large corpus from the target domain. For that, we further pre-train a BERT language model on a financial corpus. The second approach is pre-training the model only on the sentences from the training classification dataset. Although the second corpus is much smaller, using data from the direct target might provide better target domain adaptation.
Sentiment classification is conducted by adding a dense layer after the last hidden state of the [CLS] token. This is the recommended practice for using BERT for any classification task (Devlin et al., 2018). Then, the classifier network is trained on the labeled sentiment dataset. An overview of all the steps involved in the procedure is presented on figure 2.
While the focus of this paper is classification, we also implement regression with almost the same architecture on a different dataset with continuous targets. The only difference is that the loss function being used is mean squared error instead of the cross entropy loss.
As it was pointed out by Howard & Ruder (2018), catastrophic forgetting is a significant danger with this fine-tuning approach. Because the fine-tuning procedure can quickly cause model to ”forget” the information from language modeling task as it tries to adapt to the new task. In order to deal with this phenomenon, we apply three techniques as it was proposed by Howard & Ruder (2018): slanted triangular learning rates, discriminative fine-tuning and gradual unfreezing.
Slanted triangular learning rate applies a learning rate schedule in the shape of a slanted triangular, that is, learning rate first linearly increases up to some point and after that point linearly decreases.
Discriminative fine-tuning is using lower learning rates for lower layers on the network. Assume our learning rate at layer l is α. Then for discrimination rate of θ we calculate the learning rate for layer l− 1 as αl−1 = θαl. The assumption behind this method is that the lower layers represent the deep-level language information, while the upper ones include information for actual classification task. Therefore we fine-tune them differently.
With gradual freezing, we start training with all layers but the classifier layer as frozen. During training we gradually unfreeze all of the layers starting from the highest one, so that the lower level features become the least fine-tuned ones. Hence, during the initial stages of training it is prevented for model to ”forget” low-level language information that it learned from pre-training.

heading: 3 EXPERIMENTAL SETUP

heading: 3.1 DATASETS

text:
In order to further pre-train BERT, we use a financial corpus we call TRC2-financial. It is a subset of Reuters’ TRC21, which consists of 1.8M news articles that were published by Reuters between 2008 and 2010. We filter for some financial keywords in order to make corpus more relevant and in limits with the compute power available. The resulting corpus, TRC2-financial, includes 46,143 documents with more than 29M words and nearly 400K sentences.
The main sentiment analysis dataset used in this paper is Financial PhraseBank2 from Malo et al. 2014 Malo et al. (2014). Financial Phrasebank consists of 4845 english sentences selected randomly from financial news found on LexisNexis database. These sentences then were annotated by 16 people with background in finance and business. The annotators were asked to give labels according to how they think the information in the sentence might affect the mentioned company stock price. The dataset also includes information regarding the agreement levels on sentences among annotators. The distribution of agreement levels and sentiment labels can be seen on table 1. We set aside 20% of all sentences as test and 20% of the remaining as validation set. In the end, our train set includes 3101 examples. For some of the experiments, we also make use of 10-fold cross validation.
FiQA Maia et al. (2018b) is a dataset that was created for WWW ’18 conference financial opinion mining and question answering challenge3. We use the data for Task 1, which includes 1,174 financial news headlines and tweets with their corresponding sentiment score. Unlike Financial Phrasebank, the targets for this datasets are continuous ranging between [−1, 1] with 1 being the most positive. Each example also has information regarding which financial entity is targeted in the sentence. We do 10-fold cross validation for evaluation of the model for this dataset.

heading: 3.2 BASELINE METHODS

text:
For contrastive experiments, we consider baselines with three different methods: LSTM classifier with GLoVe embeddings, LSTM classifier with ELMo embeddings and ULMFit classifier. It should be noted that these baseline methods are not experimented with as thoroughly as we did with BERT. Therefore the results should not be interpreted as definitive conclusions of one method being better.
We implement two classifiers using bidirectional LSTM models. In both of them, a hidden size of 128 is used, with the last hidden state size being 256 due to bidirectionality. A fully connected feed-forward layer maps the last hidden state to a vector of three, representing likelihood of three
1The corpus can be obtained for research purposes by applying here: https://trec.nist.gov/data/reuters/reuters.html
2The dataset can be found here: https://www.researchgate.net/publication/251231364 FinancialPhraseBank-v10
3Data can be found here: https://sites.google.com/view/fiqa/home
LPS (Malo et al., 2014), HSC (Krishnamoorthy, 2018) and FinSSLX (Maia et al., 2018b) results are taken from their respective papers. For LPS and HSC, overall accuracy is not reported on the papers. We calculated them using recall scores reported for different classes. For the models implemented by us, we report 10-fold cross validation results.
labels. The difference between two models is that one uses GLoVe embeddings, while the other uses ELMo embeddings. A dropout probability of 0.3 and a learning rate of 3e-5 is used in both models. We train them until there is no improvement in validation loss for 10 epochs.
Classification with ULMFit consists of three steps. The first step of pre-training a language model is already done and the pre-trained weights are released by Howard and Ruder (2018). We first further pre-train AWD-LSTM language model on TRC2-financial corpus for 3 epochs. After that, we finetune the model for classification on Financial PhraseBank dataset, by adding a fully-connected layer to the output of pre-trained language model.

heading: 3.3 IMPLEMENTATION DETAILS

text: For our implementation BERT, we use a dropout probability of p = 0.1, warm-up proportion of 0.2, maximum sequence length of 64 tokens, a learning rate of 2e − 5 and a mini-batch size of 64. We train the model for 6 epochs, evaluate on the validation set and choose the best one. For discriminative fine-tuning we set the discrimination rate as 0.85. We start training with only the classification layer unfrozen, after each third of a training epoch we unfreeze the next layer. An Amazon p2.xlarge EC2 instance with one NVIDIA K80 GPU, 4 vCPUs and 64 GiB of host memory is used to train the models.

heading: 4 EXPERIMENTAL RESULTS

text:
The results of FinBERT, the baseline methods and state-of-the-art on Financial PhraseBank dataset classification task can be seen on table 2. We present the result on both the whole dataset and subset with 100% annotator agreement.
For all of the measured metrics, FinBERT performs clearly the best among both the methods we implemented ourselves (LSTM and ULMFit) and the models reported by other papers (LPS Malo et al. (2014), HSC Krishnamoorthy (2018), FinSSLX Maia et al. (2018a)). LSTM classifier with no language model information performs the worst. In terms of accuracy, it is close to LPS and HSC, (even better than LPS for examples with full agreement), however it produces a low F1-score. That is due to it performing much better in neutral class. LSTM classifier with ELMo embeddings improves upon LSTM with static embeddings in all of the measured metrics. It still suffers from low average F1-score due to poor performance in less represented labels. But it’s performance is comparable with LPS and HSC, besting them in accuracy. So contextualized word embeddings produce close performance to machine learning based methods for dataset of this size.
ULMFit significantly improves on all of the metrics and it doesn’t suffer from model performing much better in some classes than the others. It also handily beats the machine learning based models LPS and HSC. This shows the effectiveness of language model pre-training. AWD-LSTM is a very
Yang et al. (2018) and Piao & Breslin (2018) report results on the official test set. Since we don’t have access to that set our MSE, and R2 are calculated with 10-Fold cross validation.
large model and it would be expected to suffer from over-fitting with this small of a dataset. But due to language model pre-training and effective training strategies, it is able to overcome small data problem. ULMFit also outperforms FinSSLX, which has a text simplification step as well as pre-training of word embeddings on a large financial corpus with sentiment labels.
FinBERT outperforms ULMFit, and consequently all of the other methods in all metrics. In order to measure the performance of the models on different sizes of labeled training datasets, we ran LSTM classifiers, ULMFit and FinBERT on 5 different configurations. The result can be seen on figure 2, where the cross entropy losses on test set for each model are drawn. 100 training examples is too low for all of the models. However, once the training size becomes 250, ULMFit and FinBERT starts to successfully differentiate between labels, with an accuracy as high as 80% for FinBERT. All of the methods consistently get better with more data, but ULMFit and FinBERT does better with 250 examples than LSTM classifiers do with the whole dataset. This shows the effectiveness of language model pre-training.
The results for FiQA sentiment dataset, are presented on table 3. Our model outperforms state-ofthe-art models for both MSE and R2. It should be noted that the test set these two papers Yang et al. (2018) Piao & Breslin (2018) use is the official FiQA Task 1 test set. Since we don’t have access to that we report the results on 10-Fold cross validation. There is no indication on Maia et al. (2018b) that the train and test sets they publish come from different distributions and our model can be interpreted to be at disadvantage since we need to set aside a subset of training set as test set, while state-of-the-art papers can use the complete training set.

heading: 5 EXPERIMENTAL ANALYSIS

heading: 5.1 EFFECTS OF FURTHER PRE-TRAINING

text:
We first measure the effect of further pre-training on the performance of the classifier. We compare three models: 1) No further pre-training (denoted by Vanilla BERT), 2) Further pre-training on
Results are reported on 10-fold cross validation.
classification training set (denoted by FinBERT-task), 3) Further pre-training on domain corpus, TRC2-financial (denoted by FinBERT-domain). Models are evaluated with loss, accuracy and macro average F1 scores on the test dataset. The results can be seen on table 4.
The classifier that were further pre-trained on financial domain corpus performs best among the three, though the difference is not very high. There might be four reasons behind this result: 1) The corpus might have a different distribution than the task set, 2) BERT classifiers might not improve significantly with further pre-training, 3) Short sentence classification might not benefit significantly from further pre-training, 4) Performance is already so good, that there is not much room for improvement. We think that the last explanation is the likeliest, because for the subset of Financial Phrasebank that all of the annotators agree on the result, accuracy of Vanilla BERT is already 0.96. The performance on the other agreement levels should be lower, as even the humans can’t agree fully on them. More experiments with another financial labeled dataset is necessary to conclude that effect of further pre-training on domain corpus is not significant.

heading: 5.2 CATASTROPHIC FORGETTING

text:
For measuring the performance of the techniques against catastrophic forgetting, we try four different settings: No adjustment (NA), only with slanted triangular learning rate (STL), slanted triangular learning rate and gradual unfreezing (STL+GU) and the techniques in the previous one, together with discriminative fine-tuning. We report the performance of these four settings with loss on test function and trajectory of validation loss over training epochs. The results can be seen on table 5 and figure 3.
Applying all three of the strategies produce the best performance in terms of test loss and accuracy. Gradual unfreezing and discriminative fine-tuning have the same reasoning behind them: higher level features should be fine-tuned more than the lower level ones, since information learned from language modeling are mostly present in the lower levels. We see from table 5 that using only discriminative fine-tuning with slanted triangular learning rates performs worse than using the slanted triangular learning rates alone. This shows that gradual unfreezing is the most important technique for our case.
Results are reported on 10-fold cross validation. STL: slanted triangular learning rates, GU: gradual unfreezing, DFT: discriminative fine-tuning.
One way that catastrophic forgetting can show itself is the sudden increase in validation loss after several epochs. As model is trained, it quickly starts to overfit when no measure is taken accordingly. As it can be seen on the figure 3, that is the case when none of the aforementioned techniques are applied. The model achieves the best performance on validation set after the first epoch and then starts to overfit. While with all three techniques applied, model is much more stable. The other combinations lie between these two cases.

heading: 5.3 TRAINING ONLY A SUBSET OF THE LAYERS

text:
BERT is a very large model. Even on small datasets, fine-tuning the whole model requires significant time and computing power. Therefore if a slightly lower performance can be achieved with finetuning only a subset of all parameters, it might be preferable in some contexts. Especially if training set is very large, this change might make BERT more convenient to use. Here we experiment with fine-tuning only the last k many encoder layers.
The results are presented on table 6. Fine-tuning only the classification layer does not achieve close performance to fine-tuning other layers. However fine-tuning only the last layer handily outperforms the state-of-the-art machine learning methods like HSC. After Layer-9, the performance becomes virtually the same, only to be outperformed by fine-tuning the whole model. This result shows that in order to utilize BERT, an expensive training of the whole model is not mandatory. A fair trade-off can be made for much less training time with a small decrease in model performance.

heading: 5.4 WHERE DOES THE MODEL FAIL?

text:
With 97% accuracy on the subset of Financial PhraseBank with 100% annotator agreement, we think it might be an interesting exercise to examine cases where the model failed to predict the true
label. Therefore in this section we will present several examples where model makes the wrong prediction. Also in Malo et al. (2014), it is indicated that most of the inter-annotator disagreements are between positive and neutral labels (agreement for separating positive-negative, negative-neutral and positive-neutral are 98.7%, 94.2% and 75.2% respectively). Authors attribute that the difficulty of distinguishing ”commonly used company glitter and actual positive statements”. We will present the confusion matrix in order to observe whether this is the case for FinBERT as well.
Example 1: Pre-tax loss totaled euro 0.3 million , compared to a loss of euro 2.2 million in the first quarter of 2005 .
True value: Positive Predicted: Negative
Example 2: This implementation is very important to the operator , since it is about to launch its Fixed to Mobile convergence service in Brazil
True value: Neutral Predicted: Positive
Example 3: The situation of coated magazine printing paper will continue to be weak .
True value: Negative Predicted: Neutral
The first example is actually the most common type of failure. The model fails to do the math in which figure is higher, and in the absence of words indicative of direction like ”increased”, might make the prediction of neutral. However, there are many similar cases where it does make the true prediction too. Examples 2 and 3 are different versions of the same type of failure. The model fails to distinguish a neutral statement about a given situation from a statement that indicated polarity about the company. In the third example, information about the company’s business would probably help.
73% of the failures happen between labels positive and negative, while same number is 5% for negative and positive. That is consistent with both the inter-annotator agreement numbers and common sense. It is easier to differentiate between positive and negative. But it might be more challenging to decide whether a statement indicates a positive outlook or merely an objective observation.

heading: 6 CONCLUSION AND FUTURE WORK

text:
In this paper, we implemented BERT for the financial domain by further pre-training it on a financial corpus and fine-tuning it for sentiment analysis (FinBERT). This work is the first application of BERT for finance to the best of our knowledge and one of the few that experimented with further pre-training on a domain-specific corpus. On both of the datasets we used, we achieved state-of-theart results by a significant margin. For the classification task, we increased the state-of-the art by 15% in accuracy.
In addition to BERT, we also implemented other pre-training language models like ELMo and ULMFit for comparison purposes. ULMFit, further pre-trained on a financial corpus, beat the previous state-of-the art for the classification task, only to a smaller degree than BERT. These results show the effectiveness of pre-trained language models for a down-stream task such as sentiment analysis especially with a small labeled dataset. The complete dataset included more than 3000 examples, but FinBERT was able to surpass the previous state-of-the art even with a training set as small as 500 examples. This is an important result, since deep learning techniques for NLP have been traditionally labeled as too ”data-hungry”, which is apparently no longer the case.
We conducted extensive experiments with BERT, investigating the effects of further pre-training and several training strategies. We couldn’t conclude that further pre-training on a domain-specific corpus was significantly better than not doing so for our case. Our speculation is that BERT already performs good enough with our dataset that there is not much room for improvement that further pretraining can provide. We also found that learning rate regimes that fine-tune the higher layers more
aggressively than the lower ones perform better and are more effective in preventing catastrophic forgetting. Another conclusion from our experiments was that, comparable performance can be achieved with much less training time by fine-tuning only the last 2 layers of BERT.

title: FINBERT: FINANCIAL SENTIMENT ANALYSIS WITH PRE-TRAINED LANGUAGE MODELS


summaries: This paper claims to be the first to tackle unconditional singing voice generation. It is noted that previous singing voice generation approaches leverage explicit pitch information (either of an accompaniment via a score or for the voice itself), and/or specified lyrics the voice should sing. The authors first create their own dataset of singing voice data with accompaniments, then use a GAN to generate singing voice waveforms in three different settings:

summaries: This paper has set a new problem: singing voice synthesis without any score/lyrics supervision. The authors provide a significance of such a problem in section 1. Also, the authors successfully design and implement a novel neural network architecture to solve the problem. It’s also notable that the authors kindly open-source their code to mitigate the reproducibility issue. This paper may serve as baseline results for the proposed problem in the future.

abstractText: Generative models for singing voice have been mostly concerned with the task of “singing voice synthesis,” i.e., to produce singing voice waveforms given musical scores and text lyrics. In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time. In particular, we propose three either unconditioned or weakly conditioned singing voice generation schemes. We outline the associated challenges and propose a pipeline to tackle these new tasks. This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metrics for evaluation.

authors:
David Berthelot
Thomas Schumm
Luke Metz

title: BEGAN: Boundary equilibrium generative adversarial networks

venue: arXiv preprint,

authors:
Mikołaj Bińkowski
Jeff Donahue
Sander Dieleman
Aidan Clark
Erich Elsen
Norman Casagrande
Luis C. Cobo
Karen Simonyan

title: High fidelity speech synthesis with adversarial networks

authors:
Merlijn Blaauw
Jordi Bonada
Ryunosuke Daido

title: Data efficient voice cloning for neural singing synthesis

venue: In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing,

authors:
Sebastian Böck
Filip Korzeniowski
Jan Schlüter
Florian Krebs
Gerhard Widmer

title: madmom: a new python audio and music signal processing library

venue: arXiv preprint arXiv:1605.07008,

authors:
Jordi Bonada
Xavier Serra

title: Synthesis of the singing voice by performance sampling and spectral models

venue: IEEE Signal Processing Magazine,

authors:
Estefanı́a Cano
Judith Liebetrau
Derry Fitzgerald
Karlheinz Brandenburg

title: The dimensions of perceptual quality of sound source separation

authors:
Kyunghyun Cho
Bart van Merrienboer
Dzmitry Bahdanau
Yoshua Bengio

title: On the properties of neural machine translation: Encoder-decoder approaches

venue: In Proc. Workshop on Syntax, Semantics and Structure in Statistical Translation,

authors:
Junyoung Chung
Caglar Gulcehre
KyungHyun Cho
Yoshua Bengio

title: Empirical evaluation of gated recurrent neural networks on sequence modeling

venue: In Proc. NIPS Workshop on Deep Learning,

title: Singing voice synthesis: History, current work, and future directions

venue: Computer Music Journal,

authors:
Alexandre Défossez
Neil Zeghidour
Nicolas Usunier
Léon Bottou
Francis Bach

title: SING: Symbol-to-instrument neural generator

venue: In Proc. Advances in Neural Information Processing Systems,

authors:
Chris Donahue
Julian McAuley
Miller Puckette

title: Adversarial audio synthesis

venue: In Proc. International Conference on Learning Representations,

title: Development of musical schemata in children’s spontaneous singing

venue: Advances in Psychology,

authors:
Jesse Engel
Kumar Krishna Agrawal
Shuo Chen
Ishaan Gulrajani
Chris Donahue
Adam Roberts

title: GANsynth: Adversarial neural audio synthesis

authors:
Andrew Gibiansky
Sercan Arik
Gregory Diamos
John Miller
Kainan Peng
Wei Ping
Jonathan Raiman
Yanqi Zhou

title: Deep Voice 2: Multi-speaker neural text-to-speech

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
D. Griffin
J. Lim

title: Signal estimation from modified short-time fourier transform

venue: IEEE Transactions on Acoustics, Speech, and Signal Processing,

authors:
Gaëtan Hadjeres
François Pachet
Frank Nielsen

title: DeepBach: A steerable model for Bach chorales generation

venue: In Proc. International Confernece on Machine Learning,

authors:
Curtis Hawthorne
Erich Elsen
Jialin Song
Adam Roberts
Ian Simon
Colin Raffel
Jesse Engel
Sageev Oore
Douglas Eck

title: Onsets and frames: Dual-objective piano transcription

venue: In Proc. International Society for Music Information Retrieval Conference,

authors:
Romain Hennequin
Anis Khlif
Felix Voituret
Manuel

title: Moussalam. Spleeter: A fast and stateof-the art music source separation tool with pre-trained models

authors:
Yukiya Hono
Shumma Murata
Kazuhiro Nakamura
Kei Hashimoto
Keiichiro Oura
Yoshihiko Nankaku
Keiichi Tokuda

title: Recent development of the DNN-based singing voice synthesis systemSinsy

venue: In Proc. Asia-Pacific Signal and Information Processing Association Annual Summit and Conference,

authors:
Yukiya Hono
Kei Hashimoto
Keiichiro Oura
Yoshihiko Nankaku
Keiichi Tokuda

title: Singing voice synthesis based on generative adversarial networks

title: Cepstral analysis synthesis on the mel frequency scale

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A. Efros

title: Image-to-image translation with conditional adversarial networks

venue: In Proc. Conference on Computer Vision and Pattern Recognition,

authors:
Kantapon Kaewtip
Fernando Villavicencio
Fang-Yu Kuo
Mark Harvilla
Iris Ouyang
Pierre Lanchantin

title: Enhanced virtual singers generation by incorporating singing dynamics to personalized text-to-speech-to-singing

authors:
Nal Kalchbrenner
Erich Elsen
Karen Simonyan
Seb Noury
Norman Casagrande
Edward Lockhart
Florian Stimberg
Aaron van den Oord
Sander Dieleman
Koray Kavukcuoglu

title: Efficient neural audio synthesis

venue: In Proc. International Conference on Machine Learning,

authors:
Jong Wook Kim
Justin Salamon
Peter Li
Juan Pablo Bello

title: CREPE: A convolutional representation for pitch estimation

authors:
Juntae Kim
Heejin Choi
Jinuk Park
Minsoo Hahn
Sangjin Kim
Jong-Jin Kim

title: Korean singing voice synthesis system based on an LSTM recurrent neural network

venue: In Proc. INTERSPEECH,

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Sangeun Kum
Juhan Nam

title: Joint detection and classification of singing voice melody using convolutional recurrent neural networks

venue: Applied Sciences,

authors:
Juheon Lee
Hyeong-Seok Choi
Chang-Bin Jeon
Junghyun Koo
Kyogu Lee

title: Adversarially trained end-to-end Korean singing voice synthesis system

title: Multi-singer singing voice synthesis system

authors:
Kyungyun Lee
Keunwoo Choi
Juhan Nam

title: Revisiting singing voice detection: a quantitative review and the future outlook

authors:
Hyungui Lim
Seungyeon Rhyu
Kyogu Lee

title: Chord generation from symbolic melody using BLSTM networks

authors:
Jen-Yu Liu
Yi-Hsuan Yang

title: Dilated convolution with dilated GRU for music source separation

venue: In Proc. International Joint Conference on Artificial Intelligence,

authors:
Xudong Mao
Qing Li
Haoran Xie
Raymond Y.K. Lau
Zhen Wang
Stephen Paul Smolley

title: Least squares generative adversarial networks

venue: In Proc. IEEE International Conference on Computer Vision,

authors:
Andrés Marafioti
Nicki Holighaus
Nathanaël Perraudin
Piotr Majdak

title: Adversarial generation of time-frequency features with application in audio synthesis

authors:
Brian McFee
Colin Raffel
Dawen Liang
Daniel P. W . Ellis
Matt McVicar
Eric Battenberg
Oriol Nieto

title: librosa: Audio and music signal analysis in Python

venue: In Proc. Python in Science Conf., pp

authors:
Mehdi Mirza
Simon Osindero

title: Conditional generative adversarial nets

venue: arXiv preprint arXiv:1411.1784,

authors:
Kazuhiro Nakamura
Kei Hashimoto
Keiichiro Oura
Keiichi Tokuda Yoshihiko Nankaku

title: Singing voice synthesis based on convolutional neural networks

authors:
Masanari Nishimura
Kei Hashimoto
Keiichiro Oura
Yoshihiko Nankaku
Keiichi Tokuda

title: Singing voice synthesis based on deep neural networks

authors:
Keiichiro Oura
Ayami Mase
Tomohiko Yamada
Satoru Muto
Yoshihiko Nankaku
Keiichi Tokuda

title: Recent development of the HMM-based singing voice synthesis system–sinsy

venue: In Proc. ISCA Workshop on Speech Synthesis,

authors:
Giyasettin Ozcan
Cihan Isikhan
Adil Alpkock

title: Melody extraction on MIDI music files

venue: In Proc. IEEE International Symposium on Multimedia,

authors:
Zafar Rafii
Antoine Liutkus
Fabian-Robert Stöter
Stylianos Ioannis Mimilakis
Rachel Bittner

title: The MUSDB18 corpus for music separation, 2017

authors:
Zafar Rafii
Antoine Liutkus
Fabian-Robert Stter
Stylianos Ioannis Mimilakis
Derry FitzGerald
Bryan Pardo

title: An overview of lead and accompaniment separation in music

venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,

authors:
Keijiro Saino
Heiga Zen
Yoshihiko Nankaku
Akinobu Lee
Keiichi Tokuda

title: An HMM-based singing voice synthesis system

venue: In Proc. International Conference on Spoken Language Processing,

authors:
Tim Salimans
Diederik P. Kingma

title: Weight normalization: A simple reparameterization to accelerate training of deep neural networks

venue: Proc. Advances in Neural Information Processing Systems,

authors:
Tom Sercu
Vaibhava Goel

title: Dense prediction on sequences with time-dilated convolutions for speech recognition

venue: arXiv preprint arXiv:1611.09288,

authors:
Jonathan Shen
Ruoming Pang
Ron J. Weiss
Mike Schuster
Navdeep Jaitly
Zongheng Yang
Zhifeng Chen
Yu Zhang
Yuxuan Wang
R.J. Skerry-Ryan
Rif A. Saurous
Yannis Agiomyrgiannakis
Yonghui Wu

title: Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions

authors:
Fabian-Robert Stöter
Antoine Liutkus
Nobutaka Ito

title: The 2018 signal separation evaluation campaign

venue: In Proc. Latent Variable Analysis and Signal Separation,

authors:
Fabian-Robert Stöter
Stefan Uhlich
Antoine Liutkus
Yuki Mitsufuji

title: Open-Unmix - a reference implementation for music source separation

venue: Journal of Open Source Software,

authors:
Hiroki Tamaru
Yuki Saito
Shinnosuke Takamichi
Tomoki Koriyama
Hiroshi Saruwatari

title: Generative moment matching network-based random modulation post-filter for DNN-based singing voice synthesis and neural double-tracking

authors:
Martı́ Umbert
Jordi Bonada
Masataka Goto
Tomoyasu Nakano
Johan Sundberg

title: Expression control in singing voice synthesis: Features, approaches, evaluation, and challenges

authors:
Aaron van den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew Senior
Koray Kavukcuoglu

title: WaveNet: A generative model for raw audio

venue: arXiv preprint arXiv:1609.03499,

authors:
Sean Vasquez
Mike Lewis

title: MelNet: A generative model for audio in the frequency domain

venue: arXiv preprint arXiv:1906.01083,

authors:
Elliot Waite
Douglas Eck
Adam Roberts
Dan Abolafia

title: Project Magenta: Generating longterm structure in songs and stories, 2016

authors:
Li-Chia Yang
Szu-Yu Chou
Yi-Hsuan Yang

title: MidiNet: A convolutional generative adversarial network for symbolic-domain music generation

heading: 1 INTRODUCTION

text:
The task of computationally producing singing voices is usually referred to as singing voice synthesis (SVS) in the literature (Cook, 1996). Most researchers assume that the note sequence and the lyrics of the waveform to be generated are given as the model input, and aim to build synthesis engines that sound as natural and expressive as a real singer (Blaauw et al., 2019; Hono et al., 2019; Kaewtip et al., 2019; Lee et al., 2019a; Tamaru et al., 2019). As such, the content of the produced singing voice is largely determined by the given model input, which is usually assigned by human. And, accordingly, progress in SVS has followed closely with that in text-to-speech (TTS) synthesis (Umbert et al., 2015; Shen et al., 2017; Gibiansky et al., 2017).
However, we argue that singing according to a pre-assigned musical score and lyrics is only a part of the human singing activities. For human beings, singing can also be a spontaneous activity. We learn to spontaneously sing when we were children (Dowling, 1984). We do not need a score to sing when we are humming on the road or in the bathroom. The voices sung do not have to be intelligible. Jazz vocalists can improvise according to a chord progression, an accompaniment, or even nothing.
We aim to explore such a new task in this paper: teaching a machine to sing with a training collection of singing voices, but without the corresponding musical scores and lyrics of the training data. Moreover, the machine has to sing without pre-assigned score and lyrics as well even in the inference (generation) time. This task is challenging in that, as the machine sees no lyrics at all, it hardly has any knowledge of the human language to pronounce or articulate either voiced or unvoiced sounds. And, as the machine sees no musical scores at all, it has to find its own way learning the language of music in creating plausible vocal melodies. It also makes the task different from TTS.
Specifically, we consider three types of such score- and lyrics-free singing voice generation tasks, as shown in Figures 1(b)–(d). A free singer sings with only random noises as the input. An accompanied singer learns to sing over a piece of instrumental music, which is given as an audio waveform (again without score information). Finally, a solo singer also sings with only noises as the input, but it uses the noises to firstly generate some kind of ‘inner ideas’ of what to sing.
From a technical point of view, we can consider SVS as a strongly conditioned task for generating singing voices, as the target output is well specified by the input. In contrast, the proposed tasks are either unconditioned or weakly conditioned. This work therefore contributes to expanding the “spectrum” (in terms of the strength of conditional signals) of singing voice generation. Doing so has at least two implications. First, while our models are more difficult to train than SVS models, they enjoy more freedom in the generation output. Such freedom may be desirable considering the artistic nature of singing. Second, we can more easily use a larger training set to train our model—
due to the difficulty in preparing time-aligned scores and lyrics, the training set employed in existing work on SVS usually consists of tens of songs only (Lee et al., 2019a); in contrast, in our case we do not need labeled and aligned data and can therefore use more than hundreds of songs for training. This may help establish a universal model based on which extensions can be made.
The proposed accompanied singer also represents one of the first attempts to produce singing voice given an accompaniment. One intuitive approach to achieve this is to first generate a score according to an accompaniment in the symbolic domain and then synthesize the singing voices according to the score. The second step of synthesis is relatively well-established, but the first step of generating a score given an accompaniment is not explored yet. Extensive researches have been done in generating scores of one or several instruments (Hadjeres et al., 2017; Yang et al., 2017; Huang et al., 2019; Payne, 2019). However, to the best of our knowledge, very few, if any, researches have been done on generating scores of singing voices given an accompaniment. Our approach bypasses the step of generating scores by directly generating the mel-spectrogram representation.
We outline below the challenges associated with the proposed tasks and the solutions we investigate.
First, the tasks are unsupervised as we do not provide any labels (e.g., annotations of phonemes, pitches, or onset times) for the training singing files. The machine has to learn the complex structure of music directly from audio signals. We explore the use of generative adversarial network (GAN) (Goodfellow et al., 2014) to address this issue, for its demonstrated effectiveness for SVS (Hono et al., 2019) and pitch-conditioned instrument note synthesis (Engel et al., 2019). Specifically, we design a novel GAN-based architecture to learn to generate the mel-spectrogram of singing voice, and then use WaveRNN (Kalchbrenner et al., 2018), a single-layer recurrent neural network, as the vocoder to generate the audio waveform. Rather than considering the mel-spectrograms as a fixedsize image as done in recent work on audio generation (Engel et al., 2019; Marafioti et al., 2019), we use gated recurrent units (GRUs) (Cho et al., 2014) and dilated convolutions (van den Oord et al., 2016) in both the generator and discriminator, to model both the local and sequential patterns in music and to facilitate the generation of variable-length waveforms.
Second, for training the free singer, unaccompanied vocal tracks are needed. As for the accompanied singer, we need additionally an accompaniment track for each vocal track. However, public-domain multi-track music data is hard to find. We choose to implement a vocal source separation model with state-of-the-art separation quality (Liu & Yang, 2019) for data preparation. The proposed pipeline for training and evaluating an accompanied singer is illustrated in Figure 2. The advantage of having a vocal separation model is that we can use as many audio files as we have to compile the training data. The downside is that the singing voice generation models may suffer from the artifacts (Cano et al., 2018) of the source separation model, which is moderate but not negligible.1
Third, for the accompanied singer, there is no single “ground truth” and the relationship between the model input and output may be one-to-many. This is because there are plenty of valid ways to
1Research on singing voice separation has made remarkable progress in recent years. As reported in (Stöter et al., 2018), state-of-the-art models perform comparably to an oracle approach for many cases. Open-source implementations of state-of-the-art models are also available (Stöter et al., 2019; Hennequin et al., 2019). Visit https://bit.ly/2Xattua for samples of the separation result of the model we employ.
sing over an accompaniment track. For diversity and artistic freedom, we cannot ask the machine to generate any specific singing voice in response to an accompaniment track, even if we have paired data of vocal and accompaniment tracks. We investigate using conditional GAN (Mirza & Osindero, 2014) to retain the possibility of generating singing voices with multiple modes.
Fourth, as the proposed tasks are new, there are no established ways for performance evaluation. According to our setting, we desire our machine to generate audio waveforms with high quality and diversity, vocal-like timbre, plausible pitch contour, emotion expression, and, for the accompanied singer, that are in harmony with the given accompaniment track. But, the singing does not have to be intelligible. We propose customized objective and subjective metrics to evaluate our models in these aspects. For example, we adapt the melody harmonization model proposed by Lim et al. (2017) to measure the matchness between the generated vocal track and the given accompaniment track.
Finally, reproducibility is a major issue, especially for a subjective task. We intend to use publiclyavailable copyright-free instrumental music as the conditional signals for testing the accompanied singer, so that other researchers can use the same testing conditions for model comparison in the future. We will also release the testing conditions for the solo singer, the generated singing voices for all our models, as well as open source our code through a public git repository [URL removed].
We focus on Jazz music in this work. Samples of the generated singing voices can be found at https://bit.ly/2mIvoIc. Our models have many possible use cases. For example, we can use the accompanied singer as a backing vocalist. In addition, we can use the free singer as a sound source—to demonstrate this, we make a song by hand in the style of Jazz Hiphop by sampling the output of our free singer. This song can be listened to at https://bit.ly/2QkUJoJ.

heading: 2 SCHEMES OF SINGING VOICE GENERATION

text:
A free singer takes no conditions at all as the input. We want it to sing freely. The singing voices from a free singer may not even sound good, but they should sound like singing voice. A free singer is like we are freely humming or singing on the road walking or in the bathroom taking a shower. We may not even know what we are singing and likely there is no underlying musical score.
From the viewpoint of a generative model, training a free singer amounts to modeling a distribution P (Y), where Y ∈ RK×T is a matrix representing a sequence of K-dimensional features and T is the number of time frames. A free singing is sampled from this distribution without conditions.
An accompanied singer takes as the input a sequence of accompaniment-derived features. An accompanied singer tries to generate singing voices that match the accompaniment track in some way. It is similar to the case of Karaoke, where a backing accompaniment track is played from a speaker, the lyrics and a video are displayed on a screen, and a user tries to sing according to the
lyrics and the backing track. The difference is that, this time the user is a trained model and we do not ask it to follow the lyrics or the exact pitch contour of the accompaniment. The note sequence found in the singing has to be in harmony with, but not a duplicate of, that in the backing track.
Training an accompanied singer amounts to modeling a distribution P (Y|A), where Y ∈ RK×T represents a feature sequence of the vocal track, and A ∈ RH×T is a feature sequence of the given accompaniment track. In our implementation, we use the mel-spectrograms for Y in compliance with the need of the vocoder. For A, different features can be tried, and we investigate using a transcription model (Hawthorne et al., 2018) to extract pitch features. See Section 4.1 for details.
A solo singer is similar to a free singer in that both takes no conditions as the input. However, a solo singer would generate an ‘inner idea’ first, and then sing according to that. In other words, it learns a joint distribution P (Y|I)Q(I), where I ∈ RJ×T is a matrix representing the idea sequence. The singer first samples I from a distribution Q, and then uses that to conditionally sample Y from P . The inner idea I can take several forms. In this work, we instantiate this scheme with I being a chord progression (namely a sequence of chord labels). The distribution Q is modeled by an autoregressive recurrent network we build for chord progression generation (with a network architecture adapted from that in (Waite et al., 2016)), as described in Section 3.3.
Alternatively, we can think of a solo singer as a combination of an idea generator and an an accompanied singer. For an accompanied singer, the information extracted from the given accompaniment track can take several forms such as transcribed pitches and chord progressions. A solo singer learns to generate such information on its own, without reference to an actual accompaniment track.

heading: 3 MODELS

text: To account for the absence of supervised data and the highly complicated spatio-temporal patterns in audio spectrograms, we propose a new adversarial net that features heavy use of GRUs (Cho et al., 2014; Chung et al., 2014), dilated convolutions (van den Oord et al., 2016; Sercu & Goel, 2016), and feature grouping to build our singer models. We provide the algorithmic details below.

heading: 3.1 BLOCK OF GRU-GROUPED DILATED CONVOLUTION-GROUP NORMALIZATION

text:
Network architectures with stacked blocks of GRUs and dilated convolutions have been used to attain state-of-the-art performance in blind musical source separation (Liu & Yang, 2019). In a source separation task, a model learns to decompose, or unmix, different sources (e.g., vocal, piano, bass, drum) from a mixture signal (Rafii et al., 2018). This requires the abilities to model the relationships between different sources as well as the relationships between neighboring time frames. The output spectrograms are also expected to be distortion-less and of high audio quality. For it has demonstrated its capability in source separation, we adopt it as a building block of the singer models. Especially, we want the singer models to also consider accompaniment information.
Specifically, one such block we adopted in our models is a stack of GRU, dilated convolution with feature grouping, and group normalization (Wu & He, 2018). The input to the GRU, the output of the GRU, and the output of the group normalization are summed to form the output of the block. We note that the original ‘D2 block’ used in (Liu & Yang, 2019) uses dilated GRU and uses weight normalization (Salimans & Kingma, 2016) for the dilated convolution layers. However, empirically we find that it is easier for the singer models to converge by replacing weight normalization with group normalization, and using plain GRUs is as good as using dilated GRUs. We refer to our blocks as GRU-grouped dilated convolution-group normalization block (‘G3 block’).

heading: 3.2 SINGER MODELS WITH BEGAN, G3 BLOCKS AND FRAME-WISE NOISES (G3BEGAN)

text:
The accompanied singers and solo singers have to take conditions as part of their input. One desirable property of the models is the ability to generate voices with arbitrary length, as the conditional signal can be of variable length. Besides, the model has to deal with the one-to-many issue mentioned in Section 1, and the absence of supervisory signals. With these issues in mind, we design a GAN architecture for score and lyrics-free voice generation. In particular, we pay special attention to the following three components: 1) the network architecture, 2) the input noises for GAN, and 3) the loss function of the discriminator.
Let us first take a look at two existing GAN models for audio generation: (Engel et al., 2019) and (Donahue et al., 2019). Their generators and discriminators are both based on 2D convolutions, transposed 2D convolutions and dense (linear) layers. The generators take a vector z ∈ RU as the input noise and use transposed convolutions to expand z so that a temporal dimension emerges in the expanded intermediate matrices. The number of temporal frames in the final output depends on the total strides used in all the transposed convolutions. The discriminators take the output of the generators or the real signal as the input, and compress the input matrix with convolution layers until the output becomes a single value represents the prediction of true (real) or false (generated) data.
A main reason why existing models cannot generate variable-length output is the need to expand z by transposed convolution layers. We remedy this by using an architecture consisting of the proposed G3 blocks, and convolutions without strides, for both the generators G(·) and discriminators D(·). Moreover, instead of using a single noise vector, our models take as input a sequence of noise vectors, denoted as Z ∈ RU×T , that has the same temporal length as the desired output Y. Each column of Z is sampled independently from a Gaussian distribution Normal(0, 1). At the first glance, it might feel unnatural to have one noise vector per frame as that may result in fast oscillations in the noises. However, we note that the output of G(·) for the t-th frame depends not only on the t-th column of Z (and C or I), but the entire Z (and the condition matrices), due to the recurrent GRUs in the model. We expect that the GRUs in the discriminator D(·) would force G(·) to generate consistent consecutive frames. Therefore, the effect of the frame-wise noises might be introducing variations to the generation result (e.g., by adjusting the modes of the generated frame-wise features).
As for the loss function of D(·), we experiment with the following three options: the vanilla GAN, the LSGAN (Mao et al., 2017) that adopts the least squares loss function for the discriminator, and the boundary equilibrium GAN (BEGAN) (Berthelot et al., 2017) that adopts an “auto-encoder style” discriminator loss. The D(·) in either GAN or LSGAN is implemented as a classifier aiming to distinguish between real and generated samples, whereas the D(·) in BEGAN is an autoencoder aiming to reconstruct its input. Specifically, in BEGAN, the loss functions lD and lG for the discriminator and generator, as in the case of the accompanied singer, are respectively:
lD =L(X,C)− τsL(G(Z,C),C) , (1) lG =L(G(Z,C),C) , (2)
where X ∈ RK×T is the feature sequence of a real vocal track sampled from the training data, G(Z,C) ∈ RK×T is the feature sequence for the generated vocal track, and L(·) is a function that measures how well the discriminator D(·), implemented as an auto-encoder, reconstructs its input:
L(M,C) = 1
WT ∑ w,t |D(M,C)w,t −Mw,t| , for an arbitrary W × T matrix M , (3)
where we use Mw,t to denote the (w, t)-th element of a matrix M (and similarly for D(M,C)w,t). Moreover, the variable τs in Eq. (1) is introduced by BEGAN to balance the power ofD(·) andG(·) during the learning process. It is dynamically set to be τs+1 = τs+λ(γL(X,C)−L(G(Z,C),C)), for each training step s, with τs ∈ [0, 1]. λ and γ are manually-set hyperparameters.2
Empirical comparison of the performance of GAN, LSGAN and BEGAN can be found in Appendix D. It turns out that the BEGAN-based one, referred to as G3BEGAN hereafter, works the best.

heading: 3.3 CHORD PROGRESSION GENERATOR

text:
We also use auto-regressive RNN to build a chord progression generator for implementing the solo singer. Our chord generator is trained on the Wikifonia dataset (http://www.wikifonia. org/), a set of 6,670 songs in the leadsheet format (i.e., with separated symbolic melody and chord tracks). Its chord vocabulary covers 612 different chords. We set the harmonic rhythm of the chord generator such that a chord change may occur every beat. We desire the chord generator to freely generate chord progressions across different tempo values, time signatures, and keys. Moreover, the generated chord progression has to be rhythmically correct. In achieving so, we encode the tempo,
2We replace C in the above equations by an empty matrix for the free singer, and by I for the solo singer. For the accompanied singer, we note that D(·) has two ways to discriminate true and fake data: from whether G(Z,C) sounds like real singing voice, and from whether G(Z,C) matches the given condition C.
time signatures, and key information (which are available in the Wikifonia dataset) as the initial hidden state of the RNN, and concatenate the chord vector from last time step with the beat position (e.g., 1, 2, 3, 4) of that time step as the input to the RNN. For data augmentation, we transpose the chord progressions found in Wikifonia to 12 possible keys. Once trained, we use the chord generator to create chord progressions for testing the solo singer. With a one-hot encoding, each column of the condition matrix I would have dimension J = 660.
More details of G3BEGAN and the chord generator can be found in Appendices A.1 and A.2.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP AND DATASETS

text:
In our implementation, we use 80-dimensional mel-spectrograms as the acoustic features modeled and generated by the singer models (i.e., K = 80). We use the python package librosa (McFee et al., 2015), with default settings, to compute the mel-spectrograms from audio. A mel-spectrogram is passed to a WaveRNN vocoder (Kalchbrenner et al., 2018) to generate an audio signal from melspectrograms. Our implementation of the WaveRNN vocoder is based on the code from Fatchord.3 Instead of using off-the-shelf pre-trained vocoders, which are typically trained for TTS, we train our vocoder from scratch with a set of 3,500 vocal tracks we separate (by a separation model) from an in-house collection of music that covers diverse musical genres.
One main difficulty in conducting this research is to get vocal tracks for training our singer models. Existing multitrack datasets that contain clean vocal tracks are usually diverse in musical genre and the singing timbre, making it hard for our models to learn the relationship between singing and accompaniment. As we set out to focus on Jazz music in this work, we opt for collecting our own vocal tracks from Jazz music only. We therefore implement our source separation model following the architecture proposed by Liu & Yang (2019), which represents the state-of-the-art as evaluated on the MUSDB benchmark (Rafii et al., 2017). We use the whole MUSDB dataset to train our separation model. This dataset contains clean vocal and accompaniment tracks. However, for constructing the condition matrix A of our accompanied singer, we desire to have separated piano tracks as well (we will explain why shortly). We therefore collect additionally 4.5 hours of Jazz piano solo audio and use them to augment MUSDB for training our source separation model, which can as a result isolate out not only the vocal track but also the piano track from an arbitrary song.
We collect 17.4 hours of Jazz songs containing female voices and 7.6 hours of Jazz songs with male voices. We use the aforementioned separation model to get the vocal tracks. For batched training, we divide the tracks into 10-second sub-clips. Sub-clips that contain less than 40% vocals, as measured from energy, are removed. This leads to 9.9-hour and 5.0-hour training data for female and male Jazz vocals, respectively. 200 and 100 sub-clips are reserved from the training set as the validation set for female singing and male singing, respectively. Each model is trained for 500 epochs. For GAN and LSGAN, we use the models at the 500th epoch for evaluation. For BEGAN, the parameters of the epoch with the best convergence rate (Berthelot et al., 2017) are used for evaluation.
For the accompanied singer, we experiment with extracting pitch-related information from the accompaniment track to form the matrix A that conditions the generation of the vocal track. The assumption here is that whether the generated vocal track is in harmony with the accompaniment track can be largely determined by pitch-related information. For this purpose, we implement a piano transcription model to transcribe the separated piano track, leading to 88-dimensional transcribed frame-wise pitch as the accompaniment condition (i.e., H = 88, as there are 88 piano notes). We implement a piano transcription model with the G3 blocks introduced in Section 3.1, following the training procedure of (Hawthorne et al., 2018). We also implement the model of Hawthorne et al. (2018). Under the same training setting, we find that ours is slightly worse than theirs in note F1 score (0.779 vs 0.794) but slightly better in the note precision score (0.834 vs 0.823). We decide to use our model for the better precision, but the difference of using either of them should be small.
The clips in the training set of our singer models may not contain piano playing (see Table 5 for a summary of the datasets). Even if a clip contains piano playing, the piano may not play across the entire clip. Hence, the models have to learn to sing either with or without the piano accompaniment.
3https://github.com/fatchord/WaveRNN
For performance evaluation, we collect 5.3 hours of Jazz music from Jamendo (https://www. jamendo.com), an online platform for sharing copyright-free music. As said in Section 1, this test set is meant to be public. We apply source separation to the audios, divide each track into 20-second sub-clips,4 and remove those that do not contain piano. Piano transcription is also applied to the separated piano track, yielding 402 20-second sub-clips for evaluation. 402 chord progressions are generated by the chord generator to evaluate the solo singer and to compute the matchness.

heading: 4.2 BASELINES

text:
As this is a new task, there is no previous work that we can compare with. Therefore, we establish the baselines by 1) computing the baseline objective metrics (see Section 4.3) from the training data of the singing models, and 2) using existing SVS systems for synthesizing singing voices.
For the SVS baselines, we employ Sinsy (Oura et al., 2010; Hono et al., 2018) and Synthesizer V (Hua et al., 2019), the two well-known SVS systems that are publicly accessible. For Sinsy, we use the publicly available repository5 to query the Sinsy API (http://sinsy.jp/); we use the HMM version (Oura et al., 2010) instead of the deep learning version as the latter cannot generate male voices. For Synthesizer V, we use their software (https://synthesizerv.com/). We use Sinsy for both objective and subjective tests, but Synthesizer V for subjective test only, for the latter does not provide a functionality to batch process a collection of MIDI files and lyrics.
SVS systems have to take lyrics and a melody to synthesize singing voices. For the lyrics, we choose to use multiple ‘la,’ the default lyrics for Synthesizer V.6 For the melodies, we consider two methods:
1. Vocal transcription from singer training data. We use CREPE to transcribe the separated vocals from the singer training data, and convert it to MIDI format.
2. Piano transcription from the Jamendo testing data. As described in Section 4.1, we have separated and transcribed the piano part of the Jamendo data. Yet, the piano transcription often contains multiple notes at the same time. We use the skyline algorithm (Ozcan et al., 2005) to the transcription result to get a melody line comprising the highest notes.

heading: 4.3 OBJECTIVE METRICS AND OBJECTIVE EVALUATION RESULT

text:
The best way to evaluate the performance of the singer models is perhaps by listening to the generated results. Therefore, we encourage our readers to listen to the audio files provided in the supplementary material. However, objective evaluation remains desirable, either for model development or for gaining insights into the generation result. We propose the following metrics for our tasks.
• Vocalness measures whether an audio clip contains singing voices. There are different publicly available tools for detecting singing voices in an audio mixture (e.g., (Lee et al., 2018)). We choose the JDC model (Kum & Nam, 2019) for it represents the state-of-theart. In this model, the pitch contour is also predicted in addition to the vocal activation. If the pitch at a frame is outside a reasonable human pitch range (73–988 Hz defined by JDC), the pitch is set to 0 at that frame. We consider a frame as being vocal if it has a vocal activation ≥ 0.5 AND has a pitch > 0. Moreover, we define the vocalness of an audio clip as the proportion of its frames that are vocal. The tool is applied to the non-silence part of an audio7 of the generated singing voices only, excluding the accompaniment.
• Average pitch: We estimate the pitch (in Hz) for each frame with two pitch detection models: the state-of-the-art monophonic pitch tracker CREPE (Kim et al., 2018a), and JDC. The average pitch is computed by averaging the pitches across the frames with confidence higher than 0.5 for CREPE, and across the frames that are estimated to be vocal for JDC.
4Please note that this is longer than the 10-second sub-clips we used to train the singer models. This is okay as our model can generate variable-length output.
5https://github.com/mathigatti/midi2voice 6As our models do not contain meaningful lyrics, to be fair the baselines should not contain meaningful lyrics either. We choose ‘la’ because people do sometimes sing with ‘la’ and it has no semantic meaning. An alternative way to get the lyrics is by randomly sampling a number of characters. However, randomly sampling a reasonable sequence of characters is not a trivial task as well.
7The non-silence frames are derived by using the librosa function ‘effects. signal to frame nonsilent.’
• Singing-accompaniment matchness: As detailed in the appendix, to objectively measure matchness, we build a melody harmonization RNN by adapting the chord generator described in Section 3.3. Given a pair of melody and chord sequences, the model computes the likelihood of observing that chord sequence as the output when taking the melody sequence as the model input. We use the average of the log likelihood across time frames as the matchness score. As the harmonization model considers symbolic sequences, we use CREPE to transcribe the generated voices, and Madmom (Böck et al., 2016) to recognize the chord sequence from the accompaniment track.
Several observations can be made from the result shown in Table 1. In terms of the average pitch, we can see that the result of our model is fairly close to that of the singing voices in the training data. Moreover, the average pitch of the generated female voices is higher than that of the generated male voices as expected. We can also see that the Sinsy singing voices tend to have overly high pitches, when the melody line is derived from a piano playing (denoted as ‘testing piano skyline.’).
In terms of vocalness, our models score in general lower than Sinsy, and the singing voices in the training data. However, the difference is not that far. As a reference, we also compute the vocalness of the accompaniments in the training set (denoted as ‘accomp.’) and it is indeed quite low.8
As for matchness, we show in Table 1 the score computed from the real melody-chords pairs of Wikifonia (–7.04) and that from random pairs of Wikifonia (–13.16). We can see that the accompanied singers score higher than the random baseline and the free singer as expected.9 Moreover, the matchenss scores of the accompanied singers are close to that of the singer training data.
Examples of the generated spectrograms of our models can be found in the appendix. From visually inspecting the spectrograms and listening to the result, the models seem to learn the characteristics of the singing melody contour (e.g., the F0 is not stable over time). Moreover, the female singer models learn better than the male counterparts, possibly because of the larger training set.
8We note that Sinsy even scores higher in vocalness than the training data. This may be due to the fact that real singing voices are recorded under different conditions and effects.
9The matchness scores of the free singers are computed by pairing them with the 402 test clips.

heading: 4.4 USER STUDY AND SUBJECTIVE EVALUATION RESULT

text:
We conduct two online, non-paid user studies to evaluate the accompanied singer, the female one. In the first user study, we compare the ‘final’ model (with the number of epochs selected according to a validation set) against two early versions of the model trained with less epochs. In the second one, we compare the proposed accompanied singer with Sinsy and Synthesizer V.
In the first study, we recruit 39 participants to each rate the generated singing for three different accompaniment tracks (each 20 seconds), one accompaniment track per page. The subjects are informed the purpose of our research (i.e., score and lyrics-free singing voice generation) and the user study (to compare three computer models), and are asked to listen in a quiet environment with proper headphone volume. No post-processing (e.g., noise removal, EQ adjustment) is applied to the audio. The ordering of the result of the three models is randomized.
The process of the second study is similar to the first one, but it includes five different accompaniments (randomly chosen from those used in the first user study) and the respective generated/synthesized singing voices. The melodies used for synthesis are those from the piano skyline of the test data, so that our model can be compared with the synthesis methods with the same accompaniment. A separate set of 21 subjects participate in this study. The audio files used in this user study can be downloaded from https://bit.ly/2qNrekv.
Tables 2 and 3 show the result of the two studies. We can see that the model indeed learns better with more epochs. Among the four evaluation criteria, the Sound Quality is rated lower than the other three in both studies, suggesting room for improvement. By comparing the proposed model with the two SVS systems, we see that Synthesizer V performs the best for all the evaluation criteria. Our model achieves better Matchness than Sinsy, and achieves a rating close to Sinsy in Expression.10 In general, we consider the result as promising considering that our models are trained from scratch with little knowledge of human language.

heading: 5 RELATED WORK

text:
While early work on SVS is mainly based on digital signal processing (DSP) techniques such as sampling concatenation (Cook, 1996; Bonada & Serra, 2007), machine learning approaches offer greater flexibility and have been more widely studied in recent years. Hidden Markov models
10We note that Sinsy and Synthesizer V have an unfair advantage on matchness because their singing voices are basically synthesized according to the melody lines of the accompaniment. From Table 3, we see that Synthesizer V does exhibit this advantage, while Sinsy does not. We observe that the Sinsy singing voices do not always align with the provided scores. The fact that Synthesizer V has higher audio quality seem to promote its score in the other criteria. The presence of the result of Synthesizer V seems to also make the subjects in the second study rate the proposed model lower than the subjects do in the first study.
(HMMs), in particular, have been shown to work well for the task (Saino et al., 2006). The Sinsy system, a baseline model in Section 4, is also based on HMMs (Oura et al., 2010). Nishimura et al. (2016) report improved naturalness by using deep neural nets instead of HMMs. Since then, many neural network models have been proposed.
The model presented by Nishimura et al. (2016) uses simple fully-connected layers to map symbolic features extracted from the user-provided scores and lyrics, to a vector of acoustic features for synthesis. The input and output features are time-aligned frame-by-frame beforehand by well-trained HMMs. The input features consist of score-related features (e.g., the key of the current bar and the pitch of the current musical note), and lyrics-related ones (the current phoneme identify, the number of phonemes in the current syllable, and the duration of the current phoneme). The output features consist of spectral and excitation parameters and their dynamic features (Hono et al., 2018), which altogether can then be turned into audio with a DSP technique called the MLSA filter (Imai, 1983).
The aforementioned model has been extended in many aspects. For instance, using convolutional layers and recurrent layers in replacement of the fully-connected layers for learning the mapping between input and output features has been respectively investigated by Nakamura et al. (2019) and Kim et al. (2018b). Using neural vocoders such as the WaveNet (van den Oord et al., 2016) instead of the MLSA filter has been shown to improve naturalness by Nakamura et al. (2019). Rather than using hand-crafted features for the input and output, Lee et al. (2019a) train a model to predict the mel-spectrogram directly from time-aligned lyrics and pitch labels, and then use the Griffin-Lim algorithm (Griffin & Lim, 1984) to synthesize the audio. Modern techniques such as adversarial loss and attention module have also been employed (Lee et al., 2019a). A follow-up work adds a speaker encoder to the network to achieve multi-singer SVS (Lee et al., 2019b). Synthesizer V (Hua et al., 2019), the other baseline model we employ in Section 4, is based on a hybrid structure that uses both deep learning and sample-based concatenation.11
While exciting progress has been made to SVS, the case of score and lyrics-free singing voice generation, to our best knowledge, has not been tackled thus far. Similar to (Lee et al., 2019a), we do not use hand-crafted features and we train our model to predict the mel-spectrograms.
Using neural nets for score-conditioned instrumental audio generation have also been investigated in recent years. However, existing work is mostly concerned with the generation of single notes of, for example, 4-second long (Défossez et al., 2018; Donahue et al., 2019; Engel et al., 2019). A deep neural network that is capable of generating variable-length audio (e.g., a “recurrent generator”) as the proposed singer models do, to our knowledge, has not been much studied.

heading: 6 CONCLUSION

text:
In this paper, we have introduced a novel task of singing voice generation that does not use musical scores and lyrics. Specifically, we proposed three singing schemes with different input conditions: free singer, accompanied singer, and solo singer. We have also proposed a BEGAN based architecture that uses GRUs and grouped dilated convolutions to learn to generate singing voices in an adversarial way. For evaluating such models, we proposed several objective metrics and implemented a model to measure the compatibility between a given accompaniment track and the generated vocal track. The evaluation shows that the audio quality of the generated voices still leave much room for improvement, but in terms of humanness and emotion expression our models work fine.
Score and lyrics-free singing voice generation is a new task, and this work represents only a first step tackling it. There are many interesting ideas to pursue. For example, we have chosen to extract pitch-related information only from the accompaniment track for the accompanied singer, but a more interesting way is to let the model learns to extract relevant information itself. In the near future, we plan to investigate advanced settings that allow for timbre and expression control, and experiment with other network architectures, such as coupling a fine-grained auto-regressive model with a multiscale generation procedure as done in MelNet (Vasquez & Lewis, 2019), using a discriminator that examines different chunks of the generated audio as done in PatchGAN for the vision domain (Isola et al., 2017), or using multiple discriminators that evaluate the generated audio based on multi-frequency random windows as done in GAN-TTS (Bińkowski et al., 2019).
11https://synthv.fandom.com/wiki/File:Synthesizer_V_at_the_Forefront_of_ Singing_Synth (last accessed: Nov. 12, 2019)

heading: A APPENDIX: TRAINING SETTING AND MODEL DETAILS

text:
A.1 G3BEGAN
Singer models with G3BEGAN are trained with Adam (Kingma & Ba, 2014) with 0.0001 learning rate and mini-batch size being 5. Each model is trained with 500 epochs. Gradient norm clipping with magnitude 3 is used.
The generator in G3BEGAN is implemented with a stack of two G3 blocks. Please see Table 4 for details of the network architecture.
A.2 CHORD GENERATOR
The chord generator is aimed to generate chord progressions freely under some given conditions. It supports 12 major and 12 minor keys, 10 tempo options from 60 to 240 BPM, 6 time signature options, and 51 chord qualities (612 chords in total). The conditions, key, tempo, and time signatures, are encoded into one-hot representation and concatenated together as a 40-dimension vector. The model mainly consists with 3 stacked GRU layers, each with 512 hidden variables. The input of each time step is a 524-dimensional vector consisting of a chord embedding and a beat-related one-hot positional encoding (to encourage the model to follow certain rhythmical pattern. This input array passes through a fully-connected layer to 512-dimension and is used as the input of the GRUs. The training data are the leadsheets from the Wikifonia dataset. We augmented the data by rotating the keys, leading to in total 80,040 leadsheets for training.
A.3 MELODY HARMONIZATION MODEL
The melody harmonization model is modified from the chord generator described in Appendix A.2, using additionally the melody tracks found in the Wikifonia dataset. Specifically, the model intends to generate a chord sequence given a melody sequence. Such a model can be learned by using the pairs of melody and chord tracks in Wikifonia. We add the chroma representation of the melody with window size of a quarter-note to the input vector.
The matchness of the real pairs of melody and chord progression in Wikifonia is –7.04±2.91. If we pair a melody with a randomly selected chord progression and calculate the matchness, the score becomes –13.16±3.72.

heading: B APPENDIX: DATASETS

heading: C APPENDIX: EXAMPLES OF THE SPECTROGRAM OF THE GENERATED

text:
SINGING VOICES
Examples of the spectrograms of the generated singing voices can be found in Figures 3 and 4.

heading: D APPENDIX: ABLATION STUDY ON DIFFERENT GAN LOSSES

text: We experiment different GANs, including BEGAN (Berthelot et al., 2017), vanilla GAN (Goodfellow et al., 2014), and LSGAN (Mao et al., 2017), for the case of building the free singer. The output of the discriminators in the vanilla GAN and LSGAN is a single real/fake value. To compare the three GANs as fairly as possible, the discriminators used in vanilla GAN and LSGAN are almost the same as the one used in the BEGAN. The only difference is that the discriminators used in vanilla GAN and LSGAN have an extra average-pooling layer in the output. The validation losses at different training epochs are shown in Figure 5, and the metrics are shown in Table 6. In Figure 5, we can see that, according to our implementation, only BEGAN converges. In Table 6, we can see that the BEGAN model has much higher Vocalness than other models. By listening to the generated singing voices of the GAN and LSGAN models, we find that they are basically noises.