summaries: This paper introduces a new dataset consisting of images of various objects placed on store shelves that are labeled with object boundaries and what are described as “ultrafine-grained” class labels. The accompanying task is to predict the labels of each object given the individual images as well as their spatial layout relative to each other. To solve this task, a deep structured model is used consisting of CNN features for each image which are fed into a linear-chain CRF. To better deal with the large number of classes, pairwise potentials are represented as the multiplication of two lower-rank matrices which represent a sort of “class embedding” for each potential label. Training efficiency is improved by considering an objective based on a form of piecewise pseudolikelihood, which allows for training-time inference to be conducted with linear complexity relative to the number of labels. This objective also allows for easy use of batch normalization for the input features to the CRF model. This model/training procedure are compared against a number of models/training procedures to demonstrate its utility.

summaries: This paper proposed to tackle a large-scale fine-grained object classification problem by approximated CRF. The main motivation is to exploit the spatial conference of object labels to reduce noises in the instance-wise prediction. To this end, the task is formulated by sequential inference problem using CRF. To speed up training, several techniques are applied such as factorized pairwise-potential and approximation of CRF objective.

abstractText: This paper presents a novel deep learning architecture for classifying structured objects in ultrafine-grained datasets, where classes may not be clearly distinguishable by their appearance but rather by their context. We model sequences of images as linear-chain CRFs, and jointly learn the parameters from both localvisual features and neighboring class information. The visual features are learned by convolutional layers, whereas class-structure information is reparametrized by factorizing the CRF pairwise potential matrix. This forms a context-based semantic similarity space, learned alongside the visual similarities, and dramatically increases the learning capacity of contextual information. This new parametrization, however, forms a highly nonlinear objective function which is challenging to optimize. To overcome this, we develop a novel surrogate likelihood which allows for a local likelihood approximation of the original CRF with integrated batch-normalization. This model overcomes the difficulties of existing CRF methods to learn the contextual relationships thoroughly when there is a large number of classes and the data is sparse. The performance of the proposed method is illustrated on a huge dataset that contains images of retail-store product displays, and shows significantly improved results compared to linear CRF parametrization, unnormalized likelihood optimization, and RNN modeling.

authors:
Yoshua Bengio
Aaron Courville
Pascal Vincent

title: Representation learning: A review and new perspectives

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

title: Statistical analysis of non-lattice data

venue: The statistician, pp

authors:
Jane Bromley
Isabelle Guyon
Yann LeCun
Eduard Säckinger
Roopak Shah

title: Signature verification using a

venue: siamese” time delay neural network. In NIPS,

authors:
Gang Chen
Yawei Li
Sargur N Srihari

title: Word recognition with deep conditional random fields

venue: In ICIP,

authors:
Liang-Chieh Chen
Alexander Schwing
Alan Yuille
Raquel Urtasun

title: Learning deep structured models

venue: In ICML,

authors:
Liang-Chieh Chen
George Papandreou
Iasonas Kokkinos
Kevin Murphy
Alan L Yuille

title: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Anna Choromanska
Mikael Henaff
Michael Mathieu
Gérard Ben Arous
Yann LeCun

title: The loss surfaces of multilayer networks

venue: In AISTATS,

authors:
Wenqing Chu
Deng Cai

title: Deep feature based contextual model for object detection

authors:
Santosh K Divvala
Derek Hoiem
James H Hays
Alexei A Efros
Martial Hebert

title: An empirical study of context in object detection

venue: In CVPR,

authors:
Pedro F Felzenszwalb
Ross B Girshick
David McAllester
Deva Ramanan

title: Object detection with discriminatively trained part-based models

venue: IEEE Transactions on pattern analysis and machine intelligence,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Elad Hoffer
Nir Ailon

title: Deep metric learning using triplet network

venue: In International Workshop on SimilarityBased Pattern Recognition,

authors:
Ninghang Hu
Gwenn Englebienne
Zhongyu Lou
Ben Kröse

title: Learning latent structure for activity recognition

venue: In IEEE Int. Conf. on Robotics and Automation (ICRA),

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

authors:
Sham Kakade
Yee Whye Teh
Sam T Roweis

title: An alternate objective function for markovian fields

authors:
Aditya Khosla
Nityananda Jayadevaprakash
Bangpeng Yao
Li Fei-Fei

title: Novel dataset for fine-grained image categorization

venue: In First Workshop on Fine-Grained Visual Categorization,

authors:
Filip Korzeniowski
Gerhard Widmer

title: A fully convolutional deep auditory model for musical chord recognition

venue: In MLSP,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks

venue: In NIPS,

authors:
John Lafferty
Andrew McCallum
Fernando CN Pereira

title: Conditional random fields: Probabilistic models for segmenting and labeling sequence data

authors:
Di Lin
Xiaoyong Shen
Cewu Lu
Jiaya Jia

title: Deep lac: Deep localization, alignment and classification for fine-grained recognition

authors:
Tsung-Yi Lin
Priyal Goyal
Ross Girshick
Kaiming He
Piotr Dollár

title: Focal loss for dense object detection

title: Single shot multibox detector

venue: In ECCV,

authors:
Yuxin Peng
Xiangteng He
Junjie Zhao

title: Neural networks: tricks of the trade

authors:
2017. Shaoqing Ren
Kaiming He
Ross Girshick
Jian Sun

title: Faster R-CNN: Towards real-time object detection

venue: IEEE Transactions on Image Processing,

authors:
Charles Sutton
Andrew McCallum

title: Piecewise training for undirected models

venue: arXiv preprint arXiv:1409.1556,

title: Contextual priming for object detection

venue: International journal of computer vision,

authors:
2011. Yuxuan Wang
DeLiang Wang

title: Cocktail party processing via structured prediction

authors:
Aron Yu
Kristen Grauman

title: Fine-grained visual comparisons with local learning

venue: In CVPR, Jun 2014. Sergey Zagoruyko and Nikos Komodakis. Learning

authors:
Ning Zhang
Jeff Donahue
Ross Girshick
Trevor Darrell

title: Partbased R-CNNs for fine-grained category

authors:
Chang Huang
Philip HS Torr

title: Conditional random fields as recurrent neural networks

venue: In ICCV,

authors: Sutton McCallum

heading: 1 INTRODUCTION

text:
Object recognition is one of the fundamental problems in computer vision. It involves finding and identifying objects in images, and plays an important role in many real-world applications such as advanced driver assistance systems, military target detection, diagnosis with medical images, video surveillance, and identity recognition. Over the past few years deep convolutional neural networks (CNN) have led to remarkable progress in image classification (Krizhevsky et al., 2012; He et al., 2016), and resulted in reliable appearance-based detectors; e.g., (Ren et al., 2015; Liu et al., 2016; Redmon & Farhadi, 2017; Lin et al., 2018).
Fine-grained object recognition aims to identify subcategory object classes, which includes finding subtle differences among visually similar subcategories such as dog breeds, product brands, car models, etc. The differences between classes are often small but always visually measurable, making visual recognition challenging but possible. Some of these datasets (e.g. UT-Zap50K (Yu & Grauman, 2014)) provide each class with an in-vitro image: a catalog or studio image isolated and captured under ideal imaging conditions; other datasets (e.g. Caltech-UCSD Birds (Wah et al., 2011), Stanford Dogs (Khosla et al., 2011), FGVC-Aircraft (Maji et al., 2013)) provide each class with several in-situ images, captured in natural real-world environments. Nonetheless, the image quality is mostly satisfactory for the task of visual classification. In fact, recent studies achieved good performance on fine-grained tasks (Lin et al., 2015; Zhang et al., 2014; Peng et al., 2018).
However, the problem remains extremely difficult when the dataset categories are nearly identical in terms of their visual appearance. In this case, the object categories may be virtually indistinguishable, since the discriminant features are often masked by inadequate observation or visual artifacts. Here we present an ultrafine-grained structured classification dataset; unlike other fine-grained classification datasets, our images are in-situ low-resolution cropped patches whose classes are often virtually indistinguishable by visual inspection alone. Therefore, incorporating additional sources of information to the classifier is imperative. Since the object-patches originate from larger scenes,
we can model contextual relations between the objects based on their geometric layout. This study tackles the challenge of fine-grained, large-scale structured classification, and describes a novel, state-of-the-art technique for this task.
We address the problem of classifying a sequence of objects based on their visual appearance and their relative locations. Our dataset contains photos of retail store product displays, taken in varying settings and viewpoints. Our task is to identify the class of each product at the front of the shelves. The dataset is exclusively characterized by having a distinct geometric object structure made up of sequences of shelves, a large number of classes, and very subtle visual differences between groups of classes in that some classes only differ in size or minor design details. The unique challenges in this task involve (a) large-scale classification: handling the large number of classes in the dataset, and (b) ultrafine-grained structured classification: the fact that the classes are not clearly distinguishable by their appearance but rather by their context. For example, products with an identical appearance but different container volumes are considered different classes (See Fig. 1 and 2).
Context has been used to improve performance for image understanding tasks in various ways (Torralba, 2003; Divvala et al., 2009; Felzenszwalb et al., 2010). Graphical models have been widely applied to visual and auditory analysis tasks, by jointly modeling local features, and contextual relations. The tasks addressed by these models include image segmentation and object recognition (Chen et al., 2018; Yao et al., 2012; Zheng et al., 2015), as well as speech (Wang & Wang, 2012), music(Korzeniowski & Widmer, 2016), text (Chen et al., 2016) and video analysis (Hu et al., 2014).
Few studies have applied deep learning features or detection results to context models: Chen et al. (2015) explored several techniques to learn structured models jointly with deep features that form MRF potentials. Chu & Cai (2018) evaluated the performance of a joint CRF model on Faster R-CNN (Ren et al., 2015) detection results using an a-priori statistical summary for the pairwise potentials. Korzeniowski & Widmer (2016) introduced a two-stage learning model for musical chord recognition: one network learns a single-frame representation, and the other learns the potentials of a linear-chain CRF model using the frame-representations as the CRF input. These models use the vanilla CRF parametrization, which includes pairwise potentials to represent object-pair interactions. They allocate a different parameter to each class pair. This approach, which ignores class similarities, is only sufficient for small sets of distinct classes. In effect, they have solely been tested on OCR datasets, which contain 26 classes (Chen et al., 2015), a chord-recognition dataset with
25 classes (Korzeniowski & Widmer, 2016) and PASCAL VOC 2007 with 20 classes (Chu & Cai, 2018). However, this formulation is not sufficient for a large class-set that contains visually similar classes. Our dataset, which includes many visually similar categories, nearly a thousand classes and a million possible pairwise transitions overall, requires a more advanced learning mechanism. Furthermore, whereas in most previous object recognition studies the visual information was dominant, in our task, context information also makes a significant contribution.
In this study we present a CRF based method that explicitly learns the embedding of classes with respect to their neighbor’s class and appearance. This is achieved by factorizing the CRF pairwise potential matrix to impose the structure of class embedding in a low-dimensional space. Our model learns the factorized parameters, and yields a joint contextual-visual embedding of the classes. The factorization drastically increases the learning capacity of contextual information, but also forms a multimodal likelihood function which is more challenging to optimize. To overcome this, we develop a local surrogate likelihood and apply the proper regularization required for convergence. To train the network, we introduce a pairwise softmax architecture that optimizes a local approximation of the likelihood. Since the global factorized loss function is not convex, we favor optimizing the approximate surrogate likelihood, which allows us to include batch-norm related regularization for the object samples, and achieve dramatic improvement not only in training time and model simplicity but also in terms of the overall performance of the trained model. At test time, dynamic programming techniques are used for efficient exact inference of the classes.
The contributions of this work are the following:
1. Combining deep class embedding into a CRF formulation that enables handling datasets with a huge number of classes.
2. An approximated-likelihood training procedure that is both computational efficient and, unlike exact CRF likelihood, enables us to incorporate batch-normalization into the training procedure.

heading: 2 CRF WITH CLASS EMBEDDING

heading: 2.1 MODEL FORMULATION

text:
Our input data are sequences of image patches which correspond to horizontal layouts of individual product, as demonstrated in Fig. 3.
We want to predict the sequence of the target labels y = [y1, · · · , yn], given a sequence of observations x = [x1 · · · , xn]. Standard classification approaches use a CNN to predict each object-level observation individually, which implicitly assumes independence between object samples. In order to include context in the classification process, we model the sequences as a CRF.
Linear-chain Conditional Random Field (LC-CRF) (Lafferty et al., 2001) is a type of discriminative undirected probabilistic graphical model, whose conditional distribution p(y|x) obeys a conditional Markov property. The joint probability distribution of a linear-chain CRF is:
p(y|x) = 1 Z n∏ t=1 ϕ(yt, xt, yt−1) (1)
where x is the sequence of observation feature vectors, y is the corresponding sequence of the target labels, ϕ the model’s potential function, Z the partition function defined as the global probability
normalization over all possible sequence label-assignments of length n, and y0 = 0. Assume that the potential function is defined as:
ϕ(yt, xt, yt−1) = exp(y > t−1Pyt + x > t Uyt + b >yt) (2)
where matrix P the pairwise potentials matrix, U the unary potentials, and vector b the label bias, are all model parameters, and we use a one-hot encoding for the labels. The likelihood function, therefore, is log-linear and concave.
We can train a local CNN to classify individual objects, and then interpret the hidden layer activations as a non-linear representation of the input image. Similar to the concept of transfer-learning, we can now discard the CNN softmax layer, and use the convolutional layers to compute the featurevectors of the input images. For image-patch xt we define the feature vector ht = h(xt) as the activations of the last hidden fully-connected or global-average-pooling (GAP) layer (Bengio et al., 2013; Lin et al., 2013), and use it as the CRF input observation feature vector:
ϕ(yt, ht, yt−1) = exp(y > t−1Pyt + h > t Uyt + b >yt). (3)
The score function (3) is still concave, but its input is computed from a non-concave source, a deep CNN. The rationale for using a deep representation for the input images is clear: as introduced by Krizhevsky et al. (2012), the immense complexity of the visual object recognition task requires a model with a very large learning capacity. Convolutional layers provide the structure required for learning visual features of the unary input. We aim to craft a suitable structure to learn the pairwise contextual relations as well.
CRF was originally applied to language processing tasks such as Part of Speech (POS) tagging and Named Entity Recognition (NER) (Lafferty et al., 2001). In most applications of CRF to either language or image understanding, there are no more than a few dozen different classes. In our dataset, we have hundreds of classes. The pairwise transition between two classes has nearly a million possible states, whereas the CRF function (3) has a log-linear form, and contains a single parameter per transition ordered-pair. In order to properly learn and generalize the massive variety of possible neighboring patterns, we enforce a structure on the pairwise potential matrix: the goal is to learn neighboring-class embedding in a feature vector space. For this purpose, we define a low-dimensional decomposition of the pairwise potential matrix P as the product of the left-side neighbor embedding matrix R and the class embedding matrix Q:
P = R>Q. (4)
The columns of Q are low-dimensional embeddings of the target classes, and the columns of R are embeddings of the classes of the left-side object. Assigning the matrix factorization (4) to the CRF potential function (3) we get:
ϕ(yt, ht, yt−1) = exp(y > t−1R >Qyt + h > t Uyt + b >yt). (5)
The objective function is no longer linear or concave with respect to the network parameters, but deep learning training techniques have been shown to yield good results for non-convex optimization tasks (Choromanska et al., 2015). Hence, we apply deep learning approaches not only for the input image representations, but also for the neighboring transition parameters.

heading: 2.2 TRAINING

text:
The CRF model defined above can be trained in a supervised manner by maximizing the loglikelihood of all sequences in the training dataset (Lafferty et al., 2001):
L(R,Q,U, b) = S∑
i=1
log p(yi|h(xi)) (6)
where the vector yi contains the ground-truth labels of the ith sequence, h(xi) contains the corresponding object feature vectors of the sequence of observations, i goes over the sequences in the training data, and the loss function is defined in (1) with the potentials (5). Since the underlying graph is loop-free, it is tractable to compute the likelihood function and its gradient using the forward-backward algorithm (Sutton & McCallum, 2006). However, the optimization is relatively
slow for a large number of classes, because its complexity is quadratic in the number of possible classes. In order to speed up the training process, we can estimate the parameters locally, by optimizing an approximate objective function. A local approximation of the likelihood would require samples of individual objects and their immediate neighbors rather than entire sequences.
Linear-chain CRFs were originally introduced as an improvement on the Maximum Entropy Markov model (MEMM) (McCallum et al., 2000), which is essentially a Markov model in which the transition distributions are given by a logistic regression model. The main difference between CRFs and MEMMs is that a MEMM uses per-state exponential models for the conditional probabilities of next-states given the current-state, whereas the CRF has a single exponential model for the joint probability of the entire sequence of labels given the observation-sequence. CRF and MEMM can be written with the same set of parameters. The MEMM directed graphical modeling in our case is:
p(y|h(x)) = ∏ t p(yt|ht, yt−1) (7)
where p(yt|ht, yt−1) = 1
Z(t) exp(y>t−1R >Qyt + h > t Uyt + b >yt) (8)
One major advantage of MEMMs over CRFs (and HMMs) is that training can be considerably more efficient. Unlike CRFs, in MEMMs the parameters of the maximum-entropy distributions used for the transition probabilities can be estimated for each transition distribution separately. When applying MMEM for inference it suffers from the label bias problem (Lafferty et al., 2001; Kakade et al., 2002) which may lead to a drop in performance in some applications. Here, however, the MEMM objective is used only as a local approximation to learn the parameter set of the linear-chain CRF model whereas the test time inference uses a global normalization of CRF modeling and thus avoids the label bias problem. The objective function is now defined as the conditional probability of the current-object class, given the class of the left-side neighbor object:
L = ∑ i ∑ t log p(yi,t|hi,t, yi,t−1) (9)
where i goes over the sequences and t goes over the objects in the sequence, hi,t is the object CNNbased representation, yi,t is the true class label and p is as defined at (8). Note that the computational complexity of the MEMM likelihood (9) is linear in the number of classes unlike the CRF likelihood whose computational complexity is quadratic.
This surrogate likelihood function whose samples are pairs of objects and the corresponding neighboring labels can be used at train time to accelerate the training process. Since the class-embedding CRF objective is non-convex, there is no theoretical reason to prefer global training of sequencelevel samples over the fast local approximation of pairwise object-level samples. Rather, the approximate likelihood allows incorporating straightforward batch-normalization to whiten the class embeddings, and intensifies the stochastic nature of the optimization process by batching large numbers of randomly sampled pairs form the entire dataset.
In fact, as we empirically show in the next section, optimizing the local approximate likelihood with object-level batch-normalization yields better results than optimizing the unnormalized global LC-CRF likelihood. In appendix 5.2 we review standard likelihood approximation strategies for efficient CRF training and show that the training method we use in this study can be viewed as a simplified version of the piecewise-pseudolikelihood approximation (Sutton & McCallum, 2007).

heading: 2.3 FEATURE SCALING WITH BATCH NORMALIZATION

text:
In optimization, feature standardization or whitening is a common procedure that has been shown to reduce the convergence rates (Orr & Müller, 2003). In deep neural networks, whitening the inputs to each layer may also prevent converging into poor local optima. However, training a deep neural network is complicated by the fact that the inputs to each layer are affected by the parameters of all preceding layers, and need to continuously adapt to the new distribution. The batch-normalization (BN) (Ioffe & Szegedy, 2015) method draws its strength from making normalization part of the model architecture and performing the normalization for each training mini-batch.
In our model, we found it advantageous to whiten the input features of the softmax layer. Since we use pre-trained CNN layers we can standardize the visual features by an offline pre-processing
stage. In contrast, the embeddings are jointly learned with the softmax layer; hence we use the batch-normalization (Ioffe & Szegedy, 2015) method to learn their mini-batch normalization during the training process. In fact, since the input of the embedding layer is a one-hot vector, the batchnormalization process directly standardizes each feature in the embedding space.
A major advantage of the approximate-likelihood we use is that unlike sequential models such as CRF, it is very simple and effective to apply embedding batch normalization to each neighboringlabel sample.

heading: 2.4 INFERENCE

text: At test time, global classification is applied to the linear-chain CRF. Dynamic programming algorithms may be used for efficient and exact inference as follows: the Viterbi algorithm finds the most probable sequence label assignment, and the Forward-Backward algorithm extracts the marginal probability of each item by summation over all possible assignments (Sutton & McCallum, 2006). Note that although the training is done by local likelihood approximation, and we assumed that the predecessor label is known, at test time we apply the global normalization over all possible object sequences. These training and inference methods are summarized in Table 1. Fig. 4 shows an illustration of the training architecture. In appendix 5.1 we visualize the embedding space and shed light on the nature of the information learned.

heading: 3 EXPERIMENTS

heading: 3.1 THE DATASET

text:
Our dataset contains 76,081 sequences of 460,121 single-object images of fixed-size image patches, originated from 24,024 in-situ photos of retail store displays taken in supermarkets and grocery stores. The objects are the inventory items positioned at the front of the displays, and the classes are 972 possible stock-keeping-unit (SKU) unique identifiers. Each object was originally annotated by its class label and bounding-box coordinates. The image patches were cropped and reshaped into single-object images of size 150×450 pixels, and grouped into shelves; i.e., sequences of horizontal layouts, arranged from left to right. Hence, the problem we address is sequence-level classification rather than detection. Sequence lengths can vary from 2 to 32, and are typically between 4-12. The average sequence length is 6 and the length standard deviation is 2.4. To perform k-fold crossvalidation, we split the dataset into 5 mixes of 80% training and 20% testing.
Many groups of classes belong to the same archetype, and only differ in terms of minor details such as volume, flavor, nutrient content etc. They often share similar visual features, which makes appearance-based classification very difficult. On the other hand, the object layout behavior is very coherent: it is dictated by the supplier planograms (specified product layouts) and extracted from the image realograms (observed product layouts). Although realograms are non-deterministic by nature, consistent semantic patterns are frequently spotted. Class transition behavior may be discovered, revealing tendencies of pairs to appear as left-to-right neighbors, and individual classes to appear multiple times successively (Fig. 2). The unique challenges we faced in our task are derived from the large number of visually similar classes, which co-occur in distinct structures in large-scale images. Since the images capture arbitrary subsections of the shelf displays, the visual appearances of object sequences in shelves vary unpredictably in terms of their relative positions and occasional unnoticed or absent elements. Nevertheless, the co-occurrence data statistics remains stable in most cases, which justifies stationarity and Markovity assumptions for the structure modeling.

heading: 3.2 COMPARISONS WITH OTHER METHODS

text:
In order to validate the performance of our method we implemented several alternatives. They are all based on the same contextless CNN local information and only differ in the way they learn the object contextual information from the training dataset and integrate the context model with the local CNN. The following list outlines the major context models we implemented. The full list of experiments we conducted and their implementation details is in appendix 5.4.
Unary The baseline comparison model is the original CNN we trained without any contextual information.
Pairwise Statistics Based on Chu & Cai (2018), we created a CRF model with unary potentials taken from the CNN classifier prediction results, and the pairwise potentials are pairwise statistics that are estimated from the training dataset.
Mixture of Statistics CRFs Still relying on the pairwise statistics summary model, we can also model global context information. We clustered the sequences into a mixture of k Markov models, using the Expectation-Maximization (EM) algorithm.
Log-Linear CRF We implemented both the log-linear (3) and factorized (5) parametrizations of the linear-chain CRF , and trained each one with both global maximum likelihood (6) and local approximate likelihood (9) methods.
Approximate Factorized CRF with BN: This is the model proposed in this study: the CRF pairwise weight matrix is factorized as defined in Eq. (5) and is thus enhanced into a much richer, but non-convex model. To optimize this non-convex objective we need to whiten the embedding features during training. This is achievable by using the surrogate likelihood (9) for local approximate training, and adding a BN layer after the embedding layer. The local approximate training is not only significantly faster, but also supports the integration of BN which significantly improves performance. The network is trained as described in subsection 2.2. Full implementation details are given in appendix 5.3.
Recurrent Neural Network Another modeling option for a sequence estimation is the BiDirectional Recurrent Neural Network (Schuster & Paliwal, 1997) with LSTM (Hochreiter & Schmidhuber, 1997) as memory block (BiLSTM). This model computes the posterior distribution of the current object label based on all the visual information provided by the CNN: p(yt|x) = p(yt|x1, ..., xn). The BiLSTM architecture learns a context vector ct for each object, which encapsulates the bidirectional information in the sequence input observations transferred from the CNN output h1, ..., hn, and learns a softmax prediction p(yt|ct) for each object label. The softmax output layer provides a separate prediction for each class and thus ignores class similarities. In our domain, the most important information in addition to the object local appearance is the label relations between neighboring objects which are not captured here. As hypothesized, this approach did not outperform the original contextless CNN. We further elaborate in appendix 5.4.

heading: 3.3 CLASSIFICATION RESULTS

text: The table in Fig. 5 lists the results in terms of model error rate, indicates the incremental improvement in accuracy over model variations, and shows that the non-linear method based on batchednormalized class embedding yields significantly better results than the other alternatives. Note that the approximate objective is much faster, and enables the BN whitening of class embeddings in the factorized CRF formulation. The graph in Fig. 5 depicts the Precision-Recall curve measured for the different methods by applying different confidence thresholds. It shows how greatly the method improved performance for the practical objective of maximizing recall while preserving high (90%+) precision: Our method achieved a recall of 85.75%, whereas the unary baseline recall was only 79.97%, and the second best alternative was 82.46% - all preserving the same 90% precision. It is worth pointing out that our benchmark is considerably large, which means that we correctly identified around 5,000 more objects than the unary model, and 3,000 more objects than the second best alternative.

heading: 4 CONCLUSION

text:
We introduced a novel technique to learn deep contextual and visual features for fine-grained structured prediction of object categories, and tested it on a dataset that contains spatial sequences of objects, and a large number of visually similar classes.
Our model clearly outperforms all the other tested models. This architecture appears to be the most straightforward generalization of a context-less classifier to become context-dependent when both the input and the context data require a large learning capacity: the network learns deep feature vectors for neighboring classes, analogously to the learned deep input representations. The Markovity and stationarity assumptions make it sufficient to train with individual objects as samples to enrich the training data diversity, allow for simple embedding batch normalization, and boost the non-convex optimization process both in terms of time and performance.

heading: 5 APPENDICES

heading: 5.1 CLASS EMBEDDING ANALYSIS

text:
As a byproduct of the classification model we also obtain a low-dimensional embedding of the different classes. Each column of the neighbor embedding matrix R is a vector representation of the corresponding class. A common similarity metric is the cosine of the angle between the vectors. We can measure the distance between classes by the cosine of their vector representation. Fig. 6 shows several examples of an object class and its most similar classes. We can see that this similarity does not reflect visual appearance similarity, e.g. in the second example the similar classes have very different colors. This situation has been studied extensively for the linguistic problem of word embedding. The goal of word embedding algorithms is to represent similar words by similar vectors. It is often useful to distinguish between two kinds of similarity or association between words (Schütze & Pedersen, 1993). Two words are said to have first-order co-occurrence if they are typically nearby each other (e.g. wrote is a first-order associate of book or poem). Two words have second-order co-occurrence if they have similar neighbors (e.g. wrote is a second-order associate of words like said or remarked). Secondorder word similarity is thus expected to capture a semantic meaning and measure the extent to which the two classes are replaceable based on their tendencies to appear in similar contexts. In Fig. 6 we show that object class embedding captures second-order information. Proximity here corresponds to the mutual tendency to have similar neighbors. We can see in the figure that similar classes, although looking visually different, represent products of similar container-types, volumes and brands.
Visual similarity and second-order semantic similarity are based on two profoundly different criteria, and may be uncorrelated or even have a negative correlation in some cases as we demonstrate in Fig. 7: classes are visually close when it is easy to confuse them based on their visual appearance, and are semantically close when it is statistically reasonable to switch one for the other on a shelf (i.e. “synonymous” classes). The rows in Fig. 7 contain classes that are visually close but semantically far; i.e., they look alike but tend to appear in different contexts, whereas the columns contain classes which are semantically close but visually far; i.e., they look different, but tend to appear in similar contexts. The examples from the retail world refer to classes of similar brands but with different form-factors or volumes, which tend to appear in different displays in stores. A speech analogy would be comparing homophones (e.g. meet vs. meat, sale vs. sail) with synonyms (e.g big vs. large, fast vs. quick).
It is hence clear why these two types of similarity contribute two different types of information, and need to be used jointly for the task of object classification. The visual similarity is relevant for the visual image information whereas the class similarity in the embedded space is relevant for the contextual information.
To demonstrate the two types of similarity representations learned, we sampled activations from the two types of feature vectors and visualized the two similarity spaces: The visual-similarity feature vector is the activation vector at the end of the CNN. We use a t-SNE to reduce the 2048D space of h2048×1 into a 2D space. Fig. 8a portrays examples of classes which are often visually distinguishable, whereas the samples in Fig. 8b belong to classes of visually similar red-labeled bottles, which are clearly not visually separable. This further emphasizes the importance of a supplementary source of information required for the classification task, which cannot be extracted from the visual features alone (i.e. cannot be extracted from improved CNN architectures, Siamese networks etc.)
To visualize the semantic-similarity space we perform a t-SNE to reduce the 32D space of R32×972 columns into a 2D space. Fig. 9 shows the embeddings of the 972 classes in the semantic similarity space. Note that
the red-labeled bottle classes are now completely distinct (circled in green), but they are very close to visually different classes which are semantically similar (e.g each red-labeled bottle is embedded in close proximity to a green bottle class with the same volume, shape and form-factor). Thus many visual similarities are disrupted in the semantic similarity space whereas many semantic similarities are disrupted in the visual similarity space. This analysis depicts examples of classes that are close in one space but distal in the other, and sheds light on the challenges our of dataset, the types of similarities required for object classification, and the actual information successfully learned by our model.

heading: 5.2 LOCAL LIKELIHOOD APPROXIMATION

text:
In this appendix we show how the objective function that we used for optimization is related to previously suggested approaches. Pseudolikelihood (Besag, 1975) is a classical approximation of the CRF likelihood function that simultaneously classifies each node given its neighbors in the graph. The pseudolikelihood objective function only depends on the object and its Markov blanket. The pseudolikelihood of our model (1) is:
log p(y|x) = ∑ t log p(yt|yt−1, yt+1,x) (10)
where p(yt|yt−1, yt+1,x) is
exp(y>t−1Pyt + y > t Pyt+1 + x > t Uyt + y > t b)∑
a exp(y > t−1Pa+ a >Pyt+1 + x>t Ua+ a >b)
. (11)
Piecewise training (Sutton & McCallum, 2005) is a heuristic method to predict the graph factors from separate “pieces” of the graph. The piecewise objective function is equivalent to the likelihood function of a node-split graph (Sutton & McCallum, 2007), which contains all the single-factor components split from the original graph. Using the CRF notation in Eq. (1), the piecewise likelihood approximation in our case is:
log p(y|x) = ∑ t log ϕ(yt, xt, yt−1)∑ a,b ϕ(a, xt, b) . (12)
Note that due to the term in the denominator, computing the piecewise likelihood is quadratic in the number of classes. Piecewise Pseudolikelihood (PWPL) is the standard pseudolikelihood applied to the node-split graph. Its computation is efficient because the objective function is simply the sum of local conditional probabilities. In our case, applying the pseudolikelihood approach on the piecewise objective (12) would give us the following PWPL form:
log p(y|x) = ∑ t log( ϕ(yt, xt, yt−1)∑ a ϕ(a, xt, yt−1) · ϕ(yt, xt, yt−1)∑ a ϕ(yt, xt, a) ). (13)
Sutton & McCallum (2007) showed that in many cases the PWPL has better accuracy than standard pseudolikelihood, and in some scenarios has nearly equivalent performance to piecewise approximation and even to global maximum likelihood. The first term inside the log function is equivalent to the forward MEMM objective function (8) that we used. The second term can be written in the form:
p(yt−1|yt) = exp(y>t−1Pyt)∑ a exp(a >Pyt) . (14)
This term is independent of the CRF visual input. The PWPL approximation can be thus expressed as: log p(y|x) = ∑ t (log p(yt|yt−1, xt) + log p(yt−1|yt)). (15)
Hence the MEMM-like objective function we used (7) can be viewed as a simplified version of the piecewisepseudolikelihood objective (13) that was found to be the preferred likelihood approximation for language processing tasks (Sutton & McCallum, 2007).

heading: 5.3 IMPLEMENTATION DETAILS

text:
The benchmark contains 76,081 sequences of 460,121 single-object images, originated from 24,024 photos of store displays. Each object is labeled as one of 972 different classes. Sequence lengths can vary from 2 to 32, and are typically between 4-12. The average sequence length is 6 and the length standard deviation is 2.4. To perform k-fold cross-validation, we split the dataset into 5 mixes of 80% training and 20% testing.
We first train a ResNet50 CNN (He et al., 2016) from scratch to compute the hidden representation vector hs×1 for each image-patch. In our implementation the hidden layer size (after global average pooling) was s = 2048. Then, as a preprocessing step for the CRF model, we calculate the mean and standard deviation of each feature of the hidden representation vector from the training dataset: µs×1, σs×1.
The number of classes in our dataset is m = 972, and the class embedding dimensionality we use is d = 32. We learn a class embedding matrixQd×m, a neighbor embedding matrixRd×m, a unary potential matrixUs×m and a bias vector bm×1. We initialize the bias parameter to 0 and the weight parameters with random Gaussian samples N (0, 0.01) for symmetry breaking. We train the network as described in subsection 2.2, using SGD with mini-batches of size 128, and maximizing the log-likelihood function (9) with p as defined at 8 and l2 regularization factor λ = 5 · 10−4 for all network parameters. The training samples in each mini-batch are object-pairs selected randomly from the benchmark. Each sample is a horizontally adjacent pair of left-label ym×1 in a one-hot encoding, and right image-patch representation hs×1. If the object has no left neighbor, y is assigned to the zero vector, so the pairwise related parameters are not affected. After convergence of the training stage, we apply the batch-normalization inference procedure (Ioffe & Szegedy, 2015) to standardize the context embedding matrix R by the training population statistics, and multiply it by the target embedding matrix Q to restore the CRF pairwise potential matrix P for the inference stage. At test-time, we compute the CNN representation vector h for each object in the sequence, normalize each of its features with the precalculated µ and σ, and classify the objects as described in subsection 2.4.
The experiments ran on GeForce GTX TITAN X NVIDIA-GPU, CUDA V8.0.44 and CuDNN 5.1. A single epoch of the baseline unary system took 46 sec, a global optimization algorithm took 780 sec and our local
optimization took 47 sec. The local training procedure is much more efficient than computing the global maximum likelihood, because its time complexity is linear in the number of classes, whereas the global training procedure is quadratic in the number of classes. The most important contribution of the approximate likelihood, however, is in performance due to its ability to add batch-normalization to the nonlinear objective. At test time it took less than 0.1 seconds to classify all the objects in a single image.

heading: 5.4 FULL EXPERIMENTS

text:
Below is a detailed list of the different experiments we conducted, including implementation details, results and analysis.
Baselines:
Unary The baseline comparison model is the original CNN we trained without any contextual information.
Pairwise Statistics Based on the work of Chu & Cai (2018), we created a CRF model with unary potentials taken from the CNN classifier prediction results, and the pairwise potentials are pairwise statistic Pij = p(j|i) = p(yt = j|yt−1 = i) that are estimated from the training dataset. In other words, the context information is modeled by a stationary first-order Markov chain. No additional NN training is applied. The only single parameter we need to set is the relative weight of the unary and pairwise potentials. This weight, which adjusts the trade-off between the local appearance and the contextual information, was selected via crossvalidation. This approach did not increase the total accuracy, although it moderately increased recall while preserving precision in limited cases.
Mixture of Statistics CRFs Still relying on the pairwise statistics summary model, we can also model global context information; for instance, the fact that all the objects in the sequence have the same label. We clustered the sequences into a mixture of k Markov models, using the Expectation-Maximization (EM) algorithm. The training sequences are eventually split into k different groups, and pairwise statistics are separably calculated for each one of them. At test time, the most probable Markov model is selected for each sequence, and the corresponding pairwise statistics CRF model is used. The mixture of k Markov models method was examined with k values ranging from 2 to 16. It revealed chain groupings to some extent, but did not lead to a significant improvement in the overall classification performance compared to the baseline CRF model of k = 1.
Log-linear CRF This method learns the log-linear parameters of the linear-chain CRF (3). We implemented both global and local approximate likelihood training methods and tried both l1 and l2 regularizations for the pairwise potential matrix. We also applied standardization on the one-hot input vectors. The results in all cases were comparable, and provided some improvement over the baseline contextless classifier.
Our Method:
Class-embedding CRF: This is the model proposed in this study, where the CRF is enhanced into a much richer, but non-convex model by extending the pairwise weight matrix as defined in Eq. (5). The network is trained as described in subsection 2.2, using the surrogate likelihood (9), with l2 weight regularization, and batch-normalization for the embedding features. This approach provides state-of-the-art results and is significantly better than any other alternative we tried 5.
Ablation Study:
We performed an ablation analysis aimed at isolating the effect of the various innovations suggested. Each experiment uses the same configuration as in our method with only one alteration.
Feature Scaling We tested the following variants:
• removing the batch-normalization layer entirely. • removing the whitening of the CNN activations. • Whitening the one-hot vectors at the input of the embedding layer instead of batch-normalizing its
output.
In each case the results became much worse and were comparable to the contextless unary network. On the other hand, when adding or removing the scale and shift from the BN parameters, the results remained comparable to our state-of-the-art results. This suggests that the BN layer has enormous impact since it whitens the embedding activations during training, similar to the whitening applied for the CNN activations.
Regularization We tested l1, l2 or no regularization for the embedding weights. The results were significantly better with l2 regularization, which encourages all the weights for each class in the embedding space to be used in training.
Pairwise Matrix Factorization We considered other variants of the class embedding concept in which the embedding parameters of the target and neighboring labels are tied. For that purpose, we impose the structure of the embedding matrix R on the current class as well as the neighboring class. The pairwise potential in this case is factorized as P = R>DR to get the same embedding for the class and its neighbor. We may also apply the class embedding on the unary potentials matrix by factorizing U = V >R. In these parametrizations, applying the embedding-batch-norm requires parameter tying between the softamx inputs and the softmax weights, and thus compromises the effectiveness of the batch normalization process.
Surrogate Likelihood Variances Global optimization of LC-CRF is not only much more time-consuming, but also lacks the ability to apply a straightforward batch normalization strategy, since the weights are shared in multiple locations in each sample in the mini-batch, and the sample-level activations have different statistics for each sequence element. Similar problems appears in other known methods of local likelihood approximation such as piecewise, pseudolikelihood and Piecewise-Pseudolikelihood (PWPL) which are close variants of our local training model (See details in appendix 5.2). Applying an embedding-batch-norm to the pseudolikelihood or PWPL methods would once again require parameter tying between the softamx inputs and the softmax weights. However, the PWPL in our case can be reduced to the form of a forward term which is equivalent to the MEMM-like objective (8) and an additive backwards term which is independent of the CRF input. Hence, the MEMM-like objective function is theoretically highly related to PWPL. In the appendix 5.2 we explain how our surrogate likelihood can be viewed as a simplified form of the PWPL objective.
Different CNNs We tested our model with various CNN architectures: ResNet50 showed minor improvement in unary CNN performance over Alexnet (Krizhevsky et al., 2012) and VGG (Simonyan & Zisserman, 2014), and comparable improvement when incorporating our context-aware methods. Note that local visual information is often insufficient in our data, hence adding context is very helpful even with very strong CNNs.
Different data We cross-validated our methods on 5 different train-test splits and obtained comparable results and small variances for the different mixes.
Increased Learning Capacity We tried increasing the model’s non-linearity by adding another fully connected layer and nonlinear ReLU between the one-hot vector input and the fully connected embedding layer. We also tried learning the embedding in a higher dimensional space. These enhancements, however, did not improve performance, and turned out to be redundant.
Other Approaches:
We also tried the following non-CRF approaches, which were not useful for our task:
Recurrent Neural Network Another modeling option for a sequence estimation is the Bi-Directional Recurrent Neural Network (Schuster & Paliwal, 1997) with LSTM (Hochreiter & Schmidhuber, 1997) as memory block (BiLSTM). In this approach we compute the posterior distribution of the current object label based on all the visual information provided by the CNN: p(yt|x) = p(yt|x1, ..., xn). The BiLSTM architecture learns a context vector ct for each object, which encapsulates the bidirectional information in the sequence input observations transferred from the CNN output h1, ..., hn, and learns a softmax prediction p(yt|ct) for each object label.
This approach, however, did not outperform the original contextless CNN. In our case the visual features of the neighbors hardly provide any additional information to the local visual features. The most important information, in addition to the object local appearance, is the label relations between neighboring objects which are not captured here; the BiLSTM network uses a softmax output layer that provides a separate prediction for each class and thus ignores class similarities. Although there is some structural similarity between RNN techniques and our local likelihood approach for CRF, the underlying probabilistic model is very different. Hence, CRFs are preferable for the element-wise classification of observed sequences because (a) they can explicitly learn second-order class similarity which is often the dominant source of contextual information, and (b) the Markovity assumption provides an optimal solver over the entire sequence.
Similarity Networks An additional noteworthy approach to identifying visually similar classes involves using an architecture which receives multiple samples as training input and compares pairs of samples in order to better discriminate between classes based on their visual features. These methods include the Siamese Network (Bromley et al., 1994), Triplet Network, (Hoffer & Ailon, 2015), Pseudo-siamese and 2-Channel Networks (Zagoruyko & Komodakis, 2015).
In our case, however, such approaches were unhelpful due to the lack of labeled attributes, the limited priors and the ultrafine-grained nature of our dataset. For example, attributes such as object volume or flavor are often visually unmeasurable. Furthermore, these approaches ignore class neighbor information which is usually the dominant source of contextual information in observed sequences. The futility of such methods can be understood from the analysis in appendix 5.1.


summaries: This paper proposes an approach that uses GAN framework to generate audio through modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Experiments on NSynth dataset show that it gives better results then WaveNet. The most successful deep generative models are WaveNET,  Parallel WaveNet and Tacotran that are applied to speech synthesis, the method should be tested for speech synthesis and compared with WaveNet, Parallel WaveNet as well as Tacotran.

summaries: This paper proposes a strategy to generate audio samples from noise with GANs. The treatment is analogous to image generation with GANs, with the emphasis being the changes to the architecture and representation necessary to make it possible to generate convincing audio that contains an interpretable latent code and is much faster than an autoregressive Wavenet based model ("Neural Audio Synthesis of Musical Notes with WaveNet AutoEncoders" - Engel et al (2017)). Like the other two related works (WaveGAN - "Adversarial Audio Synthesis" - Donahue et al 2018) and the Wavenet model above, it uses the NSynth dataset for its experiments.

abstractText: Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure but have slow iterative sampling and lack global latent structure. In contrast, Generative Adversarial Networks (GANs) have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.1

authors:
Mathieu Andreux
Stephane Mallat

title: Music generation and transformation with moment matching-scattering inverse networks

venue: In International Society for Music Information Retrevial Conference,

authors:
Martin Arjovsky
Soumith Chintala
Léon Bottou

title: Wasserstein generative adversarial networks

venue: Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning,

authors:
David Berthelot
Thomas Schumm
Luke Metz

title: BEGAN: Boundary equilibrium generative adversarial networks

venue: arXiv preprint arXiv:1703.10717,

title: Estimating and interpreting the instantaneous frequency of a signal

venue: Proceedings of the IEEE,

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale GAN training for high fidelity natural image synthesis

venue: In International Conference on Learning Representations,

authors:
Alexandre Defossez
Neil Zeghidour
Nicolas Usunier
Leon Bottou
Francis Bach

title: Sing: Symbol-to-instrument neural generator

venue: Advances in Neural Information Processing Systems

authors:
Sander Dieleman
Benjamin Schrauwen

title: End-to-end learning for music audio

venue: In ICASSP, pp. 6964–6968,

title: The phase vocoder: A tutorial

venue: Computer Music Journal,

authors:
Chris Donahue
Julian McAuley
Miller Puckette

title: Adversarial audio synthesis

venue: In ICLR,

authors:
Jesse Engel
Cinjon Resnick
Adam Roberts
Sander Dieleman
Douglas Eck
Karen Simonyan
Mohammad Norouzi

title: Neural audio synthesis of musical notes with WaveNet autoencoders

authors:
Philippe Esling
Axel Chemla-Romeu-Santos
Adrien Bitton

title: Bridging audio analysis, perception and synthesis with perceptually-regularized variational timbre spaces

venue: In Proceedings of the 19th International Society for Music Information Retrieval Conference,

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Ishaan Gulrajani
Faruk Ahmed
Martin Arjovsky
Vincent Dumoulin
Aaron C Courville

title: Improved training of Wasserstein GANs

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: GANs trained by a two time-scale update rule converge to a local Nash equilibrium

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A Efros

title: Image-to-image translation with conditional adversarial networks

authors:
Yanghua Jin
Jiakai Zhang
Minjun Li
Yingtao Tian
Huachun Zhu
Zhihao Fang

title: Towards the automatic anime characters creation with generative adversarial networks

venue: arXiv preprint arXiv:1708.05509,

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of GANs for improved quality, stability, and variation

authors:
Tero Karras
Samuli Laine
Timo Aila

title: A style-based generator architecture for generative adversarial networks. CoRR, abs/1812.04948, 2018b

venue: URL http://arxiv.org/abs/1812

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Naveen Kodali
Jacob Abernethy
James Hays
Zsolt Kira

title: On convergence and stability of gans

venue: arXiv preprint arXiv:1705.07215,

authors:
Ziwei Liu
Ping Luo
Xiaogang Wang
Xiaoou Tang

title: Deep learning face attributes in the wild

venue: In Proceedings of International Conference on Computer Vision (ICCV),

authors:
Soroush Mehri
Kundan Kumar
Ishaan Gulrajani
Rithesh Kumar
Shubham Jain
Jose Sotelo
Aaron Courville
Yoshua Bengio

title: SampleRNN: An unconditional end-to-end neural audio generation model

authors:
Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida

title: Spectral normalization for generative adversarial networks

authors:
Noam Mor
Lior Wolf
Adam Polyak
Yaniv Taigman

title: Autoencoder-based music translation

authors:
Augustus Odena
Christopher Olah
Jonathon Shlens

title: Conditional image synthesis with auxiliary classifier gans

venue: In ICML,

authors:
Alec Radford
Luke Metz
Soumith Chintala

title: Unsupervised representation learning with deep convolutional generative adversarial networks

authors:
Tim Salimans
Ian Goodfellow
Wojciech Zaremba
Vicki Cheung
Alec Radford
Xi Chen

title: Improved techniques for training gans

authors:
Tim Salimans
Andrej Karpathy
Xi Chen
Diederik P. Kingma

title: Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications

authors:
Jose Sotelo
Soroush Mehri
Kundan Kumar
Joao Felipe Santos
Kyle Kastner
Aaron Courville
Yoshua Bengio

title: Char2wav: End-to-end speech synthesis

authors:
Lucas Theis
Aäron van den Oord
Matthias Bethge

title: A note on the evaluation of generative models

authors:
Aäron van den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew W Senior
Koray Kavukcuoglu

title: Wavenet: A generative model for raw audio

venue: In SSW,

authors:
Yuxuan Wang
RJ Skerry-Ryan
Daisy Stanton
Yonghui Wu
Ron J Weiss
Navdeep Jaitly
Zongheng Yang
Ying Xiao
Zhifeng Chen
Samy Bengio

title: Tacotron: Towards end-to-end speech synthesis

authors:
Lior Wolf
Yaniv Taigman
Adam Polyak

title: Unsupervised creation of parameterized avatars

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A Efros

title: Unpaired image-to-image translation using cycle-consistent adversarial networks

authors:
Zhenyao Zhu
Jesse H Engel
Awni Y Hannun

title: Learning multiscale features directly fromwaveforms

venue: CoRR, vol

heading: 1 INTRODUCTION

text:
Neural audio synthesis, training generative models to efficiently produce audio with both highfidelity and global structure, is a challenging open problem as it requires modeling temporal scales over at least five orders of magnitude (∼0.1ms to ∼100s). Large advances in the state-of-the art have been pioneered almost exclusively by autoregressive models, such as WaveNet, which solve the scale problem by focusing on the finest scale possible (a single audio sample) and rely upon external conditioning signals for global structure (van den Oord et al., 2016). This comes at the cost of slow sampling speed, since they rely on inefficient ancestral sampling to generate waveforms one audio sample at a time. Due to their high quality, a lot of research has gone into speeding up generation, but the methods introduce significant overhead such as training a secondary student network or writing highly customized low-level kernels (van den Oord et al., 2018; Paine et al., 2016). Furthermore, since these large models operate at a fine timescale, their autoencoder variants are restricted to only modeling local latent structure due to memory constraints (Engel et al., 2017).
On the other end of the spectrum, Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have seen great recent success at generating high resolution images (Radford et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Berthelot et al., 2017; Kodali et al., 2017; Karras et al., 2018a; Miyato et al., 2018). Typical GANs achieve both efficient parallel sampling and global latent control by conditioning a stack of transposed convolutions on a latent vector, The potential for audio GANs extends further, as adversarial costs have unlocked intriguing domain transformations for images that could possibly have analogues in audio (Isola et al., 2017; Zhu et al., 2017; Wolf et al., 2017; Jin et al., 2017). However, attempts to adapt image GAN architectures to generate waveforms in a straightforward manner (Donahue et al., 2019) fail to reach the same level of perceptual fidelity as their image counterparts.
1Online resources: Colab Notebook: http://goo.gl/magenta/gansynth-demo, Audio Examples: http://goo.gl/magenta/gansynth-examples, Code: http://goo.gl/magenta/gansynth-code

heading: 1.1 GENERATING INSTRUMENT TIMBRES

text:
GAN researchers have made rapid progress in image modeling by evaluating models on focused datasets with limited degrees of freedom, and gradually stepping up to less constrained domains. For example, the popular CelebA dataset (Liu et al., 2015) is restricted to faces that have been centered and cropped, removing variance in posture and pose, and providing a common reference for qualitative improvements (Radford et al., 2016; Karras et al., 2018a) in generating realistic texture and fine-scale features. Later models then built on that foundation to generalize to broader domains (Karras et al., 2018b; Brock et al., 2019).
The NSynth dataset (Engel et al., 2017)2 was introduced with similar motivation for audio. Rather than containing all types of audio, NSynth consists solely of individual notes from musical instruments across a range of pitches, timbres, and volumes. Similar to CelebA, all the data is aligned and cropped to reduce variance and focus on fine-scale details, which in audio corresponds to timbre and fidelity. Further, each note is also accompanied by an array of attribute labels to enable exploring conditional generation.
The original NSynth paper introduced both autoregressive WaveNet autoencoders and bottleneck spectrogram autoencoders, but without the ability to unconditionally sample from a prior. Follow up work has explored diverse approaches including frame-based regression models (Defossez et al., 2018), inverse scattering networks (Andreux & Mallat, 2018), VAEs with perceptual priors (Esling et al., 2018), and adversarial regularization for domain transfer (Mor et al., 2019). This work builds on these efforts by introducing adversarial training and exploring effective representations for noncausal convolutional generation as typical found in GANs.
2https://magenta.tensorflow.org/datasets/nsynth

heading: 1.2 EFFECTIVE AUDIO REPRESENTATIONS FOR GANS

text:
Unlike images, most audio waveforms—such as speech and music—are highly periodic. Convolutional filters trained for different tasks on this data commonly learn to form logarithmically-scaled frequency selective filter banks spanning the range of human hearing (Dieleman & Schrauwen, 2014; Zhu et al., 2016). Human perception is also highly sensitive to discontinuities and irregularities in periodic waveforms, so maintaining the regularity of periodic signals over short to intermediate timescales (1ms - 100ms) is crucial. Figure 1 shows that when the stride of the frames does not exactly equal a waveform’s periodicity, the alignment (phase) of the two precesses over time. This condition is assured as at any time there are typically many different frequencies in a given signal. This is a challenge for a synthesis network, as it must learn all the appropriate frequency and phase combinations and activate them in just the right combination to produce a coherent waveform. This phase precession is exactly the same phenomena observed with a short-time Fourier transform (STFT), which is composed of strided filterbanks just like convolutional networks. Phase precession also occurs in situations where filterbanks overlap (window or kernel size < stride).
In the middle of Figure 1, we diagram another approach to generating coherent waveforms loosely inspired by the phase vocoder (Dolson, 1986). A pure tone produces a phase that precesses. Unwrapping the phase, by adding 2π whenever it crosses a phase discontinuity, causes the precessing phase to grow linearly. We then observe that the derivative of the unwrapped phase with respect to time remains constant and is equal to the angular difference between the frame stride and signal periodicity. This is commonly referred to as the instantaneous angular frequency, and is a time varying measure of the true signal oscillation. With a slight abuse of terminology we will simply refer to it as the instantaneous frequency (IF) (Boashash, 1992). Note that for the spectra at the bottom of Figure 1, the pure harmonic frequencies of a trumpet cause the wrapped phase spectra to oscillate at different rates while the unwrapped phase smoothly diverges and the IF spectra forms solid bands where the harmonic frequencies are present.

heading: 1.3 CONTRIBUTIONS

text:
In this paper, we investigate the interplay of architecture and representation in synthesizing coherent audio with GANs. Our key findings include:
• Generating log-magnitude spectrograms and phases directly with GANs can produce more coherent waveforms than directly generating waveforms with strided convolutions.
• Estimating IF spectra leads to more coherent audio still than estimating phase. • It is important to keep harmonics from overlapping. Both increasing the STFT frame size
and switching to mel frequency scale improve performance by creating more separation between the lower harmonic frequencies. Harmonic frequencies are multiples of the fundamental, so low pitches have tightly-spaced harmonics, which can cause blurring and overlap.
• On the NSynth dataset, GANs can outperform a strong WaveNet baseline in automatic and human evaluations, and generate examples ∼54,000 times faster.
• Global conditioning on latent and pitch vectors allow GANs to generate perceptually smooth interpolation in timbre, and consistent timbral identity across pitch.

heading: 2 EXPERIMENTAL DETAILS

heading: 2.1 DATASET

text:
We focus our study on the NSynth dataset, which contains 300,000 musical notes from 1,000 different instruments aligned and recorded in isolation. NSynth is a difficult dataset composed of highly diverse timbres and pitches, but it is also highly structured with labels for pitch, velocity, instrument, and acoustic qualities (Liu et al., 2015; Engel et al., 2017). Each sample is four seconds long, and sampled at 16kHz, giving 64,000 dimensions. As we wanted to included human evaluations on audio quality, we restricted ourselves to training on the subset of acoustic instruments and fundamental pitches ranging from MIDI 24-84 (∼32-1000Hz), as those timbres are most likely to sound natural
to an average listener. This left us with 70,379 examples from instruments that are mostly strings, brass, woodwinds, and mallets. We created a new test/train 80/20 split from shuffled data, as the original split was divided along instrument type, which isn’t desirable for this task.

heading: 2.2 ARCHITECTURE AND REPRESENTATIONS

text:
Taking inspiration from successes in image generation, we adapt the progressive training methods of Karras et al. (2018a) to instead generate audio spectra 3. While e search over a variety of hyperparameter configurations and learning rates, we direct readers to the original paper for an in-depth analysis (Karras et al., 2018a), and the appendix for complete details.
Briefly, the model samples a random vector z from a spherical Gaussian, and runs it through a stack of transposed convolutions to upsample and generate output data x = G(z), which is fed into a discriminator network of downsampling convolutions (whose architecture mirrors the generator’s) to estimate a divergence measure between the real and generated distributions (Arjovsky et al., 2017). As in Karras et al. (2018a), we use a gradient penalty (Gulrajani et al., 2017) to promote Lipschitz continuity, and pixel normalization at each layer. We also try training both progressive and nonprogressive variants, and see comparable quality in both. While it is not essential for success, we do see slightly better convergence time and sample diversity for progressive training, so for the remainder of the paper, all models are compared with progressive training.
Unlike Progressive GAN, our method involves conditioning on an additional source of information. Specifically, we append a one-hot representation of musical pitch to the latent vector, with the musically-desirable goal of achieving independent control of pitch and timbre. To encourage the generator to use the pitch information, we also add an auxiliary classification (Odena et al., 2017) loss to the discriminator that tries to predict the pitch label.
For spectral representations, we compute STFT magnitudes and phase angles using TensorFlow’s built-in implementation. We use an STFT with 256 stride and 1024 frame size, resulting in 75% frame overlap and 513 frequency bins. We trim the Nyquist frequency and pad in time to get an “image” of size (256, 512, 2). The two channel dimension correspond to magnitude and phase. We take the log of the magnitude to better constrain the range and then scale the magnitudes to be between -1 and 1 to match the tanh output nonlinearity of the generator network. The phase angle is also scaled to between -1 and 1 and we refer to these variants as “phase” models. We optionally unwrap the phase angle and take the finite difference as in Figure 1; we call the resulting models “instantaneous frequency” (“IF”) models. We also find performance is sensitive to having sufficient frequency resolution at the lower frequency range. Maintaining 75% overlap we are able to double the STFT frame size and stride, resulting in spectral images with size (128, 1024, 2), which we refer to as high frequency resolution, “+ H”, variants. Lastly, to provide even more separation of lower frequencies we transform both the log magnitudes and instantaneous frequencies to a mel frequency scale without dimensional compression (1024 bins), which we refer to as “IF-Mel” variants. To convert back to linear STFTs we just use the approximate inverse linear transformation, which, perhaps surprisingly does not harm audio quality significantly.
It is important for us to compare against strong baselines, so we adapt WaveGAN (Donahue et al., 2019), the current state of the art in waveform generation with GANs, to accept pitch conditioning and retrain it on our subset of the NSynth dataset. We also independently train our own waveform generating GANs off the progressive codebase and our best models achieve similar performance to WaveGAN without progressive training, so we opt to primarily show numbers from WaveGAN instead (see appendix Table 2 for more details).
Beyond GANs, WaveNet (van den Oord et al., 2016) is currently the state of the art in generative modeling of audio. Prior work on the NSynth dataset used an WaveNet autoencoder to interpolate between sounds (Engel et al., 2017), but is not a generative model as it requires conditioning on the original audio. Thus, we create strong WaveNet baselines by adapting the architecture to accept the same one-hot pitch conditioning signal as the GANs. We train variants using both a categorical 8-bit mu law and 16-bit mixture of logistics for the output distributions, but find that the 8-bit model is more stable and outperforms the 16-bit model (see appendix Table 2 for more details).
3Code: http://goo.gl/magenta/gansynth-code

heading: 3 METRICS

text:
Evaluating generative models is itself a difficult problem: because our goals (perceptually-realistic audio generation) are hard to formalize, the most common evaluation metrics tend to be heuristic and have “blind spots” (Theis et al., 2016). To mitigate this, we evaluate all of our models against a diverse set of metrics, each of which captures a distinct aspect of model performance. Our evaluation metrics are as follows:
• Human Evaluation We use human evaluators as our gold standard of audio quality because it is notoriously hard to measure in an automated manner. In the end, we are interested in training networks to synthesize coherent waveforms, specifically because human perception is extremely sensitive to phase irregularities and these irregularities are disruptive to a listener. We used Amazon Mechanical Turk to perform a comparison test on examples from all models presented in Table 1 (this includes the hold-out dataset). The participants were presented with two 4s examples corresponding to the same pitch. On a five-level Likert scale, the participants evaluate the statement ”Sample A has better audio quality / has less audio distortions than Sample B”. For the study, we collected 3600 ratings and each model is involved in 800 comparisons.
• Number of Statistically-Different Bins (NDB) We adopt the metric proposed by Richardson & Weiss (2018) to measure the diversity of generated examples: the training examples are clustered into k = 50 Voronoi cells by k-means in log-spectrogram space, the generated examples are also mapped into the same space and are assigned to the nearest cell. NDB is reported as the number of cells where the number of training examples is statistically significantly different from the number of generated examples by a two-sample Binomial test.
• Inception Score (IS) (Salimans et al., 2016) propose a metric for evaluating GANs which has become a de-facto standard in GAN literature (Gulrajani et al., 2017; Miyato et al., 2018; Karras et al., 2018a). Generated examples are run through a pretrained Inception classifier and the Inception Score is defined as the mean KL divergence between the imageconditional output class probabilities and the marginal distribution of the same. IS penalizes models whose examples aren’t each easily classified into a single class, as well as models whose examples collectively belong to only a few of the possible classes. Though we still call our metric “IS” for consistency, we replace the Inception features with features from a pitch classifier trained on spectrograms of our acoustic NSynth dataset.
• Pitch Accuracy (PA) and Pitch Entropy (PE) Because the Inception Score can conflate models which don’t produce distinct pitches and models which produce only a few pitches, we also separately measure the accuracy of the same pretrained pitch classifier on generated examples (PA) and the entropy of its output distribution (PE).
• Fréchet Inception Distance (FID) (Heusel et al., 2017) propose a metric for evaluating GANs based on the 2-Wasserstein (or Fréchet) distance between multivariate Gaussians fit to features extracted from a pretrained Inception classifier and show that this metric correlates with perceptual quality and diversity on synthetic distributions. As with Inception Score, we use pitch-classifier features instead of Inception features.

heading: 4 RESULTS

text:
Table 1 presents a summary of our results on all model and representation variants. Our most discerning measure of audio quality, human evaluation, shows a clear trend, summarized in Figure 2. Quality decreases as output representations move from IF-Mel, IF, Phase, to Waveform. The highest quality model, IF-Mel, was judged comparably but slightly inferior to real data. The WaveNet baseline produces high-fidelity sounds, but occasionally breaks down into feedback and self oscillation, resulting in a score that is comparable to the IF GANs.
While there is no a priori reason that sample diversity should correlate with audio quality, we indeed find that NDB follows the same trend as the human evaluation. Additionally, high frequency resolution improves the NDB score across models types. The WaveNet baseline receives the worst NDB score. Even though the generative model assigns high likelihood to all the training data, the
autoregressive sampling itself has a tendency gravitate to the same type of oscillation for each given pitch conditioning, leading to an extreme lack of diversity. Histograms of the sample distributions showing peaky distributions for the different models can be found in the appendix.
FID provides a similar story to the first two metrics with significantly lower scores for for IF models with high frequency resolution. Comparatively, Mel scaling has much less of an effect on the FID then it does in the listener study. Phase models have high FID, even at high frequency resolution, reflecting their poor sample quality.
Many of the models do quite well on the classifier metrics of IS, Pitch Accuracy, and Pitch Entropy, because they have explicit conditioning telling them what pitches to generate. All of the high-resolution models actually generate examples classified with similar accuracy to the real data. As this accuracy and entropy can be a strong function of the distribution of generated examples, which most certainly does not match the training distribution due to mode collapse and other issues, there is little discriminative information to gain about sample quality from differences among such high scores. The metrics do provide a rough measure of which models are less reliably generating classifiable pitches, which seems to be the low frequency models to some extent and the baselines.

heading: 5 QUALITATIVE ANALYSIS

text: While we do our best to visualize qualitative audio concepts, we highly recommend the reader to listen to the accompanying audio examples provided at https://goo.gl/magenta/ gansynth-examples.

heading: 5.1 PHASE COHERENCE

text: Figure 3 visualizes the phase coherence of examples from different GAN variants. It is clear from the waveforms at the top, which are wrapped at the fundamental frequency, that the real data and IF models produce waveforms that are consistent from cycle-to-cycle. The PhaseGAN has some phase discontinuities, while the WaveGAN is quite irregular. Below we use Rainbowgrams (Engel et al., 2017) to depict the log magnitude of the frequencies as brightness and the IF as the color on a rainbow color map. This visualization helps to see clear phase coherence of the harmonics in the real data and IFGAN by the strong consistent colors. In contrast, the PhaseGAN discontinuities appear as speckled noise, and the WaveGAN appears largely incoherent.

heading: 5.2 INTERPOLATION

text:
As discussed in the introduction, GANs also allow conditioning on the same latent vector the entire sequence, as opposed to only short subsequences for memory intensive autoregressive models like WaveNet. WaveNet autoencoders, such as ones in (Engel et al., 2017), learn local latent codes that control generation on the scale of milliseconds but have limited scope, and have a structure of their own that must be modelled and does not fit a compact prior. In Figure 4 we take a pretrained WaveNet autoencoder 5 and compare interpolating between examples in the raw waveform (top), the distributed latent code of a WaveNet autoencoder, and the global code of an IF-Mel GAN.
Interpolating the waveform is perceptually equivalent to mixing between the amplitudes of two distinct sounds. WaveNet improves upon this for the two notes by mixing in the space of timbre, but the linear interpolation does not correspond to the complex prior on latents, and the intermediate sounds have a tendency to fall apart, oscillate and whistle, which are the natural failure modes for a WaveNet model. However, the GAN model has a spherical gaussian prior which is decoded to
5https://github.com/tensorflow/magenta/tree/master/magenta/models/ nsynth
produce the entire sound, and spherical interpolation stays well-aligned with the prior. Thus, the perceptual change during interpolation is smooth and all intermediate latent vectors are decoded to produce sounds without additional artifacts. As a more musical example, in the audio examples, we interpolate the timbre between 15 random points in latent space while using the pitches from the prelude to Bach’s Suite No. 1 in G major 6. As seen in appendix Figure 7, the timbre of the sounds morph smoothly between many instruments while the pitches consistently follow the composed piece.

heading: 5.3 CONSISTENT TIMBRE ACROSS PITCH

text: While timbre slightly varies for a natural instrument across register, on the whole it remains consistent, giving the instrument its unique character. In the audio examples 7, we fix the latent conditioning variable and generate examples by varying the pitch conditioning over five octaves. It’s clear that the timbral identity of the GAN remains largely intact, creating a unique instrument identity for the given point in latent space. As seen in appendix Figure 7, the Bach prelude rendered with a single latent vector has a consistent harmonic structure across a range of pitches.

heading: 6 FAST GENERATION

text:
One of the advantages of GANs with upsampling convolutions over autoregressive models is that the both the training and generation can be processed in parallel for the entire audio sample. This
6http://www.jsbach.net/midi/midi_solo_cello.html 7https://goo.gl/magenta/gansynth-examples
is quite amenable to modern GPU hardware which is often I/O bound with iterative autoregressive algorithms. This can be seen when we synthesize a single four second audio sample on a TitanX GPU and the latency to completion drops from 1077.53 seconds for the WaveNet baseline to 20 milliseconds for the IF-Mel GAN making it around 53,880 times faster. Previous applications of WaveNet autoencoders trained on the NSynth dataset for music performance relied on prerendering all possible sounds for playback due to the long synthesis latency 8. This work opens up the intriguing possibility for realtime neural network audio synthesis on device, allowing users to explore a much broader pallete of expressive sounds.

heading: 7 RELATED WORK

text:
Much of the work on deep generative models for audio tends to focus on speech synthesis (van den Oord et al., 2018; Sotelo et al., 2017; Wang et al., 2017). These datasets require handling variable length conditioning (phonemes/text) and audio, and often rely on recurrent and/or autoregressive models for variable length inputs and outputs. It would be interesting to compare adversarial audio synthesis to these methods, but we leave this to future work as adapting GANs to use variable-length conditioning or recurrent generators is a non-trivial extension of the current work.
In comparison to speech, audio generation for music is relatively under-explored. van den Oord et al. (2016) and Mehri et al. (2017) propose autoregressive models and demonstrate their ability to synthesize musical instrument sounds, but these suffer from the aforementioned slow generation. Donahue et al. (2019) first applied GANs to audio generation with coherent results, but fell short of the audio fidelity of autoregressive likelihood models.
Our work also builds on multiple recent advances in GAN literature. Gulrajani et al. (2017) propose a modification to the loss function of GANs and demonstrate improved training stability and architectural robustness. Karras et al. (2018a) further introduce progressive training, in which successive layers of the generator and discriminator are learned in a curriculum, leading to improved generation quality given a limited training time. They also propose a number of architectural tricks to further improve quality, which we employ in our best models.
The NSynth dataset was first introduced as a “CelebA of audio” (Liu et al., 2015; Engel et al., 2017), and used WaveNet autoencoders to interpolate between timbres of musical instruments, but with very slow sampling speeds. Mor et al. (2019) expanded on this work by incoporating an adversarial domain confusion loss to achieve timbre transformations between a wide range of audio sources. Defossez et al. (2018) achieve significant sampling speedups (∼2,500x) over wavenet autoencoders by training a frame-based regression model to map from pitch and instrument labels to raw waveforms. They consider a unimodal likelihood regression loss in log spectrograms and backpropagate through the STFT, which yeilds good frequency estimation, but provides no incentive to learn phase coherency or handle multimodal distributions. Their architecture also requires a large amount of channels, slowing down sample generation and training.

heading: 8 CONCLUSION

text:
By carefully controlling the audio representation used for generative modeling, we have demonstrated high-quality audio generation with GANs on the NSynth dataset, exceeding the fidelity of a strong WaveNet baseline while generating samples tens of thousands of times faster. While this is a major advance for audio generation with GANs, this study focused on a specific controlled dataset, and further work is needed to validate and expand it to a broader class of signals including speech and other types of natural sound. This work also opens up possible avenues for domain transfer and other exciting applications of adversarial losses to audio. Issues of mode collapse and diversity common to GANs exist for audio as well, and we leave it to further work to consider combining adversarial losses with encoders or more straightforward regression losses to better capture the full data distribution.
8http://g.co/nsynthsuper

heading: ACKNOWLEDGMENTS

text: We would like to thank Rif A. Saurous and David Berthelot for fruitful discussions and help in reviewing the manuscript.

heading: A MEASURING DIVERSITY ACROSS GENERATED EXAMPLES

heading: B TIMBRAL SIMILARITY ACROSS PITCH

heading: C BASELINE MODEL COMPARISONS

heading: D TRAINING DETAILS

text:
GAN architectures were directly adapted from an open source implementation in Tensorflow 10. Full details are given in Table 3, including adding a pitch classifier to the end of the discriminator as in AC-GAN. All models were trained with the ADAM optimizer (Kingma & Ba, 2014). We sweep over learning rates (2e-4, 4e-4, 8e-4) and weights of the auxiliary classifier loss (0.1, 1.0, 10), and find that for all variants (spectral representation, progressive/no progressive, frequency resolution) a learning rate of 8e-4 and classifier loss of 10 perform the best.
As in the original progressive GAN paper, both networks use box upscaling/downscaling and the generators use pixel normalization,
x = xnhwc/( 1
C ∑ c x2nhwc) 0.5 (1)
where n, h, w, and c refer to the batch, height, width, and channel dimensions respectively, x is the activations, and C is the total number of channels. The discriminator also appends the standard deviation of the minibatch activations as a scalar channel near the end of the convolutional stack as seen in Table 3.
Since we find it helpful to use a Tanh output nonlinearity for the generator, we normalize real data before passing to the discriminator. We measure the maximum range over 100 examples and independently shift and scale the log-magnitudes and phases to [-0.8, 0.8] to allow for outliers and use more of the linear regime of the Tanh nonlinearity.
We train each GAN variant for 4.5 days on a single V100 GPU, with a batch size of 8. For nonprogressive models, this equates to training on ∼5M examples. For progressive models, we train on 1.6M examples per a stage (7 stages), 800k during alpha blending and 800k after blending. At the last stage we continue training until the 4.5 days completes. Because the earlier stages train faster, the progressive models train on ∼11M examples. For the WaveNet baseline, we also adapt the open source Tensorflow implementation 11. The decoder is composed of 30 layers of dilated convolution, each of 512 channels and receptive field of 3, and each with a 1x1 convolution skip connection to the output. The layers are divided into 3 stacks of 10, with dilation in each stack increasing from 20 to 29, and then repeating. We replace the audio encoder stack with a conditioning stack operating on a one-hot pitch conditioning signal distributed in time (3 seconds on, 1 second off). The conditioning stack is 5 layers of dilated convolution, increasing to 25, and then 3 layers of regular convolution, all with 512 channels. This conditioning signal is then passed through a 1x1 convolution for each layer of the decoder and added to the output of each layer, as in other implementations of WaveNet conditioning. For the 8-bit model we use mulaw encoding of the audio and a categorical loss, while for the 16-bit model we use a quantized mixture of 10 logistics (Salimans et al., 2017). WaveNets converged to 150k iterations in 2 days with 32 V100 GPUs trained with synchronous SGD with batch size 1 per GPU, for a total batch size of 32.
10https://github.com/tensorflow/models/tree/master/research/gan/ progressive_gan
11https://github.com/tensorflow/magenta/tree/master/magenta/models/ nsynth

title: GANSYNTH: ADVERSARIAL NEURAL AUDIO SYNTHESIS