The scattering framework offers an optimal hierarchical convolutional decomposition according to its kernels. Convolutional Neural Net (CNN) can be seen as an optimal kernel decomposition, nevertheless it requires large amount of training data to learn its kernels. We propose a trade-off between these two approaches: a Chirplet kernel as an efficient Q constant bioacoustic representation to pretrain CNN. First we motivate Chirplet bioinspired auditory representation. Second we give the first algorithm (and code) of a Fast Chirplet Transform (FCT). Third, we demonstrate the computation efficiency of FCT on large environmental data base: months of Orca recordings, and 1000 Birds species from the LifeClef challenge. Fourth, we validate FCT on the vowels subset of the Speech TIMIT dataset. The results show that FCT accelerates CNN when it pretrains low level layers: it reduces training duration by -28% for birds classification, and by -26% for vowels classification. Scores are also enhanced by FCT pretraining, with a relative gain of +7.8% of Mean Average Precision on birds, and +2.3% of vowel accuracy against raw audio CNN. We conclude on perspectives on tonotopic FCT deep machine listening, and inter-species bioacoustic transfer learning to generalise the representation of animal communication systems.
1 INTRODUCTION
Representation of bioacoustic sequences started with ’Human’ speech in the 70’. Speech automatic processing yields to the efficient Mel Filter Cepstral Coefficients (MFCC) representation. Today new bioacoustic representation paradigms arise from environmental monitoring and species classification at weak Signal to Noise Ratio (SNR) and with small amount of data per species.
Several neurobiological evidences suggest that auditory cortex is tuned to complex time varying acoustic features, and consists of several fields that decompose sounds in parallel (Kowalski et al., 1996; Mercado et al., 2000). Therefore it is more than reasonable to investigate the Chirplet timefrequency representation from acoustic and neurophysiological points of view.
Chirps, or transient amplitude and frequency modulated waveforms, are ubiquitous in nature systems (Flandrin (2001)), ranging from bird songs and music, to animal vocalization (frogs, whales) and Speech. Moreover the sinusoidal models are a typical attempt to represent audio signals as a superposition of chirp-like components. Chirp signals are also commonly observed in biosonar systems.
The Chirplet transform subsumes both Fourier analysis and wavelet analysis, providing a broad framework for mapping one-dimensional sound waveforms into a n-dimensional auditory parameter space. It offers the processing described in different auditory fields, i.e. cortical regions with systematically related response sensitivities. Moreover, Chirplet spaces are highly over-complete because there is an infinite number of ways to segment a time-frequency plane, the dictionary is redundant: this corresponds well with the overlapping, parallel signal processing pathways of auditory cortex.
Then we suggest that low level CNN layers shall be pretrained by Chirplet kernels. Thus, we define and code a Fast Chirplet Transform (FCT). We conduct validation on real recordings of whale and birds, and on Speech (vowels subset of TIMIT). We demonstrate that CNN classification benefits from low level layers FCT pretraining. We conclude on the perspectives of tonotopic FCT machine listening and inter-species transfer learning.
2 FORMAL DEFINITION OF CHIRPLET
A chirplet can be seen as a complex sinus with increasing or decreasing frequency over time modulated by a Gaussian window to have a localized support in the time and Fourier domain. It is a broad class of filters which includes wavelets and Fourier basis as special cases. As a result, and as presented in (Mann & Haykin, 1991; 1992), the Chirplet transform is a generalization of many known time-frequency representations. We first present briefly the wavelet transform framework to extend it to Chirplets. Given an input signal x one can compute a wavelet transform (Mallat, 1999) through the application of multiple wavelets ψλ. A wavelet is an atom with localized support in time and frequency domain which integrates to 0. The analytical support of the wavelets is not compact but they are very well localized. It can be considered compact in the applied case where roundoff error lead to 0 quickly after moving around the center frequency. The whole filter bank is derived from a mother wavelet ψ0 and a set of dilation coefficients following a geometric progression defined as Λ = {21+j/Q, j = 0, ..., JQ − 1} with J being the number of octave to decompose and Q the number of wavelets per octave. As a result, one can create the filter-bank as the collection {ψ0( tλ ) := ψλ, λ ∈ Λ}. After application of the filter-bank, one ends up with a time-scale representation, or scalogram, Ux(λ, t) := |(x ? ψλ)(t)| where the complex modulus was applied in order to remove the phase information and contract the space. It is clear that a wavelet filter-bank is completely characterized by its mother wavelet and the set of scale parameters. Generalizing this framework for Chirplets will be straightforward by now allowing a nonconstant frequency for each filter. As for wavelets, filters are generated from a Gaussian window determining the time support however the complex sinus has nonconstant frequency over time with center-frequency fc. Since the scope of the parameters leads infinitely many different possible filters, we have to restrain ourselves, and thus create only a fixed Chirplet filter-bank allowing fast computations. The parameters defining these filters include the time position tc, the frequency center fc, the duration ∆t and the chirp rate c:
gtc,fc,log(∆t),c(t) = 1√√ π∆t e − 12 (t−tc)2 ∆2t ej2π(c(t−tc) 2+fc(t−tc)). (1)
3 PROPOSITION OF A FAST CHIRPLET TRANSFORM (FCT)
The parameter space is basically of infinite dimension. Similarly to continuous wavelet transform however, it is possible to use some a priori knowledge in order to create a finite bank-filter. For example, wavelets are generated by knowing the number of wavelets per octave and the number of octave to decompose. As a result, we used the same motivation in order to reduce the number of possible Chirplets required. The goal here is not to compute an invertible transform, but rather provide a redundant transformation highlighting transient structures which are not the same tasks as discussed in (Coifman et al., 1992; Meyer, 1993; Coifman et al., 1994). As a result, we keep the same overall framework as for wavelets with the Q and J parameters. For example parameters for bird songs in this paper are J = 6 and Q = 16 with a sampling rate (SR) of 44100Hz, and J = 4 and Q = 16 on speech and Orca with SR=16 kHz). Finally, since we are interested in frequency modulations, we compute the ascendant and descendant chirp filters as one being the symetrized version of the other. As a result, we use a more straightforward analytical formula defined with a starting frequency F0, an ending frequency F1, and the usual wavelet like parameters σ being the
bandwidth. Finally the hyperparameter p defining the polynomial order of the chirp is constant for the whole bank-filter generation. For example, the case p = 1 leads to a linear chirp, p = 2 to a quadratic chirp. The starting and ending frequencies are chosen to approximately cover one octave and are directly computed from the λ parameters which define the scales. Finally, following the scattering network inspiration from (Bruna & Mallat, 2013), in order to remove unstable noisy pattern, we apply a low-pass filter (a Gaussian blurring) and thus we increase the SNR of the representation.
Λ = {2.01+i/Q, i = 0, ..., J ×Q− 1}, (2)
F0 = Fs
2λ , λ ∈ Λ, (3)
F1 = Fs
λ , λ ∈ Λ, (4)
σ = 2 d
λ , λ ∈ Λ. (5)
4 LOW COMPLEXITY FCT ALGORITHM AND IMPLEMENTATION
We give here our code of Fast Chirplet Transform (FCT), taking advantage of the a priori knowledge for the filter-bank creation and the fast convolution algorithm 1. Therefore, we first create the Chirplet with the ascendant and descendant versions in once (see Annexe Algo 1).
Then we generate the whole filter-bank (see Algo 2 in annexe) with the defined λ and hyperparameters.
Finally, we use the scattering framework (Bruna & Mallat, 2013; Andén & Mallat, 2014): we apply a local low-pass filter to the obtained representation. In fact, the scattering coefficients Sx result from a time-averaging on the time-frequency representation Ux bringing local and up to global time-invariance. This time-averaging is computed through the application of the φ filter, usually a Gabor atom with specified standard deviation and such that∫
φ(t)dt = 1. (6)
As a result, one computes these coefficients as: Sx(λ, t) = (|x ? ψλ| ? φ) (t), where ψλ is a Chirplet with λ parameters and φ. Similarly, we perform local time-averaging on the Chirplet representation in the same manner.
We present some possible filters in Fig. 2, and some bird features Fig. 3.
The third step in our FCT consists in the reduction of the convolution task. The asymptotic complexity of the Chirplet transform is O(N. log(N)) with N being the size of the input signal. This is the same asymptotic complexity as for the continuous wavelet transform and the scattering network. However, it is possible to reach lower asymptotic complexity simply by a division of the convolution task. usually the convolutions are carried through application of an element-wise multiplication of the signal and the filter in the frequency domain and then compute the inverse Fourier transform to end up with x ? ψλ. However, if we denote by M the length of the filter ψλ it is possible to instead perform multiple times this operation on different overlapping chunks of the signal to then concatenate the results to obtain at the end the same convolution result but now in O(N. log(M)). Finally a last improvement induced by this approach is to allow easy tackling of signals with a length just above a power of 2 which otherwise would require to be padded in order to obtain a FFT with real O(N. log(N)) complexity through the Danielson-Lanczos lemma (Press, 2007). Applying this scheme allowed to compute the convolutions between 3 to 4 times faster. The variations came from the distance between N and the closest next power of 2 depending on the desired chunk size.
We validate the efficiency of FCT on real bioacoustic recordings. We processed on 10 medium speed CPUs of 4 years old, 100 hours of recording of LifeClef bird challenge (16 kHz Sampling Rate (SR), 16 bits) in 2 days. Second, we processed in 7 days the equivalent of 1 month of
1We provide our implementation in Annexe and: https://github.com/DYNI-TOULON/ fastchirplet.git
Orca whale recordings from Orcalab.org ONG (22 kHz SR, 16 bits), in Fig. 1,2,3 and at http: //sabiod.univ-tln.fr/orcalab .
.
.
5 ENHANCING CNN BIOACOUSTIC REPRESENTATION WITH FCT
A strategy for CNN fine-tuning can be to retrain a classifier on top of a CNN on a new dataset, or to fine-tune the weights of a pretrained network by continuing the backpropagation. It is possible to
fine-tune all the layers of the CNN or to freeze some of the earlier, later or central layers, and to only fine-tune some portion of the network. As the features propagate deeper and deeper in the network layers, they become increasingly invariant and discriminative (Seltzer 2013). Thus usually only the higher level are fine-tuned, the earlier features of a CNN contain more generic features that should be useful to many tasks. As denoted in later layers of the CNN becomes progressively more specific to the details of the classes contained in the original dataset.
In this paper we adapt our parametric Chirplet decomposition to a specific acoustic domain with a specific CNN. We compare a CNN trained on raw audio to one trained on Mel and Chirplet. The best model is the one trained on parametric Chirplet. Second, we show that the CNN can be enhanced by pretraining Chirp in low level layer.
5.1 CNN BIRDS CLASSIFICATION ON FCT, RAW AUDIO, VERSUS MEL
The first demonstration is conducted on complex Bird songs. We use the BIRD10 subset of LifeClef 2016 bird classification challenge. It was used as ENS Ulm data challenge 2016, and contains 3 species in a total of 15 minutes of recordings (SR 44100 Hz, 16 bits), and is available (.wav, Mel and FCT features) at http://sabiod.univ-tln.fr/workspace/BIRD10.
We train 3 CNNs (LeCun & Bengio, 1995) on the Lasagne Theano platform. The baseline CNN is trained from the raw audio. A second CNN, with similar topology (see annexe) is trained on a simple log of the simple 64 channels Mel scale of FFT spectrum ( http://pydoc.net/Python/librosa/0.2.0/librosa.feature/ ). We overlap by 90% the time windows. A third CNN is trained on our FCT. The parameters of both CNN are similar, with 64 frequency bands each (we remove top and bottom band from the Chirplet to set to 64 bands only). Then the input layer is 64 x 86, the Conv layer has 20 filters of size 8 x 10. All activation functions are relu. We maxpool 2 x 2, follow the 20 filters of size 8 x 10, maxpooling, dense layer (200), dropout at 10%, with a final softmax dense layer with 3 classes and same dropout. Each CNN is trained by cross-entropy, L2 reg., with a learning rate set of 0.001.
The Fig. 4 gives the MAP of these two CNNs having similar hyperparameters. The CNN on FCT gives the best MAP with 61.5% at epoch 280 compared to later epoch (820) for Mel with a similar MAP of 61%. Audio is slower and weaker (58% MAP at epoch 1140).
5.2 ENHANCING BIRDS CLASSIFICATION STACKING PRETRAINED CHIRPNET CNN
In order to test the efficiency of the FCT, we pretrain a CNN to encode audio to Chirplets (a.k.a. the Audio2Chirp CNN) and a CNN to convert parametric Chirplet to classes (a.k.a. the Chirp2Class CNN). The topology of these CNNs (Tab. 2, 3) is set for reasonable time of training. We also speed up the training with shorter time overlap of the time windows (only 30% instead of 90% in the previous experimentation). We then decrease the average MAP, however the objective here is to compare the gain in MAP and time of convergence in stacked Chirplet deep representations.
We then simply stack at low level layer the audio2chirp with the chirp2class CNN to build a complete audio2class CNN. We train it from random initialization, or from pretrained CNN. Note that the random seed in all the experimentation of this paper is fixed to allow fair comparisons. Results are reported in Tab. 1 for each of the stacked CNN, with the epoch giving the best MAP on the dev. set, and the corresponding MAP on the test set. Results demonstrate that the pretraining of low level layers by FCT enhances CNN. More details are given in Annexe.
5.3 ENHANCING VOWELS CLASSIFICATION STACKING PRETRAINED CHIRPNET CNN
In this section we run the same demonstration on the subset of speech vowels of all the TIMIT acoustic-phonetic corpus JS et al. (1993): 3,696 training utterances (sampled at 16kHz) from 462 speakers. The cross-validation set consists of 400 utterances from 50 speakers. The core test set of the 8 vowels subset was used to report the results: 192 utterances from 24 speakers, excluding the validation set. There are 61 hand labeled phonetic symbols but the experiments in this paper run on the time windows of 310ms centered on each of the 8 vowels of TIMIT (= iy, ih, eh, ae, aa, ah, uh, uw).
Due to similar bioacoustic voicing dynamics of the two species (near 4 Hz), we simply set the FCT parameters for vowel to the one used for Orca presented above (p = 3,j = 4,q = 16,t = 0.001,s = 0.01). The time windows are set to 310 ms as recommended in Palaz et al. (2013).
The results of the different training stages of the audio2chirp and chirp2class and stacked model are given in Tab.2 and Annexe. We run due to lack of time the experiment only on vowel classification, which does not really allow comparison with other papers, however this seminal work only aims to study the relative gain between CNN pretrained or not by FCT.
The results demonstrate that FCT pretraining of the audio2class model is improved by 2.3% of relative gain of accuracy while the training time is decreased by 26%.
6 DISCUSSION AND CONCLUSION
In this paper we propose for the first time at our knowledge the definition and implementation of a Fast Chirplet Transform (FCT). Due to its low complexity, FCT can be computed as fast as FFT.
.
Second we show that FCT pretraining accelerates CNN. For Bird10 data set, we have 280 epochs using FCT versus 820 on Mel features, or 1140 on raw audio for same MAP score. The stacked CNN with the chirpnet in low level layer also decreases training from 530 epochs to 380 epochs, while
it increases MAP by 4 points (Tab. 1). The experiment on Vowels demonstrates a training of 30 epochs on FCT, versus 60 on raw audio (for same 65% accuracy level), and an increase of 1.5 point of accuracy (Tab. 2).
These gains may be due to the sparsity of the Chirplet, and the denoising step in the FCT. These experiences bring to light the problem of deep learning for small and biased dataset for which a full learning strategy is sub-optimal due to local optimum convergence. As a result, FCT prior knowledge can be used to mitigate this drawback by reducing the complexity of the deep-net architecture.
Three main perspectives are then opened. Future work will consist on sparse Chirpnet inspired from tonotopic net Strom (1997), auditory nerve and cortex topology Pironkov et al. (2015). The acoustic vibrations are transmitted to the base of the cochlea, thus each region of the basilar membrane are excited by different frequencies. The higher frequencies excite areas closer to the cochlea base, whereas lower frequencies are closer to the apex. This implies that neurons connected to a specific zone of the basilar membrane will be simultaneously stimulated inducing tonotopic representation.
A second perspective is to integrate Chirplet computation into the CNN training itself, as a constrained embedded layer, in a framework similar to a Wavelet Neural Network (Adeli & Jiang, 2006) but with Chirplet activation functions.
Last, we currently work on transfer learning of Chirpnet from animal to speech (and reverse), in order to generalize a deep Chirpnet representation of the animal communication systems.
7 ACKNOWLEDGEMENTS
We thank colleagues from ENS Paris Data Team with S. Mallat, and P. Flandrin, for fruitful discussions on Scattering and Chirplet. We thank YLC and YB for advises on CNN. We thank V. Tassan for cleaning the code. We used Theano, Lasagne 2, Librosa 3 and Pysoundfile 4.
A BIRD DATASET
The experiment is conducted on BIRD10, an online data set http://sabiod.univ-tln.fr/ workspace/BIRD10/ which is a subset of the training LIFEClef 2016 challenge on bird classification. BIRD10 contains 454 audio files (22050 Hz SR, 16 bits) from 10 bird classes, split in 0.5s segments. 20% of the training set was used as the validation set.
Only segments with detected bird activity were kept, assuming a bird sound to have prominent energy and to be mostly harmonic. This bird detection is for a given segment:
if (energy_ratio > energy_threshold and spectral_flatness_weighted_mean < spectral_flatness_threshold) bird_detected = True
else bird_detected = False
where the energy and the spectral flatness are computed on 50% overlapping frames of 256 samples:
er = energy_ratio = mean(seg_energy)
95thpercentile(file_energy)
swf = spectral_flatness_weighted_mean = sum(seg_spectral_flatness× seg_energy)
sum(seg_energy) This naive algorithm performed quite well on a manually labelled dataset of bird vocalizations (precision=0.89, recall=0.57 for er=0.2 and sfw=0.3) after a quick grid search on the two parameters.
B BIRDS CLASSIFICATION : BASELINE CNNS
The first experiment consisted in running similar CNNs to compare the performance of using raw audio and two time-frequency representations as the input: a standard log-amplitude Mel spectrum and the Chirplet representation described in the first part of this paper. In this experiments the segments were overlapping by 90%. The topologies of the networks are given in Tab. 2. The cost function is the cross-entropy, learning rate = 0.0001 = L2 regularisation coefficient. The Mel spectrum is computed from 64 bands between 0 and 11025 Hz (=SR/2). Both Mel spectrum and Chirplets were normalized by Z-score.
In all experiments, a given topology is always initialized using the same set of random parameters, unless specified otherwise. The value * (resp. 0) after the name of the net refers to the pretrained net (resp. random initialization).
C BIRD AUDIO2CHIRP - CHIRPLET ENCODER
The chirp encoder, aka audio2chirp, aims at training a net to get a Chirplet-like representation. It is a simple CNN taking audio as input, Chirplets as output and minimizing the square error. It converges easily in 180 epochs. The topology of the audio2chirp net is given Tab. 4.
D BIRD: TRAINING, DEV AND TESTING CURVES OF THE DIFFERENT CNNS
E EXPERIMENT ON SPEECH VOWEL
The Tab. 5 gives the topology of the audio2chirp and chirp2class, and stacked models, for these vowel experiments.
In all experiments, each CNN is initialized using the same random seed. The symbol “*” refers to the optimal trained parameters of a net.
F ALGORITHM FOR THE FAST CHIRPLET TRANSFORM (FCT)
Algo 1: Chirplet Generation INPUT: F0,F1,Fs,sigma,p OUTPUT: coefficients_upward,coefficients_downward if(p):
w=cos(2*pi*((F1-F0)/((p+1)*sigma**p)*t**p+F0)*t) else:
w=cos(2*pi*((F0*(F1/F0)**(t/sigma)-F0)*sigma/log(F1/F0))) coefficients_upward=w*exp(-((t-\sigma/2.0)**2)/(2*sigma**2)) coefficients_downward=flipud(coefficients_upward).
Algo 2: Chirplet Filter-Bank Generation INPUT: J, Q, Fs, sigma, p lambdas = 2.0**(1+arrange(J*Q)/float(Q)) start_frequencies = (Fs /lambdas)/2.0 end_frequencies = Fs /lambdas distances = 2.0*d/flipud(lambdas) filters=list() for f0,f1,d in zip(start_frequencies,end_frequencies,distances):
filters.append(chirplet(Fs,f0,f1,d,p)) return filters.
G THE PYTHON CODE FOR THE FAST CHIRPLET TRANSFORM (FCT)
This code, in GPL licence (c) DYNI team, is in Github : https://github.com/DYNI-TOULON/fastchirplet.git.
import l i b r o s a import os import numpy as np from p y l a b import ∗ import s y s from numpy . l i b import pad
c l a s s C h i r p l e t :
" " " s m a l l e s t t i m e b i n among t h e c h i r p l e t " " " g l o b a l s m a l l e s t _ t i m e _ b i n s
def _ _ i n i t _ _ ( s e l f , s a m p l e r a t e , F0 , F1 , sigma , po lynome_degree ) :
" " " l o w e s t f r e q u e n c y where t h e c h i r p l e t i s a p p l i e d " " " s e l f . m in_ f r equency = F0
" " " h i g h e s t f r e q u e n c y where t h e c h i r p l e t i s a p p l i e d " " " s e l f . max_f requency = F1
" " " s a m p l e r a t e o f t h e s i g n a l " " " s e l f . s a m p l e r a t e = s a m p l e r a t e
" " " d u r a t i o n o f t h e c h i r p l e t " " " s e l f . t i m e _ b i n = sigma / 1 0
" " " de gr e e o f t h e polynome t o g e n e r a t e t h e c o e f f i c i e n t s o f t h e c h i r p l e t " " " s e l f . po lynome_degree = polynome_degree
" " " c o e f f i c i e n t s a p p l i e d t o t h e s i g n a l " " " s e l f . f i l t e r _ c o e f f i c i e n t s = s e l f .
c a l c u l _ c o e f f i c i e n t s ( )
def c a l c u l _ c o e f f i c i e n t s ( s e l f ) : " " " c a l c u l a t e c o e f f i c i e n t s f o r t h e c h i r p l e t s " " " # p r i n t ( s e l f . _ s a m p l e r a t e ) t = l i n s p a c e ( 0 , s e l f . t ime_b in , i n t ( s e l f . s a m p l e r a t e ∗
s e l f . t i m e _ b i n ) ) i f ( s e l f . po lynome_degree ) :
w = cos (2∗ p i ∗ ( ( s e l f . max_frequency−s e l f . m in_ f r equency )
/ ( ( s e l f . po lynome_degree +1) ∗ s e l f . t i m e _ b i n ∗∗ s e l f . po lynome_degree ) ∗ t ∗∗ s e l f . po lynome_degree + s e l f . m in_ f r equency ) ∗ t )
e l s e : w = cos (2∗ p i ∗ ( ( s e l f . m in_ f r equency ∗ ( s e l f .
max_f requency / s e l f . m in_ f r equency )
∗∗ ( t / s e l f . t i m e _ b i n )− s e l f . m in_ f r equency ) ∗ s e l f . t i m e _ b i n
/ l o g ( s e l f . max_f requency / s e l f . m in_ f r equency ) ) )
c o e f f s = w∗ hann ing ( l e n ( t ) ) ∗∗2
re turn c o e f f s
def smooth_up ( s e l f , i n p u t _ s i g n a l , sigma , end_smooth ing ) : # g e n e r a t e f a s t f o u r i e r t r a n s f o r m from a s i g n a l and smooth
i t
new_up = b u i l d _ f f t ( i n p u t _ s i g n a l , s e l f . f i l t e r _ c o e f f i c i e n t s , s igma ) re turn f f t _ s m o o t h i n g ( f a b s ( new_up ) , end_smooth ing )
def compute ( i n p u t _ s i g n a l , s ave = F a l s e , d u r a t i o n _ l a s t _ c h i r p l e t = 1 . 0 1 , num_octaves =5 , n u m _ c h i r p s _ b y _ o c t a v e =10 , po lynome_degree =3 ,
end_smooth ing = 0 . 0 0 1 ) : " " " main f u n c t i o n . Fas t C h i r p l e t Trans form from a s i g n a l " " "
da ta , s a m p l e r a t e = l i b r o s a . l o a d ( i n p u t _ s i g n a l , s r =None )
s i z e _ d a t a = l e n ( d a t a )
n e a r e s t _ p o w e r _ 2 = 2∗∗ ( s i z e _ d a t a −1) . b i t _ l e n g t h ( )
d a t a = np . l i b . pad ( da t a , ( 0 , n e a r e s t _ p o w e r _ 2−s i z e _ d a t a ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
c h i r p l e t s = i n i t _ c h i r p l e t _ f i l t e r _ b a n k ( s a m p l e r a t e , d u r a t i o n _ l a s t _ c h i r p l e t , num_octaves , num_ch i rps_by_oc tave , po lynome_degree )
c h i r p s = a p p l y _ f i l t e r b a n k ( da t a , c h i r p l e t s , end_smooth ing )
c h i r p s = r e s i z e _ c h i r p s ( s i z e _ d a t a , n e a r e s t _ p o w e r _ 2 , c h i r p s )
i f s ave : i f not os . p a t h . e x i s t s ( " csv " ) :
os . m a k e d i r s ( " csv " ) np . s a v e t x t ( " csv / "+os . p a t h . basename ( i n p u t _ s i g n a l ) .
s p l i t ( ’ . ’ ) [ 0 ] + ’ . c sv ’ , c h i r p s , d e l i m i t e r =" , " )
re turn c h i r p s
def r e s i z e _ c h i r p s ( s i z e _ d a t a , n e a r e s t _ p o w e r _ 2 , c h i r p s ) : s i z e _ c h i r p s = l e n ( c h i r p s ) r a t i o = s i z e _ d a t a / n e a r e s t _ p o w e r _ 2 s i z e = i n t ( r a t i o ∗ l e n ( c h i r p s [ 0 ] ) )
t a b f i n a l = np . z e r o s ( ( s i z e _ c h i r p s , s i z e ) ) f o r i in range ( 0 , s i z e _ c h i r p s ) : t a b f i n a l [ i ]= c h i r p s [ i ] [ 0 : s i z e ] re turn t a b f i n a l
def i n i t _ c h i r p l e t _ f i l t e r _ b a n k ( s a m p l e r a t e , d u r a t i o n _ l a s t _ c h i r p l e t , num_octaves , num_ch i rps_by_oc tave , p ) :
" " " g e n e r a t e a l l t h e c h i r p l e t s from a g i v e n sample r a t e " " "
lambdas = 2 . 0∗∗ ( 1 + a r a n g e ( num_octaves ∗ n u m _ c h i r p s _ b y _ o c t a v e ) / f l o a t ( n u m _ c h i r p s _ b y _ o c t a v e ) ) #Low f r e q u e n c i e s f o r a s i g n a l s t a r t _ f r e q u e n c i e s = ( s a m p l e r a t e / lambdas ) / 2 . 0 # h igh f r e q u e n c i e s f o r a s i g n a l e n d _ f r e q u e n c i e s = s a m p l e r a t e / lambdas d u r a t i o n s = 2 . 0∗ d u r a t i o n _ l a s t _ c h i r p l e t / f l i p u d ( lambdas ) C h i r p l e t . s m a l l e s t _ t i m e _ b i n s = d u r a t i o n s [ 0 ] c h i r p l e t s = l i s t ( ) f o r f0 , f1 , d u r a t i o n in z i p ( s t a r t _ f r e q u e n c i e s ,
e n d _ f r e q u e n c i e s , d u r a t i o n s ) : c h i r p l e t s . append ( C h i r p l e t ( s a m p l e r a t e , f0 , f1 ,
d u r a t i o n , p ) ) re turn c h i r p l e t s
def a p p l y _ f i l t e r b a n k ( i n p u t _ s i g n a l , c h i r p l e t s , end_smooth ing ) : " " " g e n e r a t e l i s t o f s i g n a l w i t h c h i r p l e t s " " " r e s u l t = l i s t ( ) f o r c h i r p l e t in c h i r p l e t s :
r e s u l t . append ( c h i r p l e t . smooth_up ( i n p u t _ s i g n a l , 6 , end_smooth ing ) )
re turn a r r a y ( r e s u l t )
def f f t _ s m o o t h i n g ( i n p u t _ s i g n a l , s igma ) : " " " smooth t h e f a s t t r a n s f o r m f o u r i e r " " " s i z e _ s i g n a l = i n p u t _ s i g n a l . s i z e # s h o r t e n t h e s i g n a l new_s ize = i n t ( f l o o r ( 1 0 . 0∗ s i z e _ s i g n a l ∗ s igma ) ) h a l f _ n e w _ s i z e = new_s ize / / 2
f f t x = f f t ( i n p u t _ s i g n a l ) s h o r t _ f f t x = [ ] f o r e l e in f f t x [ : h a l f _ n e w _ s i z e ] : s h o r t _ f f t x . append ( e l e ) f o r e l e in f f t x [− h a l f _ n e w _ s i z e : ] :
s h o r t _ f f t x . append ( e l e )
a p o d i z a t i o n _ c o e f f i c i e n t s = g e n e r a t e _ a p o d i z a t i o n _ c o e f f i c i e n t s ( h a l f _ n e w _ s i z e , sigma , s i z e _ s i g n a l ) # a p p l y t h e a p o d i z a t i o n c o e f f i c i e n t s s h o r t _ f f t x [ : h a l f _ n e w _ s i z e ] ∗= a p o d i z a t i o n _ c o e f f i c i e n t s # a p p l y t h e a p o d i z a t i o n c o e f f i c i e n t s i n a r e v e r s e l i s t s h o r t _ f f t x [ h a l f _ n e w _ s i z e : ] ∗= f l i p u d ( a p o d i z a t i o n _ c o e f f i c i e n t s ) r e a l i f f t x w = r e a l ( i f f t ( s h o r t _ f f t x ) ) re turn r e a l i f f t x w
def g e n e r a t e _ a p o d i z a t i o n _ c o e f f i c i e n t s ( num_coeffs , sigma , s i z e ) : " " " g e n e r a t e a p o d i z a t i o n c o e f f i c i e n t s " " " a p o d i z a t i o n _ c o e f f i c i e n t s = a r a n g e ( num_coef f s ) a p o d i z a t i o n _ c o e f f i c i e n t s = a p o d i z a t i o n _ c o e f f i c i e n t s ∗∗2
a p o d i z a t i o n _ c o e f f i c i e n t s = a p o d i z a t i o n _ c o e f f i c i e n t s / ( 2 ∗ ( s igma ∗ s i z e ) ∗∗2) a p o d i z a t i o n _ c o e f f i c i e n t s = exp(− a p o d i z a t i o n _ c o e f f i c i e n t s ) re turn a p o d i z a t i o n _ c o e f f i c i e n t s
def f f t _ b a s e d ( i n p u t _ s i g n a l , h , boundary =0) : M=h . s i z e h a l f _ s i z e = M/ / 2 i f ( boundary ==0) : #ZERO PADDING
i n p u t _ s i g n a l =pad ( i n p u t _ s i g n a l , ( h a l f _ s i z e , h a l f _ s i z e ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0) h=pad ( h , ( 0 , i n p u t _ s i g n a l . s i z e−M) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0) newx= i f f t ( f f t ( i n p u t _ s i g n a l ) ∗ f f t ( h ) ) re turn newx [M−1:−1]
e l i f ( boundary ==1) : # s y m m e t r i c i n p u t _ s i g n a l = c o n c a t e n a t e ( [ f l i p u d ( i n p u t _ s i g n a l [ :
h a l f _ s i z e ] ) , i n p u t _ s i g n a l , f l i p u d ( i n p u t _ s i g n a l [ h a l f _ s i z e : ] ) ] )
h=pad ( h , ( 0 , x . s i z e−M) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0) newx= i f f t ( f f t ( i n p u t _ s i g n a l ) ∗ f f t ( h ) ) re turn newx [M−1:−1]
e l s e : # p e r i d i c re turn r e a l ( r o l l ( i f f t ( f f t ( i n p u t _ s i g n a l ) ∗ f f t ( h ,
i n p u t _ s i g n a l . s i z e ) ) ,− h a l f _ s i z e ) )
def b u i l d _ f f t ( i n p u t _ s i g n a l , f i l t e r _ c o e f f i c i e n t s , n =2 , boundary =0) : " " " g e n e r a t e f a s t t r a n s f o r m f o u r i e r by windows " " "
M= f i l t e r _ c o e f f i c i e n t s . s i z e # p r i n t ( n , boundary ,M) h a l f _ s i z e = M/ / 2 s i g n a l _ s i z e = i n p u t _ s i g n a l . s i z e # power o f 2 t o a p p l y f a s t f o u r i e r t r a n s f o r m windows_s ize = i n t (2∗∗ c e i l ( l og2 (M∗ ( n +1) ) ) ) number_of_windows= f l o o r ( s i g n a l _ s i z e / / windows_s ize ) i f ( number_of_windows ==0) :
re turn f f t _ b a s e d ( i n p u t _ s i g n a l , f i l t e r _ c o e f f i c i e n t s , boundary )
o u t p u t = e m p t y _ l i k e ( i n p u t _ s i g n a l ) #pad w i t h 0 t o have a s i z e i n a power o f 2 z e r o p a d d i n g = pad ( f i l t e r _ c o e f f i c i e n t s , ( 0 , windows_s ize−M) , ’
c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
h _ f f t = f f t ( z e r o p a d d i n g )
# t o browse t h e whole s i g n a l c u r r e n t _ p o s =0
# a p p l y f f t t o a p a r t o f t h e s i g n a l . T h i s p a r t has a s i z e which i s a power o f 2 i f ( boundary ==0) : #ZERO PADDING #window i s h a l f padded w i t h s i n c e i t ’ s f o c u s e d on
t h e f i r s t h a l f window = i n p u t _ s i g n a l [ c u r r e n t _ p o s : c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e ] zeropaddedwindow = pad ( window , ( l e n ( h _ f f t )−l e n (
window ) , 0 ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
x _ f f t = f f t ( zeropaddedwindow ) e l i f ( boundary ==1) : #SYMMETRIC
window = c o n c a t e n a t e ( [ f l i p u d ( i n p u t _ s i g n a l [ : h a l f _ s i z e ] ) , i n p u t _ s i g n a l [ c u r r e n t _ p o s : c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e ] ] )
x _ f f t = f f t ( window ) e l s e :
x _ f f t = f f t ( i n p u t _ s i g n a l [ : windows_s ize ] )
o u t p u t [ : windows_s ize−M]= i f f t ( x _ f f t ∗ h _ f f t ) [M−1:−1] c u r r e n t _ p o s += windows_s ize−M−h a l f _ s i z e
# a p p l y f a s t f o u r i e r t r a n s o f m t o each windows whi le ( c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e <= s i g n a l _ s i z e ) :
x _ f f t = f f t ( i n p u t _ s i g n a l [ c u r r e n t _ p o s−h a l f _ s i z e : c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e ] ) o u t p u t [ c u r r e n t _ p o s : c u r r e n t _ p o s + windows_s ize−M]= r e a l ( i f f t ( x _ f f t ∗ h _ f f t ) [M−1:−1])
c u r r e n t _ p o s += windows_s ize−M
# a p p l y f a s t f o u r i e r t r a n s f o r m t o t h e r e s t o f t h e s i g n a l i f ( windows_s ize −( s i g n a l _ s i z e −c u r r e n t _ p o s + h a l f _ s i z e ) <
h a l f _ s i z e ) : window = i n p u t _ s i g n a l [ c u r r e n t _ p o s−h a l f _ s i z e : ] zeropaddedwindow = pad ( window , ( 0 , i n t ( windows_s ize −( s i g n a l _ s i z e −c u r r e n t _ p o s + h a l f _ s i z e ) ) ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
x _ f f t = f f t ( zeropaddedwindow ) o u t p u t [ c u r r e n t _ p o s : ] = r e a l ( r o l l ( i f f t ( x _ f f t ∗ h _ f f t ) ,
h a l f _ s i z e ) [ h a l f _ s i z e : h a l f _ s i z e + o u t p u t . s i z e− c u r r e n t _ p o s ] )
o u t p u t [− h a l f _ s i z e : ] = c o n v o l v e ( i n p u t _ s i g n a l [−M: ] , f i l t e r _ c o e f f i c i e n t s , ’ same ’ ) [− h a l f _ s i z e : ]
e l s e : window = i n p u t _ s i g n a l [ c u r r e n t _ p o s−h a l f _ s i z e : ] zeropaddedwindow = pad ( window , ( 0 , i n t ( windows_s ize −( s i g n a l _ s i z e −c u r r e n t _ p o s + h a l f _ s i z e ) ) ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
x _ f f t = f f t ( zeropaddedwindow ) o u t p u t [ c u r r e n t _ p o s : ] = r e a l ( i f f t ( x _ f f t ∗ h _ f f t ) [M−1:M+
o u t p u t . s i z e−c u r r e n t _ p o s −1]) re turn o u t p u t

The authors advocate use of chirplets as a basis for modeling audio signals.  They introduce a fast chiplet transform for efficient computation. Also introduced is the idea of initializing (pre-training) CNN layers to mimic chirplet transform of audio signal (similar to ideas proposed by Mallet et al. on scattering transforms).  The paper is fairly easy to follow but in a few places contains undefined terms (e.g. AM-FM, MAP).
While the idea of using chirplet transform is interesting, my main concern is that the empirical evidence provided is in a rather narrow domain of bird call classification.  Furthermore, the accuracy gains shown in that domain are relatively small (61% MAP for log-Mel features vs 61.5% for chirplet transforms).  I would recommend that authors provide evidence for how this generalizes to other audio (including speech) tasks.


We demonstrate improved text-to-image synthesis with controllable object locations using an extension of Pixel Convolutional Neural Networks (PixelCNN). In addition to conditioning on text, we show how the model can generate images conditioned on part keypoints and segmentation masks. The character-level text encoder and image generation network are jointly trained end-to-end via maximum likelihood. We establish quantitative baselines in terms of text and structureconditional pixel log-likelihood for three data sets: Caltech-UCSD Birds (CUB), MPII Human Pose (MHP), and Common Objects in Context (MS-COCO).
1 INTRODUCTION
a man wearing snow gear poses for a photo while standing on skis
A person on snow skis with a backpack skiing down a mountain.
A white body and head with a bright orange bill along with black coverts and rectrices. A young girl is wearing a black ballerina outfit and pink tights dancing.
person person person
Figure 1: Examples of interpretable and controllable image synthesis. Left: MS-COCO, middle: CUB, right: MHP. Bottom row shows segmentation and keypoint conditioning information.
Image generation has improved dramatically over the last few years. The state-of-the-art images generated by neural networks in 2010, e.g. (Ranzato et al., 2010) were noted for their global structure and sharp boundaries, but were still easily distinguishable from natural images. Although we are far from generating photo-realistic images, the recently proposed image generation models using modern deep networks (van den Oord et al., 2016c; Reed et al., 2016a; Wang & Gupta, 2016; Dinh et al., 2016; Nguyen et al., 2016) can produce higher-quality samples, at times mistakable for real.
Three image generation approaches are dominating the field: generative adversarial networks (Goodfellow et al., 2014; Radford et al., 2015; Chen et al., 2016), variational autoencoders (Kingma & Welling, 2014; Rezende et al., 2014; Gregor et al., 2015) and autoregressive models (Larochelle & Murray, 2011; Theis & Bethge, 2015; van den Oord et al., 2016b;c). Each of these approaches have significant pros and cons, and each remains an important research frontier.
Realistic high-resolution image generation will impact media and communication profoundly. It will also likely lead to new insights and advances in artificial intelligence. Understanding how to control the process of composing new images is at the core of this endeavour.
Researchers have shown that it is possible to control and improve image generation by conditioning on image properties, such as pose, zoom, hue, saturation, brightness and shape (Dosovitskiy et al., 2015; Kulkarni et al., 2015), part of the image (van den Oord et al., 2016b; Pathak et al., 2016), surface normal maps (Wang & Gupta, 2016), and class labels (Mirza & Osindero, 2014; van den Oord et al., 2016c). It is also possible to manipulate images directly using editing tools and learned generative adversarial network (GAN) image models (Zhu et al., 2016).
Language, because of its compositional and combinatorial power, offers an effective way of controlling the generation process. Many recent works study the image to text problem, but only a handful have explored text to image synthesis. Mansimov et al. (2015) applied an extension of the DRAW model of Gregor et al. (2015), followed by a Laplacian Pyramid adversarial network post-processing step (Denton et al., 2015), to generate 32× 32 images using the Microsoft COCO dataset (Lin et al., 2014). They demonstrated that by conditioning on captions while varying a single word in the caption, we can study the effectiveness of the model in generalizing to captions not encountered in the training set. For example, one can replace the word “yellow” with “green” in the caption “A yellow school bus parked in a parking lot” to generate blurry images of green school buses.
Reed et al. (2016a), building on earlier work (Reed et al., 2016b), showed that GANs conditioned on captions and image spatial constraints, such as human joint locations and bird part locations, enabled them to control the process of generating images. In particular, by controlling bounding boxes and key-points, they were able to demonstrate stretching, translation and shrinking of birds. Their results with images of people were less successful. Yan et al. (2016) developed a layered variational autoencoder conditioned on a variety of pre-specified attributes that could generate face images subject to those attribute constraints.
In this paper we propose a gated conditional PixelCNN model (van den Oord et al., 2016c) for generating images from captions and other structure. Pushing this research frontier is important for several reasons. First, it allows us to assess whether auto-regressive models are able to match the GAN results of Reed et al. (2016a). Indeed, this paper will show that our approach with autoregressive models improves the image samples of people when conditioning on joint locations and captions, and can also condition on segmentation masks. Compared to GANs, training the proposed model is simpler and more stable because it does not require minimax optimization of two deep networks. Moreover, with this approach we can compute the likelihoods of the learned models. Likelihoods offer us a principled and objective measure for assessing the performance of different generative models, and quantifying progress in the field.
Second, by conditioning on segmentations and captions from the Microsoft COCO dataset we demonstrate how to generate more interpretable images from captions. The segmentation masks enable us to visually inspect how well the model is able to generate the parts corresponding to each segment in the image. As in (Reed et al., 2016a), we study compositional image generation on the Caltech-UCSD Birds dataset by conditioning on captions and key-points. In particular, we show that it is possible to control image generation by varying the key-points and by modifying some of the keywords in the caption, and observe the correct change in the sampled images.
2 MODEL
2.1 BACKGROUND: AUTOREGRESSIVE IMAGE MODELING WITH PIXELCNN
Figure 2 illustrates autoregressive density modeling via masked convolutions, here simplified to the 1D case. At training time, the convolutional network is given the sequence x1:T as both its input and target. The goal is to learn a density model of the form:
p(x1:T ) = T∏ t=1 p(xt|x1:t−1) (1)
To ensure that the model is causal, that is that the prediction x̂t does not depend on xτ for τ ≥ t, while at the same time ensuring that the training is just as efficient as the training of standard convolutional networks, van den Oord
et al. (2016c) introduce masked convolutions. Figure 2 shows, in blue, the active weights of 5 × 1 convolutional filters after multiplying them by masks. The filters connecting the input layer to the first hidden layer are in this case multiplied by the mask m = (1, 1, 0, 0, 0). Filters in subsequent layers are multiplied by m = (1, 1, 1, 0, 0) without compromising causality. 1.
1Obviously, this could be done in the 1D case by shifting the input, as in van den Oord et al. (2016a)
In our simple 1D example, if xt is discrete, say xt ∈ {0, . . . , 255}, we obtain a classification problem, where the conditional density p(xt|x1:t−1) is learned by minimizing the cross-entropy loss. The depth of the network and size of the convolutional filters determine the receptive field. For example, in Figure 2, the receptive field for x̂t is xt−6:t−1. In some cases, we may wish to expand the size of the receptive fields by using dilated convolutions (van den Oord et al., 2016a).
van den Oord et al. (2016c) apply masked convolutions to generate colour images. For the input to first hidden layer, the mask is chosen so that only pixels above and to the left of the current pixel can influence its prediction (van den Oord et al., 2016c). For colour images, the masks also ensure that the three color channels are generated by successive conditioning: blue given red and green, green given red, and red given only the pixels above and to the left, of all channels.
The conditional PixelCNN model (Fig. 3) has several convolutional layers, with skip connections so that outputs of each layer layer feed into the penultimate layer before the pixel logits. The input image is first passed through a causal convolutional layer and duplicated into two activation maps, v and h. These activation maps have the same width and height as the original image, say N ×N , but a depth of f instead of 3, as the layer applies f filters to the input. van den Oord et al. (2016c) introduce two stacks of convolutions, vertical and horizontal, to ensure that the predictor of the current pixel has access to all the pixels in rows above; i.e. blind spots are eliminated.
In the vertical stack, a masked N ×N convolution is efficiently implemented with a 1×N convolution with f filters followed by a masked N × 1 convolution with 2f filters. The output activation maps are then sent to the vertical and horizontal stacks. When sending them to the horizontal stack, we must shift the activation maps, by padding with zeros at the bottom and cropping the top row, to ensure that there is no dependency on pixels to the right of the pixel being predicted. Continuing on the vertical stack, we add the result of convolving 2f convolutional filters. Note that since the vertical stack is connected to the horizontal stack and hence the ouput via a vertical shift operator, it can afford to look at all pixels in the current row of the pixel being predicted. Finally, the 2f activation maps are split into two activations maps of depth f each and passed through a gating tanh-sigmoid nonlinearity (van den Oord et al., 2016c).
The shifted activation maps passed to the horizontal stack are convolved with masked 1×1 convolutions and added to the activation maps produced by applying a masked 1×N horizontal convolution to the current input row. As in the vertical stack, we apply gated tanh-sigmoid nonlinearities before sending the output to the pixel predictor via skip-connections. The horizontal stack also uses residual connections (He et al., 2016). Finally, outputs v′ and h′ become the inputs to the next layer.
As shown in Figure 3, the version of model used in this paper also integrates global conditioning information, text and segmentations in this example.
2.2 CONDITIONING ON TEXT AND SPATIAL STRUCTURE
To encode location structure in images we arrange the conditioning information into a spatial feature map. For MS-COCO this is already provided by the 80-category class segmentation. For CUB and MHP we convert the list of keypoint coordinates into a binary spatial grid. For both segmentation and the keypoints in spatial format, the first processing layer is a class embedding lookup table, or equivalently a 1× 1 convolution applied to a 1-hot encoding. Text a gray elephant standing next to a woman in a red dress. Structure (H x W class labels) Convolutional encoding
Sequential encoding
(GRU)
class embedding lookup table
To PixelCNN Dilated conv.
To PixelCNN
The text is first encoded by a character-CNN-GRU as in (Reed et al., 2016a). The averaged embedding (over time dimension) of the top layer is then tiled spatially and concatenated with the location pathway. This concatenation is followed by several layers of dilated convolution. These allow information from all regions at multiple scales in the keypoint or segmentation map to be processed along with the text embedding, while keeping the spatial dimension fixed to the image size, using a much smaller number of layers and parameters compared to using non-dilated convolutions.
3 EXPERIMENTS
We trained our model on three image data sets annotated with text and spatial structure.
• The MPII Human Pose dataset (MHP) has around 25K images of humans performing 410 different activities (Andriluka et al., 2014). Each person has up to 17 keypoints. We used the 3 captions per image collected by (Reed et al., 2016a) along with body keypoint annotations. We kept only the images depicting a single person, and cropped the image centered around the person, leaving us 18K images.
• The Caltech-UCSD Birds database (CUB) has 11,788 images in 200 species, with 10 captions per image (Wah et al., 2011). Each bird has up to 15 keypoints.
• MS-COCO (Lin et al., 2014) contains 80K training images annotated with both 5 captions per image and segmentations. There are 80 classes in total. For this work we used class segmentations rather than instance segmentations for simplicity of the model.
Keypoint annotations for CUB and MHP were converted to a spatial format of the same resolution as the image (e.g. 32 × 32), with a number of channels equal to the maximum number of visible keypoints. A “1“ in row i, column j, channel k indicates the visibility of part k in entry (i, j) of the image, and “0” indicates that the part is not visible. Instance segmentation masks were re-sized to match the image prior to feeding into the network.
We trained the model on 32 × 32 images. The PixelCNN module used 10 layers with 128 feature maps. The text encoder reads character-level input, applying a GRU encoder and average pooling after three convolution layers. Unlike in Reed et al. (2016a), the text encoder is trained end-toend from scratch for conditional image modeling. We used RMSprop with a learning rate schedule starting at 1e-4 and decaying to 1e-5, trained for 200k steps with batch size of 128.
In the following sections we demonstrate image generation results conditioned on text and both part keypoints and segmentation masks. Note that some captions in the data contain typos, e.g. “bird is read” instead of “bird is red”, and were not introduced by the authors.
3.1 TEXT- AND SEGMENTATION-CONDITIONAL IMAGE SYNTHESIS
In this section we present results for MS-COCO, with a model conditioned on the class of object visible in each pixel. We also included a channel for background. Figure 5 shows several conditional samples and the associated annotation masks. The rows below each sample were generated by pointwise multiplying each active channel of the ground-truth segmentation mask by the sampled image. Here we defined “active” as occupying more than 1% of the image.
Each group of four samples uses the same caption and segmentation mask, but the random seed is allowed to vary. The samples tend to be very diverse yet still match the text and structure constraints. Much larger examples are included in the appendix.
TV
This laptop and monitor are surrounded by many wires.
category [1] => [person]
A couple of people standing in a field playing with a frisbee.
The woman is riding her horse on the beach by the water.
category [7] => [train]
a red white an blue train next to a train station loading area
A piece of cooked broccoli is on some cheese.
A bathroom with a vanity mirror next to a white toilet. A young man riding a skateboard down a ramp.
a man sits at a desk and uses a laptop on it
category [1] => [person]
category [73] => [laptop]
a man riding a wave on a surfboard in the ocean
category [1] => [person]
category [56] => [broccoli]
category [32] => [tie]
Three men wearing black and ties stand and smile at something.
A person carrying their surfboard while walking along a beach.
category [42] => [surfboard]
a man smiles down from the back of an elephant
category [1] => [person]
category [22] => [elephant]
category [1] => [person]
category [35] => [skis]
a person on snow skis with a backpack skiing down a mountain
Two women in english riding outfits on top of horses.
category [1] => [person]
category [19] => [horse]
A large cow walks over a fox in the grass.
category [18] => [dog]
category [21] => [cow]
Laptop
Person
Horse
Broccoli
Pizza
Person
Surfboard
Toilet
Sink
Person
Skateboard
Person
Tie
Dog
Cow
Person
Horse
Figure 5: Text- and segmentation-conditional general image samples.
We observed that the model learns to adhere correctly to location constraints; i.e. the sampled images all respected the segmentation boundaries. The model can assign the right color to objects based on the class as well; e.g. green broccoli, white and red pizza, green grass. However, some foreground objects such as human faces appear noisy, and in general we find that object color constraints are not captured as accurately by the model as location constraints.
5
3.2 TEXT- AND KEYPOINT-CONDITIONAL IMAGE SYNTHESIS
In this section we show results on CUB and MHP, using bird and human part annotations. Figure 6 shows the results of six different queries, with four samples each. Within each block of four samples, the text and keypoints are held fixed. The keypoints are projected to 2D for visualization purposes, but note that they are presented to the model as a 32 × 32 × K tensor, where K is the number of keypoints (17 for MHP and 15 for CUB).
We observe that the model consistently associates keypoints to the apparent body part in the generated image; see “beak” and “tail” labels drawn onto the samples according to the ground-truth location. In this sense the samples are interpretable; we know what the model was meant to depict at salient locations. Also, we observe a large amount of diversity in the background scenes of each query, while pose remains fixed and the bird appearance consistently matches the text.
Keypoints
This bird has wings that are grey and has a yellow belly.
The bird has a white colored head, breast, throat and abdomen, as well as a grey colored covert.Key-
points
Changing the random seed (within each block of four samples), the background details change significantly. In some cases this results in unlikely situations, like a black bird sitting in the sky with wings folded. However, typically the background remains consistent with the bird’s pose, e.g. including a branch for the bird to stand on.
Figure 7 shows the same protocol applied to human images in the MHP dataset. This setting is probably the most difficult, because the training set size is much smaller than MS-COCO, but the variety of poses and settings is much greater than in CUB. In most cases we see that the generated person m tches the keypoints well, and the setting is consistent with the caption, e.g. in a pool, outdoors or on a bike. However, producing the right color of specific parts, or generating objects associated to a person (e.g. bike) remain a challenge.
A man in a blue hat is holding a shovel in a dirt filled field.
We found it useful to adjust the temperature of the softmax during sampling. The probability of drawing value k for a pixel with probabilities p is pTk / ∑ i p T i , where T is the inverse temperature.
Higher values for T makes the distribution more peaked. In practice we observed that larger T resulted in less noisy samples. We used T = 1.05 by default.
Ideally, the model should be able to render any combination of of valid keypoints and text description of a bird. This would indicate that the model has “disentangled” location and appearance, and has not just memorized the (caption, keypoint) pairs seen during training. To tease apart the influence of keypoints and text in CUB, in Figure 8 we show the results of both holding the keypoints fixed while varying simple captions and fixing the captions while varying the keypoints.
This bird is yellow and orange.
This bird is blue and yellow.
To limit variation across captions due to background differences, we re-used the same random seed derived from each pixel’s batch, row, column and color coordinates2. This causes the first few generated pixels in the upper-left of the image to be very similar across a batch (down columns in Figure 8), resulting in similar backgrounds.
In each column, we observe that the pose of the generated birds satisfies the constraints imposed by the keypoints, and the color changes to match the text. This demonstrates that we can effectively control the pose of the generated birds via the input keypoints, and its color via the captions simultaneously. We also observe a significant diversity of appearance.
However, some colors work better than others, e.g. the “bright yellow” bird matches its caption, but “completely green” and “all white” are less accurate. For example the birds that were supposed to be white are shown with dark wings in several cases. This suggests the model has partially disentangled location and appearance as described in the text, but still not perfectly. One possible explanation is that keypoints are predictive of the category of bird, which is predictive of the appearance (including color) of birds in the dataset.
3.3 QUANTITATIVE RESULTS
Table 1 shows quantitative results in terms of the negative log-likelihood of image pixels conditioned on both text and structure, for all three datasets. For MS-COCO, the test negative log-likelihood is not included because the test set does not provide captions.
2Implemented by calling np.random.seed((batch, row, col, color)) before sampling.
The quantitative results show that the model does not overfit, suggesting that in future research a useful direction may be to develop higher-capacity models that are still memory- and computationallyefficient to train.
3.4 COMPARISON TO PREVIOUS WORKS
Figure 9 compares to MHP results from Reed et al. (2016a). In comparison to the approach advanced in this paper, the samples produced by the Generative Adversarial What-Where Networks are significantly less diverse. Close inspection of the GAN image samples reveals many wavy artifacts, in spite of the conditioning on body-part keypoints. As the bottom row shows, these artifacts can be extreme in some cases.
This work (T=1.05)
4 DISCUSSION
In this paper, we proposed a new extension of PixelCNN that can accommodate both unstructured text and spatially-structured constraints for image synthesis. Our proposed model and the recent Generative Adversarial What-Where Networks both can condition on text and keypoints for image synthesis. However, these two approaches have complementary strengths. Given enough data GANs can quickly learn to generate high-resolution and sharp samples, and are fast enough at inference time for use in interactive applications (Zhu et al., 2016). Our model, since it is an extension of the autoregressive PixelCNN, can directly learn via maximum likelihood. It is very simple, fast and robust to train, and provides principled and meaningful progress benchmarks in terms of likelihood.
We advanced the idea of conditioning on segmentations to improve both control and interpretability of the image samples. A possible direction for future work is to learn generative models of segmentation masks to guide subsequent image sampling. Finally, our results have demonstrated the ability of our model to perform controlled combinatorial image generation via manipulation of the input text and spatial constraints.

"First, it allows us to assess whether auto-regressive models are able to match the GAN results of Reed et al. (2016a)." Does it, though? Because the resolution is so bad. And resolution limitations aren't addressed until the second-to-last paragraph of the paper. And Figure 9 only shows 3 results (picked randomly? Picked to be favorable to PixelCNN?). That hardly seems conclusive.
The segmentation masks and keypoints are pretty strong input constraints. It's hard to tell how much coherent object and scene detail is emerging because the resolution is so low. For example, the cows in figure 5 look like color blobs, basically. Any color blob that follows an exact cow segmentation mask will look cow-like.
The amount of variation is impressive, though.
How can we assess how much the model is "replaying" training data? Figure 8 tries to get at this, but I wonder how much each of the "red birds", for instance, is mostly copied from a particular training example.
I'm unsatisfied with the answers to the pre-review questions. You didn't answer my questions. The paper would benefit from concrete numbers on training time / epochs and testing time. You didn't say why you can't make high resolution comparisons. Yes, it's slower at test time. Is it prohibitively slow? Or is it slightly inconvenient? There really aren't that many comparisons in the paper, anyway, so it if takes an hour to generate a result that doesn't seem prohibitive.
To be clear about my biases: I don't think PixelCNN is "the right way" to do deep image generation. Texture synthesis methods used these causal neighborhoods with some success, but only because there wasn't a clear way to do the optimization more globally (Kwatra et al, Texture Optimization for Example-based Synthesis being one of the first alternatives). It seems simply incorrect to make hard decisions about what pixel values should be in one part of the image before synthesizing another part of the image (Another texture synthesis strategy to help fight back against this strict causality was coarse-to-fine synthesis. And I do see some deep image synthesis methods exploring that). It seems much more correct to have a deeper network and let all output pixels be conditioned on all other pixels (this conditioning will implicitly emerge at intermediate parts of the network). That said, I could be totally wrong, and the advantages stated in the paper could outweigh the disadvantages. But this paper doesn't feel very honest about the disadvantages.
Overall, I think the results are somewhat tantalizing, especially the ability to generate diverse outputs. But the resolution is extremely low, especially compared to the richness of the inputs. The network gets a lot of hand holding from rich inputs (it does at least learn to obey them).
The deep image synthesis literature is moving very quickly. The field needs to move on from "proof of concept" papers (the first to show a particular result is possible) to more thorough comparisons. This paper has an opportunity to be a more in depth comparison, but it's not very deep in that regard. There isn't really an apples to apples comparison between PixelCNN and GAN nor is there a conclusion statement about why that is impossible. There isn't any large scale comparison, either qualitative or quantified by user studies, about the quality of the results.


We consider the learning of algorithmic tasks by mere observation of input-output pairs. Rather than studying this as a black-box discrete regression problem with no assumption whatsoever on the input-output mapping, we concentrate on tasks that are amenable to the principle of divide and conquer, and study what are its implications in terms of learning. This principle creates a powerful inductive bias that we exploit with neural architectures that are defined recursively, by learning two scale-invariant atomic operators: how to split a given input into two disjoint sets, and how to merge two partially solved tasks into a larger partial solution. The scale invariance creates parameter sharing across all stages of the architecture, and the dynamic design creates architectures whose complexity can be tuned in a differentiable manner. As a result, our model is trained by backpropagation not only to minimize the errors at the output, but also to do so as efficiently as possible, by enforcing shallower computation graphs. Moreover, thanks to the scale invariance, the model can be trained only with only input/output pairs, removing the need to know oracle intermediate split and merge decisions. As it turns out, accuracy and complexity are not independent qualities, and we verify empirically that when the learnt complexity matches the underlying complexity of the task, this results in higher accuracy and better generalization in two paradigmatic problems: sorting and finding planar convex hulls.
1 INTRODUCTION
Many algorithmic tasks can be described as discrete input-output mappings, but this “black-box” vision hides all the fundamental questions that explain how and why the task can be optimally solved, which is the starting point of the study of algorithms and complexity. A powerful and general framework that breaks into this vision is the principle that many tasks have some degree of scale invariance or self-similarity, meaning that the ability to solve the task for a certain input size is essentially all that is needed in order to solve it for larger sizes. This principle is the basis of dynamic programming and is ubiquitous in most areas of discrete mathematics, from geometry to graph theory. In the case of images and audio signals, invariance principles are also critical for success: CNNs exploit both translation invariance and scale separation with multilayer, localized convolutional operators, which breaks the curse of dimensionality and brings the essential inductive bias explaining the success of CNNs. In our scenario of discrete algorithmic tasks, we build our model on the principle of divide and conquer, which provides us with a form of parameter sharing across scales akin to that of CNNs across space or RNNs across time.
While neural networks have been successful so far at providing flexible models for discrete regression and prediction tasks, mostly in Natural Language processing and discrete Reinforcement Learning, they are typically unaware and uninterested in complexity questions. Whereas some models are trained and tested at a fixed input/output scale (such as regression problems with generic fully connected neural networks), authors have explored ways to make training and testing less dependent of the input scale. The most prominent examples are Convolutional architectures, that ∗Currently on leave from UC Berkeley
exploit translation invariance to accept variable size inputs by averaging their predictions at the last layer; and recurrent neural networks, that operate in an auto-regressive fashion to summarize any variable sized-input into a fixed-dimensional embedding. These two examples are paradigms of models whose complexity scales linearly with the input size.
Whereas CNN and RNN models define algorithms with linear complexity, attention mechanisms Bahdanau et al. (2014) generally correspond to quadratic complexity, with notable exceptions Andrychowicz & Kurach (2016). This can result in a mismatch between the intrinsic complexity required to solve a given task and the complexity that is given to the neural network to solve it. Our motivation is that learning cannot be ‘complete’ until these complexities match, and we start this quest by first focusing on problems for which the intrinsic complexity is well known and understood.
In this paper, we attempt to incorporate the complexity as yet another quantity that one wishes to minimize while training a model. We achieve this by using an architecture that learns recursively how to split a given input and learns how to merge each of the partial responses into a final output. Although these two steps could – and should – eventually be combined, in this work we start by exploring each of these architectures separately. We do so by only observing input-output pairs, getting away with the need to provide each of our artificial smaller problems with their correct output. This is a form of ‘weak’ supervision that is shown to work on tasks that are scale invariant, i.e. that can be addressed by divide and conquer. Another side benefit of our dynamic programming networks is their ability to generalize to larger scales. By construction, our model learns the same decision at each scale, and therefore can generalize well whenever the task is compatible with that inductive bias.
Summary of Contributions:
• We introduce a recursive split and merge architecture, and a learning framework that optimizes it not only for accuracy but also for computational complexity in a fully differentiable manner, using only input-output example pairs.
• We provide preliminary empirical evidence that the dynamic programming principle can be efficiently learnt on simple tasks such as sorting and planar convex hull.
2 RELATED WORK
Using neural networks to solve algorithmic tasks is an active area of current research, but its models can be traced back to context free grammars Fanty (1994). In particular, dynamic learning appears in works such as Pollack (1991) and Tabor (2000).
The current research in the area is dominated by Recurrent Neural Networks Joulin & Mikolov (2015); Grefenstette et al. (2015), LSTMs Hochreiter & Schmidhuber (1997), sequence-to-sequence neural models Sutskever et al. (2014); Zaremba & Sutskever (2014), attention mechanisms Vinyals et al. (2015b); Andrychowicz & Kurach (2016) and explicit external memory models Weston et al. (2014); Sukhbaatar et al. (2015); Graves et al. (2014); Zaremba & Sutskever (2015). We refer the reader to Joulin & Mikolov (2015) and references therein for a more exhaustive and detailed account of related work.
Amongst these works, we highlight some that are particularly relevant to us. Neural GPU Kaiser & Sutskever (2015) defines a neural architecture that acts convolutionally with respect to the input and is applied iteratively o(n) times, where n is the input size. It leads to fixed computational machines with total o(n2) complexity. Neural Programmer-Interpreters Reed & de Freitas (2015) introduce a compositional model based on a LSTM that can learn generic programs. It is trained with full supervision using execution traces. Hierarchical attention mechanisms have been explored in Andrychowicz & Kurach (2016). They improve the complexity of the model from o(n2) of traditional attention to o(n log n), similarly as our models, but they are trained very differently, using REINFORCE. Finally, Pointer Networks Vinyals et al. (2015b;a) modify classic attention mechanisms to make them amenable to adapt to variable input-dependent outputs, and illustrate the resulting models on geometric algorithmic tasks. It belongs to the o(n2) category class.
3 DIVIDE AND CONQUER WITH NEURAL NETWORKS
In this section we present our basic model architecture with its core Split and Merge blocks, and then describe how to build the global dynamic programming network. In our formulation, we choose generic data structures for split and merge. The split module accepts sets of elements as input, and outputs a disjoint partition of this set. The merge module accepts two ordered sets Ω1 and Ω2 as input, and produces a subset Λ ⊆ Ω1 ∪ Ω2 of their union that respects the partial ordering; i.e, if x1 Ω1 x2, x1, x2 ∈ Ω1 ∩ Λ, then necessarily x1 Λ x2.
3.1 SPLIT
Split blocks receive an input set of elements Ω, and output a partition Ω = Ω1 ∪ Ω2. The corresponding neural network architecture is permutation invariant. We use a simple block that computes nonlinear moments of the input. Denote n = |Ω| and assume the elements of Ω to be in Rd. Define the matrix x ∈ Rd×n as the elements in Ω organized in columns under an arbitrary ordering and xi as the i-th column. This architecture must take sets as inputs, i.e, (S(x, θ))σ = S(xσ, θ) ∀σ ∈ Sn (where σ acts in rows). We propose a Θ(n) architecture in Appendix A, a somewhat simpler version of similar set-to-set models such as those in Vinyals et al. (2015a); Sukhbaatar et al. (2016). We can compute both subsets either by sampling from the output probabilities or taking the mode (1[pi ≥ 0.5]).
3.2 MERGE
Merge blocks receive two ordered inputs and produce a subset of their union that preserves the partial ordering. We can visualize the merge block as having two different tasks:
• Choose a mask over each input to rule out some elements. • Merge both subsets preserving the partial ordering.
In this paper, for simplicity, we only learn the first task of the merge block and leave the learnability of the second for future work. Let y1 and y2 be two ordered sequences of elements in Rd of lengths n1 and n2 respectively. We parametrize the first part of the merge block using Bi-directional LSTMs. The input of the merge block will be a sequence of vectors in Rd+1. The first d dimensions of the elements in the sequence will correspond to the concatenation of y1 and y2. The last dimension of each element in the input sequence will be a boolean indicating to which sequence the element belongs. We produce the element-wise probabilities by concatenating the hidden states at every element and computing pi = σ(wT [h1, h2] + b) for i = 1, . . . , n1 and pi = σ(wT [h2, h1] + b) for i = n1 + 1, . . . , n2 where b is a scalar bias and w is a vector in R2nh with nh being the number of hidden units of each LSTM. We can also compute both subsets either by sampling from these probabilities or taking the mode (1[pi ≥ 0.5]). To merge the chosen subsets we will use a deterministic procedure which will produce the correct output if both subsets are correct and will be stable for small errors in the mask. This procedure will be specific for each task and we explain our choice in the experiments section when addressing the convex hull task.
3.3 BUILDING THE MODEL
Figure 1 illustrates our architecture. The divide and conquer principle is implemented by successively splitting inputs until the input x0 of size |x0| = n is broken into a collection of sets {xv}v∈Vi , i ≤ l such that they reach a critical size K0. At that critical scale we consider that the task at hand can be solved with a generic mapping with constant complexity to produce the outputs {xv}v∈Vl . We consider problems where the outputs are defined as partially ordered subsets of the input. The second phase consists in merging this collection of outputs into the global solution. As described earlier, in this work we consider the two phases separately, and leave the joint training of split and merge operations for future work.
The model has parameters for each split and merge, but these parameters are shared across all the instances. Also, the structure of the binary tree is dynamic: each input determines the respective sizes of the split, which in turn determine the length of each corresponding branch of the tree.
4 LEARNING
The training objective is to both maximize the accuracy at the output, but also to be able to do so with minimal complexity. This section describes how our dynamic architecture allows to optimize both objectives with gradient descent.
4.1 ADJUSTING COMPLEXITY WITH GRADIENT DESCENT
The average case complexity of the split phase satisfies the following recursion:
ECs(n) = E{Cs(αsn) + Cs((1− αs)n)}+A · n , (1)
where (αs, 1 − αs) are the fraction of input elements that are respectively sent to each output and A · n is the cost of running our split module described in Section 3.1. Since this fraction is inputdependent, the average case is obtained by taking expectations with respect to the underlying input distribution. Assuming without loss of generality that E(αs) ≥ 0.5, the resulting complexity is of the order of
ECs(n) ' An log n
logEα−1s . (2)
The overall complexity of the resulting model can be thus partially controlled by enforcing αs to be as close as possible to 0.5. The range αs ∈ [0.5, 1) gives us the ability to span complexities between Θ(n log2 n) (perfectly balanced trees) and Θ(n
2) (perfectly unbalanced trees with n − 1 and 1 elements).
Similarly, our merge selection phase performs a selection of a subset of the union of its two inputs. Let n = |Ω1| + |Ω2| be the total amount of incoming data at any given node and αm = |Λ|n denote the fraction of elements that is sent to the next level. If we assume an execution tree given by a split phase with factor αs, we verify that the resulting complexity satisfies
CM (n) = Bnα − logn logαs m + CM (nαs) + CM (n(1− αs)) . (3)
When Eαm < 1, by applying the Akra-Bazzi method Akra & Bazzi (1998) this results in
ECM (n) = Θ(n) , (4)
and ECM (n) = Θ(n log n) when Eαm = 1. The merge block thus operates in linear time as soon as each merge operation removes a constant fraction of its inputs at each scale. We shall impose that Eαm ≥ Eαs to ensure that the merge decisions do not reach the target size before the tree has reached its end. This creates an incentive to make decisions to discard elements early in the process, but as we shall see in the next section, this is offset by the fact that labeled information exists only at the bottom of the merge process. In the next subsections we shall see how to control the branching factors in a differentiable fashion, enabling learning by simple gradient descent.
4.2 WEAK VERSUS STRONG SUPERVISION
A key requirement of our approach is the ability to learn only from input-output pairs of training examples. Since the examples have arbitrary size, in general we have no supervision for each individual instance of split and merge operations. The challenge is that each of these blocks sends information to the next by making discrete sampling decisions.
One possibility is to embark in optimization strategies that differentiate under sampling, such as those arising in Reinforcement Learning and in particular the REINFORCE. However, these optimization methods involve gradient estimates with large variance, resulting in poor sample complexity. We study an alternative that exploits the powerful inductive bias given by sharing parameters at all scales and also the fact that we optimize for both accuracy and complexity, and requires only backpropagation and no intermediate sampling.
We first focus on the training of each split and merge phase separately, and we assume first that we have available labels at the output of each of these phases. That is, we assume we have a dataset {(xk, yk)}k≤K of K examples. In the case of the split phase, we assume that xk is a set of nk (possibly varying with k) elements, and yk is an ordered partition of xk. In the case of the merge, xk represents an ordered partition of a certain set zk, and yk is a certain (ordered) subset of zk. Each of these subproblems already contains the challenge of training several instances of split and merge connected through discrete, non-differentiable sampling operations. We shall now describe in detail our strategy to compute gradients with respect to parameters θ (split) and φ (merge) just with backpropagation.
4.3 TRAINING SPLIT
In order to train the split block we must create artificial targets at every node of the generated tree from the available final target partition, which is an ordered sequence of disjoint subsets of the input.
Figure 1 shows that every scale of the generated tree corresponds to a partition of the final target, and every node of the tree at that scale corresponds to one of the sets of the partition, whose size equals the length of the input at that node.
For simplicity, we drop here the superscript denoting the training instance. Let us denote by Ωv = xv ∩ yv the intersection of the input xv at node v, and the corresponding target subset yv . The elements in that intersection will provide gradient signal to update θ for each node as follows. The split block at v with current parameters computes the vector of probabilities pv(θ,x), encoding the probability that each element in xv will be sent towards the first output or the second. In order to create targets for these outputs, we first sample from pv(θ,x) to obtain a partition of xv: xv = T 1∪T 2. This partition in turn defines a ‘valid’ partition Ωv = Ωinp,1v ∪Ωinp,2v , with Ωinp,jv = Ωv∩T j for j = 1, 2.
Similarly, we consider the target partition of the same size defined by the order in the target subset Ωv = Ω targ,1 v ∪ Ωtarg,2v where |Ωtarg,iv | = |Ωinp,iv |, i = 1, 2. This partition creates the targets
t = 1(x ∈ Ωtarg,1v ) , x ∈ Ωv .
As described in Section 4.1, besides providing targets corresponding to correct solutions, we also attempt to minimize the average complexity of the model. The contribution of node v to the loss is thus
l(θ, xv, yv) = − ∑
x∈Ωtarg,1v
log(pv(θ, x))− ∑
x∈Ωtarg,2v
log(1− pv(θ, x))− βSR(v, θ) , (5)
where
R(v, θ) = 1
|Ωv| (∑ x∈Ωv pv(θ, x) 2 ) − ( 1 |Ωv| ∑ x∈Ωv pv(θ, x) )2 (6)
is a regularization term similar to an empirical variance which will encourage the split block to partition the input into equal parts when maximized. Increasing βS will give more preference to split inputs into equal parts, but using a large value can make the performance go down.
We can finally define the total loss by aggregating the losses across all the nodes of the tree:
LS(θ, η, x, y) = l−1∑ i=0 ηi 1 |Vi| ∑ v∈Vi l(θ, xv, yv) , (7)
where Vi is the set of vertices at depth i, and η = (η1, . . . , ηl) is a vector of dynamic hyperparameters that can be changed during training and βS is a real positive hyperparameter.
REMARK: if Ωv = ∅, the corresponding node will not be trained because we can’t create a target for it.
We will consider η as being a binary vector, but non-binary η can make sense in some situations. Its role is to dynamically control which scales are trained and which are not, so putting ηi = 1 will make the i-th scale trainable. Observe that there is a hierarchy in the generated tree, i.e, the ability to discriminate at a given node will strongly depend on the performance at smaller depths. This ability can be quantified by |Ωv|, which will increase as performance of previous nodes becomes better. This observation encourages us to give more preference at the top nodes at the beginning of the training, which motivates the definition of the dynamic hyperparameter η, and is a form of automatic curriculum learning, since by construction the model has to first learn to do well at the coarsest scales before targets can be defined at the finer scales.
REMARK: If we just train the split architecture neglecting the merge, there is no need to first split recursively the input storing the activations at every node and train all the nodes afterwards. We can train the nodes while we are generating the tree. This training procedure reduces considerably the amount of storing space needed for every batch. However, our goal in future work is to train both architectures together. In this case, we must save the activations because the final target for the split will only be available after a full forward pass through split and merge.
4.4 TRAINING MERGE
We describe here the procedure to learn how to perform the merging selection described in Section 3.2. Analogously to the split, the key point on the merge training is how to build proper targets at every node of the tree having only the final target available y. In this case, this question is more complicated because the merge block not only merges the two inputs preserving the inner ordering of each, but also rules some of them out. Since we only observe input-output pairs, input elements that are not in the target should be ruled out, but we don’t know at which scale of the tree they should be discarded. Let us describe how again thanks to the scale invariance and with appropriate regularization, we can train these operation with only external supervision.
The input x of the merge architecture will be an ordered sequence of disjoint subsets. They correspond to an underlying split tree, so they can be seen as the leaves of a binary tree. First we forward the input to the architecture and merge the outputs of each block recursively storing the activations at every node until we reach the root node. To create the output mask from the probabilities at every node, we always pick the elements belonging to the final convex hull, and the rest will be sampled according to the probabilities. This way, the target elements will always appear in one node at every scale. Denote by x1v, x2v the inputs at node v and yv = y ∩ (x1v ∪ x2v) the intersection of both inputs with the final target. Let pv(φ, xi, x {1,2} v ) denote the probability computed by the merge model that element xi is discarded at node v. The contribution of node v to the total loss is defined as
l(φ, x1v, x 2 v, yv)+βMR(αM , x 1 v, x 2 v) = − ∑ xi∈yv log pv(φ, xi, x{1,2}v )+βM  ∑ xi∈x1v,x2v pv(φ, xi, x{1,2}v )− αM |x1v ∪ x2v| 2 . (8) Here, βM is the hyperparameter controlling the tradeoff between accuracy and complexity, and αM is a shrinkage factor that models the rate by which each input is shrunk at each scale. This shrinkage rate depends on the task at hand and we have not enough supervision in our setup to estimate it from the data. We thus settle for a rate that is scale invariant, that is the total amount of elements at scale i will follow a law of the form αiM . Notice that the cross-entropy term is unbalanced: it only penalizes false negatives, since false positives can be recovered at other scales, but not false negatives.
Finally, we define the total loss of the architecture as
LM (φ, x, y) = l0(φ, x1v0 , x 2 v0 , yv0) + l−1∑ i=1 ∑ v∈Vi [ l(φ, x1v, x 2 v, yv) + βMR(α, x 1 v, x 2 v) ] , with (9)
l0(φ, x1v, x 2 v, yv) = − ∑ xi∈yv log pv(φ, xi, x{1,2}v )− ∑ xi∈(x1v∪x2v)−yv log(1− pv(φ, xi), x{1,2}v ) . (10)
On the last node of the tree we can penalize for both false positives and negatives because its node target corresponds to the final target.
5 EXPERIMENTS
Experiments are implemented in Tensorflow, with reproducible code soon available at https: //github.com/alexnowakvila/DP.
5.1 SORTING
We implemented the split architecture for the task of sorting. This is a good task to test the split block because we can solve it using an oracle splitting block (quicksort), which consists in finding a centered pivot – such as the median. The final targets will be the sorted vector and the input will be the set of elements of the vector. In this case, the dimensionality of the input is d = 1 because we are sorting scalars.
We train the model for vectors of length n = 256 and train until depth 8. We use 40 layers for the sorting block defined in Appendix A, resulting in a total of 240 parameters for the architecture in a whole. The dataset has 4096 input-output pairs and we train during 5 epochs, by varying the input distribution; see Figure 2. The training is performed using batches of size 32. At the beginning we only train the first two scales and train one step deeper every 100 batches.
As described in Appendix A, we normalize the input set before feeding it into the next block using the empirical mean and standard deviation. This normalization is lossless if one also feeds these two empirical moments to the split block, as explained in Appendix A, but we observed no noticeable change in the performance in the sorting case. This is consistent with the fact that sorting is invariant to affine transformations of the input. Similarly as in Ioffe & Szegedy (2015), normalizing the input by its first two moments reduces the effect of numerical instabilities and reduces covariate shift.
We sample from the output probabilities during training to connect different blocks. Sampling is shown to be helpful during the first steps of training because it provides some exploration and avoids the probabilities to get stuck. However, we observed an increase in performance using the mode at test time. We measure our accuracy using the ratio between the inversions of the output and the mean number of inversions. We denote it by Inversions Ratio (IR): IR = inversions1
2 ( n 2)
. We trained the
model using normal and exponential distributions. If we use a uniform distribution the splitting task reduces to split by the mean and this can be achieved with a much simpler block. The results show an impressive generalization performance with input length and robustness with respect to input distribution. It is remarkable to achieve this only using weak supervision (i.e, input-output pairs). Figure 2 presents our numerical results and its analysis.
5.2 PLANAR CONVEX HULL
We trained the merge architecture for the planar convex hull with both weak and strong supervision assuming an oracle split, i.e, the inputs are disjoint convex hulls. By now, for simplicity, the model only learns how to choose a subset of each of both inputs, but not how to merge both subsets. The deterministic policy to merge both chosen subsets is the following. The procedure will compute angles between the vector that goes from the barycenter to every point and the unit vector (−1, 0). The merged ordered sequence will be the points sorted by the corresponding angles. This policy will produce outputs which are stable for small errors in the masking. We used 15 hidden units for each LSTM, giving a total number of parameters of 2511. We trained with batches of size 16 over a training set of size 1024.
It is important to prevent the model to learn the resulting convex hull just by the absolute position of the points. For instance, if the points were drawn from a unit distribution on the unit square, then the model could guess if a point belongs to the convex hull just using its absolute position. As the loss in the weak supervision framework only discriminates between points belonging to the final convex hull, using the absolute position can lead to a bad performance in the intermediate scales. In order to get around this problem, our points were drawn from a uniform distribution on a half unit square [0.5, 0.5]2 whose center is a random point in the square [0.25, 0.75]2. This way, we reduce considerably the reliability of the absolute position to solve the task.
We first train the model using strong supervision, i.e, every node of the tree will use the correct target and all inputs at all nodes will be the correct convex hulls. We can think of strong supervision as a sort of curriculum learning, but here, the block is simultaneously learning with different lengths. At test time, we create the output mask by taking the mode of each output probability (can also try with probabilities). To train using weak supervision, we must put a prior for the complexity of the whole algorithm using αM . Its optimal value is unknown if just using input-output pairs, and in fact,
it depends on the scale and the data distribution. However, we estimate it by αM = ( |Ωt|∑ i |Ωi| )1/l where Ωt is the target, Ωi is the i-th input and l is the number of scales.
At test time, we measure the accuracy of the model with the following quantity: Accuracy = |Ωt∩Ω̂| |Ωt∪Ω̂|
. We adjusted the hyperparameter βM by making the gradient norms of the first part of the loss 8 to be of the same magnitude of the gradient norms of the regularization term. We found the optimal βM to be 10−2. We observed a much worse convergence using a βM larger than the optimal in the case of 3 scales. We also confirmed the necessity of the regularization term for the model to converge (see Figure 3). The average training time per epoch depends on the total number of scales and the experimental αM . For instance, the model trained using WS takes about 1:30 min and 2 min in average for 2 and 3 scales respectively. However, when training using WS for 3 scales without regularization term, the probabilities get stuck at 1, producing an increase in complexity of the model and a training time per epoch of 6 min.
6 DISCUSSION
We have presented a framework that has the ability to leverage an important smoothness prior present in many discrete algorithmic tasks, namely the scale invariance or the ability to divide and conquer. Similarly as the local translation invariance prior when learning with images using CNNs, exploiting this inductive bias breaks the curse of dimensionality and provides a solid foundation to learn complex functional dependencies.
Our approach is an attempt to mimic the behavior of dynamic programming algorithms with neural networks. It is instantiated with two atomic operations – split an input into two, and merge two outputs into a single one – that are recursively applied. This framework allows us to train the system using weak supervision (that is, by only observing input-output pairs), and gives us another bullet: the ability to optimize not only for accuracy but also for complexity, all in a fully differentiable setup. Our numerical results are by all means preliminary, and much is still to be done before this architecture becomes competitive. In particular, we are considering the following directions.
Current work: We are currently working in two major aspects. The first one is to generalize the merge step so that it can not only perform the selection but also the concatenation in a fully learnt manner. This operation can also be implemented with o(n) architectures since the partial order at the output respects the partial order in each of the two inputs. The second one is to perform the joint training of both split and merge blocks, which will allow us to learn with really weak supervision. For that purpose, it is necessary to perform target propagation to provide suitable targets for the split block. Lastly, we are in the process of comparing our results with standard baselines that do not exploit the dynamic structure of the problem.
Future work: The number of extensions and applicability of this model is vast. In particular, we want to extend split and merge architectures in graph problems such as shortest paths or spanning trees. For that purpose, we will use Graph Neural Networks Scarselli et al. (2009); Sukhbaatar et al. (2016) as atomic models to handle the data. We also want to explore more generic architectures for the atomic split and merge, perhaps with Θ(n log n) complexity, to cover more territory between linear and quadratic complexity, and to test the ability of the model to learn good approximations in tasks that are NP-hard. Another question that this model raises is the consistency of weak supervision thanks to the scale invariance. A byproduct of our scale invariance seems to be the ability to propagate and diffuse the available targets at the coarsest scale at all scales, leading to ‘selfconsistent’ supervision. We believe this is a profound question. Finally, we want to explore the links between these ideas and that of the hierarchical Reinforcement learning, which is an extreme form of very weak supervision. Designing intermediate rewards is akin to our setup of defining suitable targets at intermediate scales, albeit with an extra degree of difficulty.
A DETAILS ON SPLIT ARCHITECTURE
A.1 SPLIT
Let’s define for k = 1, . . . , r layers
F 0i = xi, F k i = F k i (xi, z k−1, F k−1i , θ) ∈ R d i = 1, . . . , n where zk = 1n ∑n i=1 F k i is the empirical moment defined by Fi and θ is a set of parameters. Then, p(x) = σ(F r) is the vector of probabilities, i.e, pi = P(xi ∈ Ω1). This is the most general expression of our architecture. The idea is that the hidden units in layer k at position i depend on xi (input at same index), zk−1 (average of all hidden states of previous layer) and F ki (previous hidden unit at same index).
We reduce the covariate shift of the split architecture by normalizing the input sets by their empirical mean and covariance:
x̃ = x− µ̂ σ̂ .
Since both µ̂ and σ̂ are two empirical averages as the zk’s, they can be seamlessly integrated as extra averages by concatenating them to zk.
We have parametrized the functions F ki in the following way:
F ki = φ k 0 × F k−1i + φ k 1 × φk2
where φk0 , φ k 1 and φ k 2 have the following form:
φk0 = σ(C k 0xi+W k 0 z (k−1)+bk0) , φ k 1 = σ(C k 1xi+W k 1 z (k−1)+bk1) , φ k 2 = tanh(C k 2xi+W k 2 z (k−1)+bk2) ,
and Cki ,W k i are d× d matrices and bki ∈ Rd a bias vector. Note that the total number of parameters is r(3d2 + 3d) where r is the number of layers and d is the dimensionality of the input elements.

The basic idea of this contribution is very nice and worth pursuing: how to use the powerful “divide and conquer” algorithm design strategy to learn better programs for tasks such as sorting or planar convex hull. However, the execution of this idea is not convincing and needs polishing before acceptance. As it is right now, the paper has a proof-of-concept feel that makes it great for a workshop contribution.
My main concern is that the method presented is currently not easily applicable to other tasks. Typically, demonstrations of program induction from input-output examples on well known tasks serves the purpose of proving, that a generic learning machine is able to solve some well known tasks, and will be useful on other tasks due to its generality. This contribution, however, presents a learning machine that is very hand-tailored to the two chosen tasks. The paper essentially demonstrates that with enough engineering (hardcoding the recurrency structure, designing problem-specific rules of supervision at lower recurrency levels) one can get a partially trainable sorter or convex hull solver.
I found the contribution relatively hard to understand. High level ideas are mixed with low-level tricks required to get the model to work and it is not clear either how the models operate, nor how much of them was actually learned, and how much was designed. The answer to the questions did hep, nut didn't make it into the paper. Mixing the descriptions of the tricks required to solve the two tasks makes things even more confusing. I believe that the paper would be much more accessible if instead of promising a general solution it clearly stated the challenges faced by the authors and the possible solutions.
Highlights:
+ Proof-of-concept of a partially-trainable implementation of the important “divide and conquer” paradigm
++ Explicit reasoning about complexity of induced programs
- The solution isn’t generic enough to be applicable to unknown problems - the networks require tricks specific to each problem
- The writing style pictures the method as very general, but falls back on very low level details specific to each task