The scattering framework offers an optimal hierarchical convolutional decomposition according to its kernels. Convolutional Neural Net (CNN) can be seen as an optimal kernel decomposition, nevertheless it requires large amount of training data to learn its kernels. We propose a trade-off between these two approaches: a Chirplet kernel as an efficient Q constant bioacoustic representation to pretrain CNN. First we motivate Chirplet bioinspired auditory representation. Second we give the first algorithm (and code) of a Fast Chirplet Transform (FCT). Third, we demonstrate the computation efficiency of FCT on large environmental data base: months of Orca recordings, and 1000 Birds species from the LifeClef challenge. Fourth, we validate FCT on the vowels subset of the Speech TIMIT dataset. The results show that FCT accelerates CNN when it pretrains low level layers: it reduces training duration by -28% for birds classification, and by -26% for vowels classification. Scores are also enhanced by FCT pretraining, with a relative gain of +7.8% of Mean Average Precision on birds, and +2.3% of vowel accuracy against raw audio CNN. We conclude on perspectives on tonotopic FCT deep machine listening, and inter-species bioacoustic transfer learning to generalise the representation of animal communication systems.
1 INTRODUCTION
Representation of bioacoustic sequences started with ’Human’ speech in the 70’. Speech automatic processing yields to the efficient Mel Filter Cepstral Coefficients (MFCC) representation. Today new bioacoustic representation paradigms arise from environmental monitoring and species classification at weak Signal to Noise Ratio (SNR) and with small amount of data per species.
Several neurobiological evidences suggest that auditory cortex is tuned to complex time varying acoustic features, and consists of several fields that decompose sounds in parallel (Kowalski et al., 1996; Mercado et al., 2000). Therefore it is more than reasonable to investigate the Chirplet timefrequency representation from acoustic and neurophysiological points of view.
Chirps, or transient amplitude and frequency modulated waveforms, are ubiquitous in nature systems (Flandrin (2001)), ranging from bird songs and music, to animal vocalization (frogs, whales) and Speech. Moreover the sinusoidal models are a typical attempt to represent audio signals as a superposition of chirp-like components. Chirp signals are also commonly observed in biosonar systems.
The Chirplet transform subsumes both Fourier analysis and wavelet analysis, providing a broad framework for mapping one-dimensional sound waveforms into a n-dimensional auditory parameter space. It offers the processing described in different auditory fields, i.e. cortical regions with systematically related response sensitivities. Moreover, Chirplet spaces are highly over-complete because there is an infinite number of ways to segment a time-frequency plane, the dictionary is redundant: this corresponds well with the overlapping, parallel signal processing pathways of auditory cortex.
Then we suggest that low level CNN layers shall be pretrained by Chirplet kernels. Thus, we define and code a Fast Chirplet Transform (FCT). We conduct validation on real recordings of whale and birds, and on Speech (vowels subset of TIMIT). We demonstrate that CNN classification benefits from low level layers FCT pretraining. We conclude on the perspectives of tonotopic FCT machine listening and inter-species transfer learning.
2 FORMAL DEFINITION OF CHIRPLET
A chirplet can be seen as a complex sinus with increasing or decreasing frequency over time modulated by a Gaussian window to have a localized support in the time and Fourier domain. It is a broad class of filters which includes wavelets and Fourier basis as special cases. As a result, and as presented in (Mann & Haykin, 1991; 1992), the Chirplet transform is a generalization of many known time-frequency representations. We first present briefly the wavelet transform framework to extend it to Chirplets. Given an input signal x one can compute a wavelet transform (Mallat, 1999) through the application of multiple wavelets ψλ. A wavelet is an atom with localized support in time and frequency domain which integrates to 0. The analytical support of the wavelets is not compact but they are very well localized. It can be considered compact in the applied case where roundoff error lead to 0 quickly after moving around the center frequency. The whole filter bank is derived from a mother wavelet ψ0 and a set of dilation coefficients following a geometric progression defined as Λ = {21+j/Q, j = 0, ..., JQ − 1} with J being the number of octave to decompose and Q the number of wavelets per octave. As a result, one can create the filter-bank as the collection {ψ0( tλ ) := ψλ, λ ∈ Λ}. After application of the filter-bank, one ends up with a time-scale representation, or scalogram, Ux(λ, t) := |(x ? ψλ)(t)| where the complex modulus was applied in order to remove the phase information and contract the space. It is clear that a wavelet filter-bank is completely characterized by its mother wavelet and the set of scale parameters. Generalizing this framework for Chirplets will be straightforward by now allowing a nonconstant frequency for each filter. As for wavelets, filters are generated from a Gaussian window determining the time support however the complex sinus has nonconstant frequency over time with center-frequency fc. Since the scope of the parameters leads infinitely many different possible filters, we have to restrain ourselves, and thus create only a fixed Chirplet filter-bank allowing fast computations. The parameters defining these filters include the time position tc, the frequency center fc, the duration ∆t and the chirp rate c:
gtc,fc,log(∆t),c(t) = 1√√ π∆t e − 12 (t−tc)2 ∆2t ej2π(c(t−tc) 2+fc(t−tc)). (1)
3 PROPOSITION OF A FAST CHIRPLET TRANSFORM (FCT)
The parameter space is basically of infinite dimension. Similarly to continuous wavelet transform however, it is possible to use some a priori knowledge in order to create a finite bank-filter. For example, wavelets are generated by knowing the number of wavelets per octave and the number of octave to decompose. As a result, we used the same motivation in order to reduce the number of possible Chirplets required. The goal here is not to compute an invertible transform, but rather provide a redundant transformation highlighting transient structures which are not the same tasks as discussed in (Coifman et al., 1992; Meyer, 1993; Coifman et al., 1994). As a result, we keep the same overall framework as for wavelets with the Q and J parameters. For example parameters for bird songs in this paper are J = 6 and Q = 16 with a sampling rate (SR) of 44100Hz, and J = 4 and Q = 16 on speech and Orca with SR=16 kHz). Finally, since we are interested in frequency modulations, we compute the ascendant and descendant chirp filters as one being the symetrized version of the other. As a result, we use a more straightforward analytical formula defined with a starting frequency F0, an ending frequency F1, and the usual wavelet like parameters σ being the
bandwidth. Finally the hyperparameter p defining the polynomial order of the chirp is constant for the whole bank-filter generation. For example, the case p = 1 leads to a linear chirp, p = 2 to a quadratic chirp. The starting and ending frequencies are chosen to approximately cover one octave and are directly computed from the λ parameters which define the scales. Finally, following the scattering network inspiration from (Bruna & Mallat, 2013), in order to remove unstable noisy pattern, we apply a low-pass filter (a Gaussian blurring) and thus we increase the SNR of the representation.
Λ = {2.01+i/Q, i = 0, ..., J ×Q− 1}, (2)
F0 = Fs
2λ , λ ∈ Λ, (3)
F1 = Fs
λ , λ ∈ Λ, (4)
σ = 2 d
λ , λ ∈ Λ. (5)
4 LOW COMPLEXITY FCT ALGORITHM AND IMPLEMENTATION
We give here our code of Fast Chirplet Transform (FCT), taking advantage of the a priori knowledge for the filter-bank creation and the fast convolution algorithm 1. Therefore, we first create the Chirplet with the ascendant and descendant versions in once (see Annexe Algo 1).
Then we generate the whole filter-bank (see Algo 2 in annexe) with the defined λ and hyperparameters.
Finally, we use the scattering framework (Bruna & Mallat, 2013; Andén & Mallat, 2014): we apply a local low-pass filter to the obtained representation. In fact, the scattering coefficients Sx result from a time-averaging on the time-frequency representation Ux bringing local and up to global time-invariance. This time-averaging is computed through the application of the φ filter, usually a Gabor atom with specified standard deviation and such that∫
φ(t)dt = 1. (6)
As a result, one computes these coefficients as: Sx(λ, t) = (|x ? ψλ| ? φ) (t), where ψλ is a Chirplet with λ parameters and φ. Similarly, we perform local time-averaging on the Chirplet representation in the same manner.
We present some possible filters in Fig. 2, and some bird features Fig. 3.
The third step in our FCT consists in the reduction of the convolution task. The asymptotic complexity of the Chirplet transform is O(N. log(N)) with N being the size of the input signal. This is the same asymptotic complexity as for the continuous wavelet transform and the scattering network. However, it is possible to reach lower asymptotic complexity simply by a division of the convolution task. usually the convolutions are carried through application of an element-wise multiplication of the signal and the filter in the frequency domain and then compute the inverse Fourier transform to end up with x ? ψλ. However, if we denote by M the length of the filter ψλ it is possible to instead perform multiple times this operation on different overlapping chunks of the signal to then concatenate the results to obtain at the end the same convolution result but now in O(N. log(M)). Finally a last improvement induced by this approach is to allow easy tackling of signals with a length just above a power of 2 which otherwise would require to be padded in order to obtain a FFT with real O(N. log(N)) complexity through the Danielson-Lanczos lemma (Press, 2007). Applying this scheme allowed to compute the convolutions between 3 to 4 times faster. The variations came from the distance between N and the closest next power of 2 depending on the desired chunk size.
We validate the efficiency of FCT on real bioacoustic recordings. We processed on 10 medium speed CPUs of 4 years old, 100 hours of recording of LifeClef bird challenge (16 kHz Sampling Rate (SR), 16 bits) in 2 days. Second, we processed in 7 days the equivalent of 1 month of
1We provide our implementation in Annexe and: https://github.com/DYNI-TOULON/ fastchirplet.git
Orca whale recordings from Orcalab.org ONG (22 kHz SR, 16 bits), in Fig. 1,2,3 and at http: //sabiod.univ-tln.fr/orcalab .
.
.
5 ENHANCING CNN BIOACOUSTIC REPRESENTATION WITH FCT
A strategy for CNN fine-tuning can be to retrain a classifier on top of a CNN on a new dataset, or to fine-tune the weights of a pretrained network by continuing the backpropagation. It is possible to
fine-tune all the layers of the CNN or to freeze some of the earlier, later or central layers, and to only fine-tune some portion of the network. As the features propagate deeper and deeper in the network layers, they become increasingly invariant and discriminative (Seltzer 2013). Thus usually only the higher level are fine-tuned, the earlier features of a CNN contain more generic features that should be useful to many tasks. As denoted in later layers of the CNN becomes progressively more specific to the details of the classes contained in the original dataset.
In this paper we adapt our parametric Chirplet decomposition to a specific acoustic domain with a specific CNN. We compare a CNN trained on raw audio to one trained on Mel and Chirplet. The best model is the one trained on parametric Chirplet. Second, we show that the CNN can be enhanced by pretraining Chirp in low level layer.
5.1 CNN BIRDS CLASSIFICATION ON FCT, RAW AUDIO, VERSUS MEL
The first demonstration is conducted on complex Bird songs. We use the BIRD10 subset of LifeClef 2016 bird classification challenge. It was used as ENS Ulm data challenge 2016, and contains 3 species in a total of 15 minutes of recordings (SR 44100 Hz, 16 bits), and is available (.wav, Mel and FCT features) at http://sabiod.univ-tln.fr/workspace/BIRD10.
We train 3 CNNs (LeCun & Bengio, 1995) on the Lasagne Theano platform. The baseline CNN is trained from the raw audio. A second CNN, with similar topology (see annexe) is trained on a simple log of the simple 64 channels Mel scale of FFT spectrum ( http://pydoc.net/Python/librosa/0.2.0/librosa.feature/ ). We overlap by 90% the time windows. A third CNN is trained on our FCT. The parameters of both CNN are similar, with 64 frequency bands each (we remove top and bottom band from the Chirplet to set to 64 bands only). Then the input layer is 64 x 86, the Conv layer has 20 filters of size 8 x 10. All activation functions are relu. We maxpool 2 x 2, follow the 20 filters of size 8 x 10, maxpooling, dense layer (200), dropout at 10%, with a final softmax dense layer with 3 classes and same dropout. Each CNN is trained by cross-entropy, L2 reg., with a learning rate set of 0.001.
The Fig. 4 gives the MAP of these two CNNs having similar hyperparameters. The CNN on FCT gives the best MAP with 61.5% at epoch 280 compared to later epoch (820) for Mel with a similar MAP of 61%. Audio is slower and weaker (58% MAP at epoch 1140).
5.2 ENHANCING BIRDS CLASSIFICATION STACKING PRETRAINED CHIRPNET CNN
In order to test the efficiency of the FCT, we pretrain a CNN to encode audio to Chirplets (a.k.a. the Audio2Chirp CNN) and a CNN to convert parametric Chirplet to classes (a.k.a. the Chirp2Class CNN). The topology of these CNNs (Tab. 2, 3) is set for reasonable time of training. We also speed up the training with shorter time overlap of the time windows (only 30% instead of 90% in the previous experimentation). We then decrease the average MAP, however the objective here is to compare the gain in MAP and time of convergence in stacked Chirplet deep representations.
We then simply stack at low level layer the audio2chirp with the chirp2class CNN to build a complete audio2class CNN. We train it from random initialization, or from pretrained CNN. Note that the random seed in all the experimentation of this paper is fixed to allow fair comparisons. Results are reported in Tab. 1 for each of the stacked CNN, with the epoch giving the best MAP on the dev. set, and the corresponding MAP on the test set. Results demonstrate that the pretraining of low level layers by FCT enhances CNN. More details are given in Annexe.
5.3 ENHANCING VOWELS CLASSIFICATION STACKING PRETRAINED CHIRPNET CNN
In this section we run the same demonstration on the subset of speech vowels of all the TIMIT acoustic-phonetic corpus JS et al. (1993): 3,696 training utterances (sampled at 16kHz) from 462 speakers. The cross-validation set consists of 400 utterances from 50 speakers. The core test set of the 8 vowels subset was used to report the results: 192 utterances from 24 speakers, excluding the validation set. There are 61 hand labeled phonetic symbols but the experiments in this paper run on the time windows of 310ms centered on each of the 8 vowels of TIMIT (= iy, ih, eh, ae, aa, ah, uh, uw).
Due to similar bioacoustic voicing dynamics of the two species (near 4 Hz), we simply set the FCT parameters for vowel to the one used for Orca presented above (p = 3,j = 4,q = 16,t = 0.001,s = 0.01). The time windows are set to 310 ms as recommended in Palaz et al. (2013).
The results of the different training stages of the audio2chirp and chirp2class and stacked model are given in Tab.2 and Annexe. We run due to lack of time the experiment only on vowel classification, which does not really allow comparison with other papers, however this seminal work only aims to study the relative gain between CNN pretrained or not by FCT.
The results demonstrate that FCT pretraining of the audio2class model is improved by 2.3% of relative gain of accuracy while the training time is decreased by 26%.
6 DISCUSSION AND CONCLUSION
In this paper we propose for the first time at our knowledge the definition and implementation of a Fast Chirplet Transform (FCT). Due to its low complexity, FCT can be computed as fast as FFT.
.
Second we show that FCT pretraining accelerates CNN. For Bird10 data set, we have 280 epochs using FCT versus 820 on Mel features, or 1140 on raw audio for same MAP score. The stacked CNN with the chirpnet in low level layer also decreases training from 530 epochs to 380 epochs, while
it increases MAP by 4 points (Tab. 1). The experiment on Vowels demonstrates a training of 30 epochs on FCT, versus 60 on raw audio (for same 65% accuracy level), and an increase of 1.5 point of accuracy (Tab. 2).
These gains may be due to the sparsity of the Chirplet, and the denoising step in the FCT. These experiences bring to light the problem of deep learning for small and biased dataset for which a full learning strategy is sub-optimal due to local optimum convergence. As a result, FCT prior knowledge can be used to mitigate this drawback by reducing the complexity of the deep-net architecture.
Three main perspectives are then opened. Future work will consist on sparse Chirpnet inspired from tonotopic net Strom (1997), auditory nerve and cortex topology Pironkov et al. (2015). The acoustic vibrations are transmitted to the base of the cochlea, thus each region of the basilar membrane are excited by different frequencies. The higher frequencies excite areas closer to the cochlea base, whereas lower frequencies are closer to the apex. This implies that neurons connected to a specific zone of the basilar membrane will be simultaneously stimulated inducing tonotopic representation.
A second perspective is to integrate Chirplet computation into the CNN training itself, as a constrained embedded layer, in a framework similar to a Wavelet Neural Network (Adeli & Jiang, 2006) but with Chirplet activation functions.
Last, we currently work on transfer learning of Chirpnet from animal to speech (and reverse), in order to generalize a deep Chirpnet representation of the animal communication systems.
7 ACKNOWLEDGEMENTS
We thank colleagues from ENS Paris Data Team with S. Mallat, and P. Flandrin, for fruitful discussions on Scattering and Chirplet. We thank YLC and YB for advises on CNN. We thank V. Tassan for cleaning the code. We used Theano, Lasagne 2, Librosa 3 and Pysoundfile 4.
A BIRD DATASET
The experiment is conducted on BIRD10, an online data set http://sabiod.univ-tln.fr/ workspace/BIRD10/ which is a subset of the training LIFEClef 2016 challenge on bird classification. BIRD10 contains 454 audio files (22050 Hz SR, 16 bits) from 10 bird classes, split in 0.5s segments. 20% of the training set was used as the validation set.
Only segments with detected bird activity were kept, assuming a bird sound to have prominent energy and to be mostly harmonic. This bird detection is for a given segment:
if (energy_ratio > energy_threshold and spectral_flatness_weighted_mean < spectral_flatness_threshold) bird_detected = True
else bird_detected = False
where the energy and the spectral flatness are computed on 50% overlapping frames of 256 samples:
er = energy_ratio = mean(seg_energy)
95thpercentile(file_energy)
swf = spectral_flatness_weighted_mean = sum(seg_spectral_flatness× seg_energy)
sum(seg_energy) This naive algorithm performed quite well on a manually labelled dataset of bird vocalizations (precision=0.89, recall=0.57 for er=0.2 and sfw=0.3) after a quick grid search on the two parameters.
B BIRDS CLASSIFICATION : BASELINE CNNS
The first experiment consisted in running similar CNNs to compare the performance of using raw audio and two time-frequency representations as the input: a standard log-amplitude Mel spectrum and the Chirplet representation described in the first part of this paper. In this experiments the segments were overlapping by 90%. The topologies of the networks are given in Tab. 2. The cost function is the cross-entropy, learning rate = 0.0001 = L2 regularisation coefficient. The Mel spectrum is computed from 64 bands between 0 and 11025 Hz (=SR/2). Both Mel spectrum and Chirplets were normalized by Z-score.
In all experiments, a given topology is always initialized using the same set of random parameters, unless specified otherwise. The value * (resp. 0) after the name of the net refers to the pretrained net (resp. random initialization).
C BIRD AUDIO2CHIRP - CHIRPLET ENCODER
The chirp encoder, aka audio2chirp, aims at training a net to get a Chirplet-like representation. It is a simple CNN taking audio as input, Chirplets as output and minimizing the square error. It converges easily in 180 epochs. The topology of the audio2chirp net is given Tab. 4.
D BIRD: TRAINING, DEV AND TESTING CURVES OF THE DIFFERENT CNNS
E EXPERIMENT ON SPEECH VOWEL
The Tab. 5 gives the topology of the audio2chirp and chirp2class, and stacked models, for these vowel experiments.
In all experiments, each CNN is initialized using the same random seed. The symbol “*” refers to the optimal trained parameters of a net.
F ALGORITHM FOR THE FAST CHIRPLET TRANSFORM (FCT)
Algo 1: Chirplet Generation INPUT: F0,F1,Fs,sigma,p OUTPUT: coefficients_upward,coefficients_downward if(p):
w=cos(2*pi*((F1-F0)/((p+1)*sigma**p)*t**p+F0)*t) else:
w=cos(2*pi*((F0*(F1/F0)**(t/sigma)-F0)*sigma/log(F1/F0))) coefficients_upward=w*exp(-((t-\sigma/2.0)**2)/(2*sigma**2)) coefficients_downward=flipud(coefficients_upward).
Algo 2: Chirplet Filter-Bank Generation INPUT: J, Q, Fs, sigma, p lambdas = 2.0**(1+arrange(J*Q)/float(Q)) start_frequencies = (Fs /lambdas)/2.0 end_frequencies = Fs /lambdas distances = 2.0*d/flipud(lambdas) filters=list() for f0,f1,d in zip(start_frequencies,end_frequencies,distances):
filters.append(chirplet(Fs,f0,f1,d,p)) return filters.
G THE PYTHON CODE FOR THE FAST CHIRPLET TRANSFORM (FCT)
This code, in GPL licence (c) DYNI team, is in Github : https://github.com/DYNI-TOULON/fastchirplet.git.
import l i b r o s a import os import numpy as np from p y l a b import ∗ import s y s from numpy . l i b import pad
c l a s s C h i r p l e t :
" " " s m a l l e s t t i m e b i n among t h e c h i r p l e t " " " g l o b a l s m a l l e s t _ t i m e _ b i n s
def _ _ i n i t _ _ ( s e l f , s a m p l e r a t e , F0 , F1 , sigma , po lynome_degree ) :
" " " l o w e s t f r e q u e n c y where t h e c h i r p l e t i s a p p l i e d " " " s e l f . m in_ f r equency = F0
" " " h i g h e s t f r e q u e n c y where t h e c h i r p l e t i s a p p l i e d " " " s e l f . max_f requency = F1
" " " s a m p l e r a t e o f t h e s i g n a l " " " s e l f . s a m p l e r a t e = s a m p l e r a t e
" " " d u r a t i o n o f t h e c h i r p l e t " " " s e l f . t i m e _ b i n = sigma / 1 0
" " " de gr e e o f t h e polynome t o g e n e r a t e t h e c o e f f i c i e n t s o f t h e c h i r p l e t " " " s e l f . po lynome_degree = polynome_degree
" " " c o e f f i c i e n t s a p p l i e d t o t h e s i g n a l " " " s e l f . f i l t e r _ c o e f f i c i e n t s = s e l f .
c a l c u l _ c o e f f i c i e n t s ( )
def c a l c u l _ c o e f f i c i e n t s ( s e l f ) : " " " c a l c u l a t e c o e f f i c i e n t s f o r t h e c h i r p l e t s " " " # p r i n t ( s e l f . _ s a m p l e r a t e ) t = l i n s p a c e ( 0 , s e l f . t ime_b in , i n t ( s e l f . s a m p l e r a t e ∗
s e l f . t i m e _ b i n ) ) i f ( s e l f . po lynome_degree ) :
w = cos (2∗ p i ∗ ( ( s e l f . max_frequency−s e l f . m in_ f r equency )
/ ( ( s e l f . po lynome_degree +1) ∗ s e l f . t i m e _ b i n ∗∗ s e l f . po lynome_degree ) ∗ t ∗∗ s e l f . po lynome_degree + s e l f . m in_ f r equency ) ∗ t )
e l s e : w = cos (2∗ p i ∗ ( ( s e l f . m in_ f r equency ∗ ( s e l f .
max_f requency / s e l f . m in_ f r equency )
∗∗ ( t / s e l f . t i m e _ b i n )− s e l f . m in_ f r equency ) ∗ s e l f . t i m e _ b i n
/ l o g ( s e l f . max_f requency / s e l f . m in_ f r equency ) ) )
c o e f f s = w∗ hann ing ( l e n ( t ) ) ∗∗2
re turn c o e f f s
def smooth_up ( s e l f , i n p u t _ s i g n a l , sigma , end_smooth ing ) : # g e n e r a t e f a s t f o u r i e r t r a n s f o r m from a s i g n a l and smooth
i t
new_up = b u i l d _ f f t ( i n p u t _ s i g n a l , s e l f . f i l t e r _ c o e f f i c i e n t s , s igma ) re turn f f t _ s m o o t h i n g ( f a b s ( new_up ) , end_smooth ing )
def compute ( i n p u t _ s i g n a l , s ave = F a l s e , d u r a t i o n _ l a s t _ c h i r p l e t = 1 . 0 1 , num_octaves =5 , n u m _ c h i r p s _ b y _ o c t a v e =10 , po lynome_degree =3 ,
end_smooth ing = 0 . 0 0 1 ) : " " " main f u n c t i o n . Fas t C h i r p l e t Trans form from a s i g n a l " " "
da ta , s a m p l e r a t e = l i b r o s a . l o a d ( i n p u t _ s i g n a l , s r =None )
s i z e _ d a t a = l e n ( d a t a )
n e a r e s t _ p o w e r _ 2 = 2∗∗ ( s i z e _ d a t a −1) . b i t _ l e n g t h ( )
d a t a = np . l i b . pad ( da t a , ( 0 , n e a r e s t _ p o w e r _ 2−s i z e _ d a t a ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
c h i r p l e t s = i n i t _ c h i r p l e t _ f i l t e r _ b a n k ( s a m p l e r a t e , d u r a t i o n _ l a s t _ c h i r p l e t , num_octaves , num_ch i rps_by_oc tave , po lynome_degree )
c h i r p s = a p p l y _ f i l t e r b a n k ( da t a , c h i r p l e t s , end_smooth ing )
c h i r p s = r e s i z e _ c h i r p s ( s i z e _ d a t a , n e a r e s t _ p o w e r _ 2 , c h i r p s )
i f s ave : i f not os . p a t h . e x i s t s ( " csv " ) :
os . m a k e d i r s ( " csv " ) np . s a v e t x t ( " csv / "+os . p a t h . basename ( i n p u t _ s i g n a l ) .
s p l i t ( ’ . ’ ) [ 0 ] + ’ . c sv ’ , c h i r p s , d e l i m i t e r =" , " )
re turn c h i r p s
def r e s i z e _ c h i r p s ( s i z e _ d a t a , n e a r e s t _ p o w e r _ 2 , c h i r p s ) : s i z e _ c h i r p s = l e n ( c h i r p s ) r a t i o = s i z e _ d a t a / n e a r e s t _ p o w e r _ 2 s i z e = i n t ( r a t i o ∗ l e n ( c h i r p s [ 0 ] ) )
t a b f i n a l = np . z e r o s ( ( s i z e _ c h i r p s , s i z e ) ) f o r i in range ( 0 , s i z e _ c h i r p s ) : t a b f i n a l [ i ]= c h i r p s [ i ] [ 0 : s i z e ] re turn t a b f i n a l
def i n i t _ c h i r p l e t _ f i l t e r _ b a n k ( s a m p l e r a t e , d u r a t i o n _ l a s t _ c h i r p l e t , num_octaves , num_ch i rps_by_oc tave , p ) :
" " " g e n e r a t e a l l t h e c h i r p l e t s from a g i v e n sample r a t e " " "
lambdas = 2 . 0∗∗ ( 1 + a r a n g e ( num_octaves ∗ n u m _ c h i r p s _ b y _ o c t a v e ) / f l o a t ( n u m _ c h i r p s _ b y _ o c t a v e ) ) #Low f r e q u e n c i e s f o r a s i g n a l s t a r t _ f r e q u e n c i e s = ( s a m p l e r a t e / lambdas ) / 2 . 0 # h igh f r e q u e n c i e s f o r a s i g n a l e n d _ f r e q u e n c i e s = s a m p l e r a t e / lambdas d u r a t i o n s = 2 . 0∗ d u r a t i o n _ l a s t _ c h i r p l e t / f l i p u d ( lambdas ) C h i r p l e t . s m a l l e s t _ t i m e _ b i n s = d u r a t i o n s [ 0 ] c h i r p l e t s = l i s t ( ) f o r f0 , f1 , d u r a t i o n in z i p ( s t a r t _ f r e q u e n c i e s ,
e n d _ f r e q u e n c i e s , d u r a t i o n s ) : c h i r p l e t s . append ( C h i r p l e t ( s a m p l e r a t e , f0 , f1 ,
d u r a t i o n , p ) ) re turn c h i r p l e t s
def a p p l y _ f i l t e r b a n k ( i n p u t _ s i g n a l , c h i r p l e t s , end_smooth ing ) : " " " g e n e r a t e l i s t o f s i g n a l w i t h c h i r p l e t s " " " r e s u l t = l i s t ( ) f o r c h i r p l e t in c h i r p l e t s :
r e s u l t . append ( c h i r p l e t . smooth_up ( i n p u t _ s i g n a l , 6 , end_smooth ing ) )
re turn a r r a y ( r e s u l t )
def f f t _ s m o o t h i n g ( i n p u t _ s i g n a l , s igma ) : " " " smooth t h e f a s t t r a n s f o r m f o u r i e r " " " s i z e _ s i g n a l = i n p u t _ s i g n a l . s i z e # s h o r t e n t h e s i g n a l new_s ize = i n t ( f l o o r ( 1 0 . 0∗ s i z e _ s i g n a l ∗ s igma ) ) h a l f _ n e w _ s i z e = new_s ize / / 2
f f t x = f f t ( i n p u t _ s i g n a l ) s h o r t _ f f t x = [ ] f o r e l e in f f t x [ : h a l f _ n e w _ s i z e ] : s h o r t _ f f t x . append ( e l e ) f o r e l e in f f t x [− h a l f _ n e w _ s i z e : ] :
s h o r t _ f f t x . append ( e l e )
a p o d i z a t i o n _ c o e f f i c i e n t s = g e n e r a t e _ a p o d i z a t i o n _ c o e f f i c i e n t s ( h a l f _ n e w _ s i z e , sigma , s i z e _ s i g n a l ) # a p p l y t h e a p o d i z a t i o n c o e f f i c i e n t s s h o r t _ f f t x [ : h a l f _ n e w _ s i z e ] ∗= a p o d i z a t i o n _ c o e f f i c i e n t s # a p p l y t h e a p o d i z a t i o n c o e f f i c i e n t s i n a r e v e r s e l i s t s h o r t _ f f t x [ h a l f _ n e w _ s i z e : ] ∗= f l i p u d ( a p o d i z a t i o n _ c o e f f i c i e n t s ) r e a l i f f t x w = r e a l ( i f f t ( s h o r t _ f f t x ) ) re turn r e a l i f f t x w
def g e n e r a t e _ a p o d i z a t i o n _ c o e f f i c i e n t s ( num_coeffs , sigma , s i z e ) : " " " g e n e r a t e a p o d i z a t i o n c o e f f i c i e n t s " " " a p o d i z a t i o n _ c o e f f i c i e n t s = a r a n g e ( num_coef f s ) a p o d i z a t i o n _ c o e f f i c i e n t s = a p o d i z a t i o n _ c o e f f i c i e n t s ∗∗2
a p o d i z a t i o n _ c o e f f i c i e n t s = a p o d i z a t i o n _ c o e f f i c i e n t s / ( 2 ∗ ( s igma ∗ s i z e ) ∗∗2) a p o d i z a t i o n _ c o e f f i c i e n t s = exp(− a p o d i z a t i o n _ c o e f f i c i e n t s ) re turn a p o d i z a t i o n _ c o e f f i c i e n t s
def f f t _ b a s e d ( i n p u t _ s i g n a l , h , boundary =0) : M=h . s i z e h a l f _ s i z e = M/ / 2 i f ( boundary ==0) : #ZERO PADDING
i n p u t _ s i g n a l =pad ( i n p u t _ s i g n a l , ( h a l f _ s i z e , h a l f _ s i z e ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0) h=pad ( h , ( 0 , i n p u t _ s i g n a l . s i z e−M) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0) newx= i f f t ( f f t ( i n p u t _ s i g n a l ) ∗ f f t ( h ) ) re turn newx [M−1:−1]
e l i f ( boundary ==1) : # s y m m e t r i c i n p u t _ s i g n a l = c o n c a t e n a t e ( [ f l i p u d ( i n p u t _ s i g n a l [ :
h a l f _ s i z e ] ) , i n p u t _ s i g n a l , f l i p u d ( i n p u t _ s i g n a l [ h a l f _ s i z e : ] ) ] )
h=pad ( h , ( 0 , x . s i z e−M) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0) newx= i f f t ( f f t ( i n p u t _ s i g n a l ) ∗ f f t ( h ) ) re turn newx [M−1:−1]
e l s e : # p e r i d i c re turn r e a l ( r o l l ( i f f t ( f f t ( i n p u t _ s i g n a l ) ∗ f f t ( h ,
i n p u t _ s i g n a l . s i z e ) ) ,− h a l f _ s i z e ) )
def b u i l d _ f f t ( i n p u t _ s i g n a l , f i l t e r _ c o e f f i c i e n t s , n =2 , boundary =0) : " " " g e n e r a t e f a s t t r a n s f o r m f o u r i e r by windows " " "
M= f i l t e r _ c o e f f i c i e n t s . s i z e # p r i n t ( n , boundary ,M) h a l f _ s i z e = M/ / 2 s i g n a l _ s i z e = i n p u t _ s i g n a l . s i z e # power o f 2 t o a p p l y f a s t f o u r i e r t r a n s f o r m windows_s ize = i n t (2∗∗ c e i l ( l og2 (M∗ ( n +1) ) ) ) number_of_windows= f l o o r ( s i g n a l _ s i z e / / windows_s ize ) i f ( number_of_windows ==0) :
re turn f f t _ b a s e d ( i n p u t _ s i g n a l , f i l t e r _ c o e f f i c i e n t s , boundary )
o u t p u t = e m p t y _ l i k e ( i n p u t _ s i g n a l ) #pad w i t h 0 t o have a s i z e i n a power o f 2 z e r o p a d d i n g = pad ( f i l t e r _ c o e f f i c i e n t s , ( 0 , windows_s ize−M) , ’
c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
h _ f f t = f f t ( z e r o p a d d i n g )
# t o browse t h e whole s i g n a l c u r r e n t _ p o s =0
# a p p l y f f t t o a p a r t o f t h e s i g n a l . T h i s p a r t has a s i z e which i s a power o f 2 i f ( boundary ==0) : #ZERO PADDING #window i s h a l f padded w i t h s i n c e i t ’ s f o c u s e d on
t h e f i r s t h a l f window = i n p u t _ s i g n a l [ c u r r e n t _ p o s : c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e ] zeropaddedwindow = pad ( window , ( l e n ( h _ f f t )−l e n (
window ) , 0 ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
x _ f f t = f f t ( zeropaddedwindow ) e l i f ( boundary ==1) : #SYMMETRIC
window = c o n c a t e n a t e ( [ f l i p u d ( i n p u t _ s i g n a l [ : h a l f _ s i z e ] ) , i n p u t _ s i g n a l [ c u r r e n t _ p o s : c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e ] ] )
x _ f f t = f f t ( window ) e l s e :
x _ f f t = f f t ( i n p u t _ s i g n a l [ : windows_s ize ] )
o u t p u t [ : windows_s ize−M]= i f f t ( x _ f f t ∗ h _ f f t ) [M−1:−1] c u r r e n t _ p o s += windows_s ize−M−h a l f _ s i z e
# a p p l y f a s t f o u r i e r t r a n s o f m t o each windows whi le ( c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e <= s i g n a l _ s i z e ) :
x _ f f t = f f t ( i n p u t _ s i g n a l [ c u r r e n t _ p o s−h a l f _ s i z e : c u r r e n t _ p o s + windows_s ize−h a l f _ s i z e ] ) o u t p u t [ c u r r e n t _ p o s : c u r r e n t _ p o s + windows_s ize−M]= r e a l ( i f f t ( x _ f f t ∗ h _ f f t ) [M−1:−1])
c u r r e n t _ p o s += windows_s ize−M
# a p p l y f a s t f o u r i e r t r a n s f o r m t o t h e r e s t o f t h e s i g n a l i f ( windows_s ize −( s i g n a l _ s i z e −c u r r e n t _ p o s + h a l f _ s i z e ) <
h a l f _ s i z e ) : window = i n p u t _ s i g n a l [ c u r r e n t _ p o s−h a l f _ s i z e : ] zeropaddedwindow = pad ( window , ( 0 , i n t ( windows_s ize −( s i g n a l _ s i z e −c u r r e n t _ p o s + h a l f _ s i z e ) ) ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
x _ f f t = f f t ( zeropaddedwindow ) o u t p u t [ c u r r e n t _ p o s : ] = r e a l ( r o l l ( i f f t ( x _ f f t ∗ h _ f f t ) ,
h a l f _ s i z e ) [ h a l f _ s i z e : h a l f _ s i z e + o u t p u t . s i z e− c u r r e n t _ p o s ] )
o u t p u t [− h a l f _ s i z e : ] = c o n v o l v e ( i n p u t _ s i g n a l [−M: ] , f i l t e r _ c o e f f i c i e n t s , ’ same ’ ) [− h a l f _ s i z e : ]
e l s e : window = i n p u t _ s i g n a l [ c u r r e n t _ p o s−h a l f _ s i z e : ] zeropaddedwindow = pad ( window , ( 0 , i n t ( windows_s ize −( s i g n a l _ s i z e −c u r r e n t _ p o s + h a l f _ s i z e ) ) ) , ’ c o n s t a n t ’ , c o n s t a n t _ v a l u e s =0)
x _ f f t = f f t ( zeropaddedwindow ) o u t p u t [ c u r r e n t _ p o s : ] = r e a l ( i f f t ( x _ f f t ∗ h _ f f t ) [M−1:M+
o u t p u t . s i z e−c u r r e n t _ p o s −1]) re turn o u t p u t
The authors advocate use of chirplets as a basis for modeling audio signals.  They introduce a fast chiplet transform for efficient computation. Also introduced is the idea of initializing (pre-training) CNN layers to mimic chirplet transform of audio signal (similar to ideas proposed by Mallet et al. on scattering transforms).  The paper is fairly easy to follow but in a few places contains undefined terms (e.g. AM-FM, MAP).
While the idea of using chirplet transform is interesting, my main concern is that the empirical evidence provided is in a rather narrow domain of bird call classification.  Furthermore, the accuracy gains shown in that domain are relatively small (61% MAP for log-Mel features vs 61.5% for chirplet transforms).  I would recommend that authors provide evidence for how this generalizes to other audio (including speech) tasks.