Understanding the communication between different animals by analysing their acoustic signals is an important topic in bioacoustics. It can be a powerful tool for the preservation of ecological diversity. We investigate probabilistic models to analyse signals issued from real-world bioacoustic sound scenes. We study a Bayesian non-parametric sequential models based on Hierarchical Dirichlet Process Hidden Markov Models (HDP-HMM). The model is able to infer hidden states, that are referred here as song units. However, using such a model raise one main issue: defining the number of hidden states the model has to learn. In bioacoustic problems we often do not know the number of song units (unlike in human speech recognition). Hence, we work with the Hierarchical Dirichlet Process (HDP)-HMM, which is a Bayesian non-parametric (BNP) model that offers a way to tackle this challenging problem. We focus our work on unsupervised learning from bioacoustic data. It consists in simultaneously finding the structure of hidden song units and automatically infer the unknown number of the hidden states to represent the data. Two real bioacoustic sound scene applications are investigated in this work: on whale and multi-species birds segmentation. The learning of these models is proceeded by using Markov-Chain Monte Carlo (MCMC) sampling techniques on Mel Frequency Cepstral Coefficients (MFCC) of audio signals. The results show an interesting song unit segmentation of the bioacoustic signals and open new insights for unsupervised analysis of such signals. This paper illustrates the potential of chunking non-human animal signals into structured parts. This can yield to a new species representation and help experts to better understand the behaviour of such species as Kershenbaum et al. (2014) wanted.
1 INTRODUCTION
Acoustic communication is common in the animal world where individuals communicate with sequences of some different acoustic elements (Kershenbaum et al., 2014). An accurate analysis is important in order to give a better identification of some animal species and interpret the identified song units in the course of time. In this paper, we automatically model the sequence of a non-human signal and determine their acoustic song units. As highlighted in Kershenbaum et al. (2014), the way according to which non-human acoustic sequences can be interpreted can be summarized as shown in Fig 4. We distinguish four common properties that are used to define potential criteria for segmenting such signals into song units. The first way, shown in Fig 4(A), consists in separating the signals using silent gaps. The second way, shown in Fig 4(B), consists in separating the signals according to the changes in the acoustic properties in the signal. The third way, shown in Fig 4(C) consists in grouping similar sounds separated with silent gaps as a single unit. The last common
way, shown in Fig 4(D) consists in separating signal in organized sound structure, considered as fundamental units.
Acoustic units can be determined either manually (e.g. from spectrogram representation), or automatically (e.g. based on a model). Manual segmentation is time consuming and not possible for a large acoustic dataset. That is why automatic approaches are needed. Furthermore, in bioacoustic signals, the problem of segmenting signals of many species, is still an issue, including for bioacoustic. Hence, a well-principled learning system based on unsupervised approach can help them to have a better understanding of bioacoustics species. In this context, we investigate statistical latent data models to automatically identify song units.
First, we study Hidden Markov Models (HMMs) (Rabiner & Juang, 1986). Which are the gold standard for sequential data, and thus could be relevant for acoustic data modeling and segmentation. The typically used algorithm to learn the model is the Expectation-Maximization (EM) algorithm (Dempster et al., 1977), also known as Baum-Welch in HMMs (Baum et al., 1970). The main issue with HMMs is the one of selecting the number of hidden states. Because of the lack of knowledge on non-human species, it is hard to have this number. This rises a model selection problem, which can be addressed by information selection criteria such as BIC, AIC (Schwarz, 1978; Akaike, 1974), which select an HMM with a number of states from pre-estimated HMMs with varying number of states.
Such approaches are limited because they require learning multiple HMMs. On the other hand, nonparametric derivations of HMMs constitute a well-principled alternative to address this issue. This approach is more flexible than using a Bayesian non-parametric (BNP) formulation for HMMs (Teh et al., 2006), also called the infinite HMM (iHMM) (Beal et al., 2002). It allows to infer the number of states (segments, units) from the data. The BNP approach for HMMs relies on Hierarchical Dirichlet Process (HDP) to define a prior over the states (Teh et al., 2006). It is known as the Hierarchical Dirichlet Process for the Hidden Markov Models (HDP-HMM) (Teh et al., 2006). The HDP-HMM parameters can be estimated by MCMC sampling techniques such as Gibbs sampling. The standard HDP-HMM Gibbs sampling has the limitation of an inadequate modeling of the temporal persistence of states (Fox et al., 2008). This problem has been addressed by Fox et al. (2008) by relying on a sticky extension which allows a more robust learning. Hence, we have a model to separate non-human signals into states that represent different activities (song units) and exploring the inference of complex data such as bioacoustic data in surroundings cases is not yet resolved.
In this paper, we investigate the BNP formulation of HMM, that is the HDP-HMM, into two challenges involving real bioacoustic data. First, a challenging problem of humpback whale song decomposition is investigated. The objective is the unsupervised structuration of whale bioacoustic data. Humpback whale songs are long cyclical sequences produced by males during the reproduction season which follows their migration from high-latitude to low-latitude waters. Singers from the same geographical region share parts of the same song. This leads to the idea of dialect (Helweg et al., 1998). Different hypotheses of these songs were emitted (Medrano et al., 1994; Frankel et al., 1995; Baker & Herman, 1984; Garland et al., 2011). Next, we investigate a challenging problem of bird song unit structuration. Catchpole & Slater (1995); Kroodsma & Miller (1996) show how birds sing and why birds have such elaborate songs. However, analysing bird song units is difficult due to the transientness of typical bird chirps, the large behavioural intra-class variability, the small amount of examples per class, the presence of wildlife noise, and so forth. As shown later in the obtained segmentation results, such automatic approaches allow large-scale analysis of environmental bioacoustics recordings
1.1 RELATED WORK
Discovering the call units (which can be considered as a kind of non-human alphabet) of such complex signals can be seen as a problem of unsupervised call units classification as Pace et al. (2010).
Picot et al. (2008) also tried to analyse bioacoustic songs using a clustering approach. They implemented a segmentation algorithm based on Payne’s principle to extract sound units from a bioacoustic song. In this paper we reformulate the problem of song decomposition as an unsupervised data classification problem. Contrary to the approach used by Pace et al. (2010), in which the number of states (call units in this case) has been fixed manually, or the one used by Picot et al. (2008), where
a K-means algorithm is used for automatic classification and then automatically define the optimal number of classes by maximizing the Davies Bouldin criterion.
Our approach is based on a probabilist approach on the MFCC; it is a non-parametric formulation, that is well-suited to the problem of automatically inferring the number of the states in the data. In the next section we describe the real-world bioacoustic challenges we used and explain our approach.
2 DATA AND METHODS
The data used represent the difficulties of bioacoustic problems, especially when the only information linked to the signal is the species name. Thus, we have to determine a sequence without ground truth.
2.1 HUMPBACK WHALE DATA
Humpback whale song data consist of a recording (about 8.6 minutes) produced at few meters from the whale in La Reunion - Indian Ocean (NIP, 2013), at a frequency sample of 44.1kHz, 32 bits, one channel.
We extract MFCC1 features from the signal, with pre-emphasis: 0.95, hamming window, FFT on 1024 points (nearly 23ms), frameshift 10 ms, 24 Mel channels, 12 MFCC coefficients plus energy and their delta and acceleration, for a total of 39 dimensions as detailed in the NIPS 2013 challenge (NIP, 2013) where the signal and the features are available. The retained data for our experiment are the 51336 first observations.
2.2 MULTI-SPECIES BIRD DATA
Bird species song data from Fernand Deroussen Jerome Sueur of Musee National d’Histoire Naturelle (F. Deroussen, 2006), consists of a training and a testing set (not used here). Theses sets were designed for the ICML4B challenge2. The recordings have a frequency sample of 44.1kHz, 16 bits, one channel. The training set is composed of 35 recordings, 30 seconds each taken from 1 microphone. Each record contains 1 bird species in the foreground for a total of 35 different birds species.
The feature extraction for this application is applied as follows. First, a high pass filter is processed to reduce the noise (set at 1.000 kHz to avoid noises). Then, we extract the MFCC features with windows of 0.06 seconds and shift of 0.03 seconds, we keep 13 coefficients, with energy as first parameter, to be compact and sufficient accurate, considering only the vocal track information and removing the source information. Also, we focus on frequencies below 8.000 kHz, because of the alterations into the spectrum. We obtain 34965 observations with 13 dimensions each for train set, that is used to learn our model.
2.3 METHOD: BIOACOUSTIC UNSUPERVISED LEARNING FOR SIGNAL REPRESENTATION
To solve bioacoustic problems and finding the number of call units we propose to use the HDPHMM model to model the complex bioacoustic data. Our approach automatically discovers and infers the number of states from the non-human song data.
In this paper we present two applications on bioacoustic data. We study the song unit structuration, for the humpback whale and for the multi-species birds signal.
In the next section we give a brief description of the Hidden Markov Model and it’s Bayesian nonparametric alternative used in our bioacoustic signal representation applications.
1The MFCC are features that represent and compress short-term power spectrum of a sound. It follows the Mel scale.
2http://sabiod.univ-tln.fr/nips4b/challenge2.html
3 BAYESIAN NON-PARAMETRIC ALTERNATIVE FOR HIDDEN MARKOV MODEL
The finite Hidden Markov Model (HMM) is very popular due to its rich mathematical structure and its stability to model sequential data (e.g. acoustic data). It assumes that the observed sequence X = (x1, . . . ,xT ) is governed by a hidden state sequence z = (z1, . . . , zT ), where xt ∈ Rd is the multidimensional observation at time t and zt represents the hidden state of xt taking values in a finite set {1, . . . ,K}, K being the possible number of states, that is unknown. The generative process of the HMM can be described in general by the following steps. First, z1 follows the initial distribution π1. Then, given the previous state (zt−1), the current state zt follows the transition distribution. Finally, given the state zt, the observation xt follows the emission distribution F (θzt) of that state. The HMM parameters, that are the initial state transition (π1), the transition matrix (π), and the emission parameters (θ) are in general estimated in a maximum likelihood estimation (MLE) framework by using the Expectation-Maximization (EM) algorithm, also known as the Bauch-Welch algorithm (Baum et al., 1970) in the context of HMMs.
Therefore, for the finite HMM, the number of states K is required to be known a priori. This model selection issue can be addressed in a two-stage scheme by using model selection criteria such as the Bayesian Information Criterion (BIC) (Schwarz, 1978), the Akaike Information Criterion (AIC) (Akaike, 1974), the Integrated Classification Likelihood criterion (ICL) (Biernacki et al., 2000), etc to select a model from pre-estimated HMMs with varying number of states. Such approaches is limited, it requires learning N HMMs, N being sufficiently high to have an equivalent of a non parametric approach. Regardless this, a non parametric approach is more efficient because it theoretically tends to an infinite number of states. Thus, we use a Bayesian non-parametric (BNP) version of the HMM, that is able to infer the number of hidden states from the data. It is more flexible than learning multiple HMM, because in bio-acoustic problems the model have to characterize multiple species/individuals, thus it possibly tends to a large number of hidden states. Thence, exploring the inference of complex data such as bioacoustic data in surroundings cases is new.
The BNP approach for the HMM, that is the infinite HMM (iHMM), is based on a Dirichlet Process (DP) (Ferguson, 1973) needs to be used. However, due to the transitions of states take independent priors, there is no coupling across transitions between different states Beal et al. (2002), therefore the DP is not sufficient to extend the HMM to an infinite model. The Hierarchical Dirichlet Process (HDP) prior distribution on the transition matrices over countability infinite state space, derived by Teh et al. (2006), extends the HMM to the infinite state space model and is briefly described in the next subsection.
3.1 HIERARCHICAL DIRICHLET PROCESS (HDP)
Suppose the data subdivided into J groups, each produced by a related, yet distinct process. The HDP extends the DP by an hierarchical Bayesian approach such that a global Dirichlet Process prior DP(α0, G0) is drawn from a global prior Gj , where G0 is itself a Dirichlet Process distribution with two parameters, a base distributionH and a concentration parameter γ. The generative process of the data with the HDP can be summarized as follows. Suppose data X, with i = 1, . . . , T observations that is grouped into j = 1, . . . , J groups. Note that the observations of the group j are given by Xj = (xj1,xj2, . . .), all observations of group j being exchangeable. Assume each observation is drawn from a mixture model, thus each observations xji is associated with a mixture component, with parameter θij . Note that from the DP property, we observe equal values in the components θij . Now, giving the model parameter θji, the data xji is drawn from the distribution F (θji). Assuming a prior distribution Gj over the model parameters associated for group j, θj = (θj1, θj2, . . .), we can define the generative process in Eq. (1).
G0|γ,H ∼ DP(γ,H), Gj |α0, G0 ∼ DP(α0, G0), ∀j ∈ 1, . . . , J,
θji|Gj ∼ Gj , ∀j ∈ 1, . . . , J and ∀i ∈ 1, . . . , T , xi|θji ∼ F (xi|θji),∀j ∈ 1, . . . , J and ∀i ∈ 1, . . . , T .
(1)
The Chinese Restaurant Process (CRP) (Pitman, 1995) is a representation of the Dirichlet Process that results from a metaphor related to the existence of a restaurant with possible infinite tables
(clusters) where customers (the observations) are sitting in it. An alternative of such a representation for the Hierarchical Dirichlet Process can be described by the Chinese Restaurant Franchise (CRF) process by extending the CRP to multiple restaurants that share a set of dishes.
The idea of CRF is that it gives a representation for the HDP by extending the Chinese Restaurant Process to a set of (J) restaurants, rather than a single restaurant. Suppose a patron of Chinese Restaurant creates many restaurants, strongly linked to each other, by a franchise wide menu, having dishes common to all restaurants. As a result, J restaurants are created (groups) with a possibility to extend each restaurant with an infinite number of tables (states) at witch the customers (observations) sit. Each customer goes to his specified restaurant j, where each table of this restaurant has a dish that shares between the customers that sit at that specific table. However, multiple tables of different existing restaurants can serve the same dish.
3.2 THE HIERARCHICAL DIRICHLET PROCESS FOR THE HIDDEN MARKOV MODEL
(HDP-HMM)
The HDP-HMM uses a HDP prior distribution providing a potential countability infinite number of hidden states and tackles the challenging problem of model selection for the HMM. This model is a Bayesian non-parametric extension for the HMM also presented as the infinite Hidden Markov Model (Beal et al., 2002). To derive the HDP-HMM model we suppose a doubly-infinite transition matrix, where each row corresponds to a CRP. Thus, in a HDP formalism, the groups correspond to states, with CRP distribution on next states. CRF links these states distributions.
We assume for simplicity a distinguished initial state z0. LetGj describes both, the transition matrix πk and the emission parameters θk, the infinite HMM can be described by the following generative process:
β|γ ∼ GEM(γ), πk|α,β ∼ DP(α,β), zt|zt−1 ∼ Mult(πzt−1), θk|H ∼ H,
xt|zt, {θk}∞k=1 ∼ F (θzt).
(2)
where, β is a hyperparameter for the DP (Sethuraman, 1994) distributed according to the stick-breaking construction noted GEM(.); zt is the indicator variable of the HDP-HMM that follows a multinomial distribution Mult(.); the emission parameters θk, are drawn independently, according to a conjugate prior distributionH; F (θzt) is a data likelihood density with the unique parameter space of θzt equal to θk.
Suppose the observed data likelihood is a Gaussian density N (xt;θk) where the emission parameters θk = {µk,Σk} are respectively the mean vector µk and the covariance matrix Σk. According to Gelman et al. (2003), the prior over the mean vector and the covariance matrix is a conjugate Normal-Inverse-Wishart distribution, denoted as NIW(µ0, κ0, ν0,Λ0), with the hyper-parameters describing the shapes and the position for each mixture components: µ0 is the mean of Gaussian should be, κ0 the number of pseudo-observations supposed to be attributed, and ν0,Λ0 being similarly for the covariance matrix.
In the generative process given in Eq. (2), π is interpreted as a double-infinite transition matrix with each row taking a CRP. Thus, in the HDP formulation ”the group-specific” distribution, πk corresponds to ”the state-specific” transition where the CRF defines distributions over the next state. In turn, Fox et al. (2008) showed that HDP-HMM inadequately models the temporal persistence of states, creating redundant and rapidly switching states and proposed an additional hyperparameter κ that increase the self-transition probabilities. This is named as sticky HDP-HMM. The distribution on the transition matrix of Eq. (2) for the sticky HDP-HMM is given as follows:
πk|α,β ∼ DP ( α+ κ,
αβ + κδk α+ κ
) , (3)
where a small positive κ > 0 is added to the kth component of αβ, thus of self-transition probability is increased by κ. Note that setting κ to 0, the original HDP-HMM is recovered. Under such assumption for the transition matrix, Fox et al. (2008) proposes an extension of the CRF to the
Chinese Restaurant Franchise with Loyal Customers. A graphical representation of (sticky) HDPHMM is given in Fig 5.
The inference of the infinite HMM (the (sticky) HDP-HMM) with the Block Gibbs sampler algorithm is given in Algorithm 3 of Supplementary Material in Fox et al. (2008) paper. The base idea of this sampler is to estimate the posterior distributions over all the parameters from the generative process of (sticky) HDP-HMM given in Eq. (2). Here, the CRF with Loyal Customers, hyperparameter κ of the transition matrix can be sampled in order to increase the self-transition probability.
Hence, the HDP-HMM model resolves the problem of advanced signal decomposition using acoustic features with respect to time. It allows identifying song units (states), behaviour and enhancing populations studies. From the other point, modelling data with the HDP-HMM offers a great alternative of the standard HMM to tackle the challenging problem of selecting the number of states, identifying the unknown number of hidden units from the used features (here: MFCC). The experimental results show the interest of a such approach.
4 EXPERIMENTS
In this section we present two applications on bioacoustic data. We study the song unit structuration, for the humpback whale signal and for multi-species birds signals.
4.1 HUMPBACK WHALE SOUND SEGMENTATION
The learning of the humpback whale song, applied via the HDP-HMM, is done with the Blocked Gibbs sampling. A number of iterations was fixed to Ns = 30000 and a truncation level, that corresponds to the maximum number of possible states in the model (being sufficient big to approximate it to an infinite model), is fixed to Lk = 30. The number of states estimated by the HDP-HMM Gibbs sampling is 6.
The Fig 6 shows the state sequences partition, for all 8.6 minutes of humpback whale song data, obtained by the HDP-HMM Gibbs sampling. For more detailed information, the result of the whole humpback whale signal segmentation is separated by several parts of 15 seconds. All the spectrograms of the humpback whale song and the obtained segmentation are made available in the demo: http://sabiod.univ-tln.fr/workspace/ICLR2017/whale/. This demo highlights the interest of using a BNP formulation of HMMs for unsupervised segmentation of whale signals. Three examples of the humpback whale song, with 15 seconds duration each, are presented and discussed in this paper (see Fig 1).
Figure 1 represents the spectrogram and the corresponding state sequence partition obtained by the HDP-HMM Gibbs inference algorithm. They respectively represent examples of the beginning, the middle and the end of the whole signal. All the obtained state sequence partitions fit the spectral patterns. We note that the estimated state 1 fits the sea noise, state 5 also fits sea noise, but it is right before units associated to whale songs. The presence of this unit can be due to an insufficient number of Gibbs samples. For a longer learning the fifth state could be merged with the first state. State 2 fits the up and down sweeps. State 3 fits low and high fundamental harmonic sounds, state 4 fits for numerous harmonics sound and state 6 fits very noisy and broad sounds. Fig 7 shows two spectrograms extracted from the 6th song unit (left) and from the 2nd song unit (right) of the whole humpback whale signal. We can see that the units fit specific patterns on the whole signal.
Pr. Gianni Pavan (Pavia University, Italy), an undersea NATO bioacoustic expert analysed the results on the humpback whale song segmentation, during his stay at DYNI in 2015. He validated the proposed representation. This highlight the interest of learning BNP model on a single species. Next, we will see how such model reacts with multiple species.
4.2 BIRDS SOUND SEGMENTATION
In this section we describe the obtained bird song unit segmentation. We segment the bird signals into song units by learning the HDP-HMM model on the training set (containing 35 different species). The main goal is to see if a such approach can model multiple species. Note that in this set, we assume there is no multiple species singing at the same time.
For this application, we considered 145000 Gibbs iterations and a truncation level of 200 for the maximum number of states. We suppose them to be sufficiently big for this data problem. Moreover, we use one mixture component per state, that appeared to give satisfactory results and we use a sticky HDP-HMM with the hyper-parameter κ set to 0.1.
We discovered 76 song units with this method. For more detailed information over the signal, we separated the whole train set into parts of 15 seconds each. All the spectrograms and the associated segmentation obtained are made available in the demo: http://sabiod.univ-tln.fr/ workspace/ICLR2017/bird/. Fig 2 contains three examples of bird bioacoustic segmentation. The species are: Carduelis chloris (left), Luscinia megarhynchos (middle) and Parus caeruleus (right).
4.2.1 EVALUATION OF THE BIRD RESULT
To evaluate the bird results, we used a ground truth produced by Simone Clemente (bird expert). We ask him to segment each recording of the dataset according to the different patterns on the signal. Then we compare this ground truth with the segments produce by the model using NMI (Strehl & Ghosh, 2002) which calculates shared information between two clustering sets.
First, we compute the NMI score between these two segmentations and we obtain a score of 0.490. Thus, the global segmentation from the model isn’t near from a segmentation done by an expert. Second, we compute the NMI score for each species to see the possible mistake done by the model. Tab 1 shows the different scores obtained with a resulting mean score of 0.367. The highest score is 0.680 (corvus corone) and the lowest score is 0.003 (garrulus glandarius). Thus, for some species, the model has difficulties to segment the data. Sometimes, it uses less states than the expert to segment the data: for the oriolus oriolus (golden oriole), the model identifies 12 song units versus 50 identified by the expert. Nevertheless, the model also uses more states than the expert to segment the data: for the fringilla coelebs (chaffinch), the model identifies 15 song units versus 3 identified
by the expert. In other cases, the model can’t differentiate 2 distinct vocalizes if they have close frequencies (phylloscopus collybita and columba palumbus), background and foreground species (streptopelia decaocto). This can be due to the feature used or an insufficient number of iterations of the Gibbs sampling. For most of species, the model and the ground truth have similar patterns observable on Fig 8.
To improve the model, we can investigate better feature representation for species with different acoustic characteristics. We can also improve noise reduction which could be useful for background activities. Nevertheless, the application highlights the interest of using BNP formulation of HMMs for unsupervised segmentation of bird signals.
5 CONCLUSIONS
In this work, we relied on real world bioacoustic applications and propose to use a BNP formulation of HMMs to realize a representation of bio-acoustic signals. It is a response for (Kershenbaum et al., 2014). We investigated this approach on real-world bioacoustic signals from two challenges. The demo for the two applications are available online.
The obtained signal segmentation on the bioacoustic data is recovered in a fully automatic way and is proceeded by a Hierarchical Dirichlet Process for Hidden Markov Model on MFCC. The BNP formulation give an estimate number of cluster needed to segment the signal and our experiments highlight the interest of such formulation on bioacoustic problems. Furthermore we compare the segmentation obtained for birds with the segmentation from an expert and the model using NMI, the results are promising. We describe a full bioacoustic perspective in the annexes.
However, the model used is computational expensive and not suitable larger dataset. Future work will consist in studying methods that should accelerate the MCMC sampling and dealing with larger data problems, like variational inference (Jordan et al., 1999) or stochastic variational inference used for HMMs (Foti et al., 2014).
Future work will consist in considering our segmentation results for classification task, where the goal is to identify the species.
ACKNOWLEDGEMENTS
We would like to thanks Pr. Gianni Pavan (Pavia University, Italy) and Simone Clemente for their bioacoustic points of view. We also want to thanks Virgil Tassan for his re-reading.
BIOACOUSTICIAN DISCUSSION
One of the main topics in ecological acoustics is the development of unsupervised methods for automatic detection of vocalized species, which would help specialists in ecological works during their monitoring activities. Although some works already have reach good classification percentage 3 4, there’s a lack of methodologies available for works focused on real world data, and with further applications in ecology and wildlife management. One of the major bottlenecks for the application of these methodologies is their inability to work under heavy complex acoustic environment, where different taxa may sing together or conversely, their extreme sensitivity which may result in an ”over classification” due to the high degree of variability insight many repertoire of the vocal species.
Our unsupervised method for automatic annotation of bioacoustics sequences seems to overtake these obstacles, by the identification of specie-specific pattern, and seems to be not influenced by the inter-individual variation whiting the song’s structure. Furthermore, during the study, we have worked whit recording that may contain more than a species vocalizing, or partial overlapping from different species and specimen, and even in these circumstance our model has shown a great ability of categorisations and generalisations, identifying the main pattern and almost all the other sequences recorded (including silence).
Good examples are provided, from the birds’ dataset, by recordings including the common wood pigeon (Columba palumbus; Figure 3 (top)), or from the files containing golden oriole (Oriolus oriolus) as a main species. Their vocalisations are partially overlapped by vocalisation of different species, such as Eurasian blue tit (Cyanistes caeruleus, figure 3 (bottom)) or cricket, and is clear in the analysis, the distinction between the bird vocalising and other animals in the background. Finally, a last example could be taken from the carrion crow (Corvus corone, figure 3 (middle) recordings. Here, multiple specimens are vocalising at the same time, but it does not influence the efficiency of our method.
By the other hand, in presence of species with complex acoustics behaviours, such as the common nightingale (Luscinia megarhynchos) the method could lead to identify as different classes each one of the different phrasing elements in the song. This last approach could be useful in order to perform behavioural analysis focused on the identification of hidden significance in the songs, but may loose his benefit when the main achievement is an ecological topic. We finally consider this method as a promising tool, as we move forward, but further analysis will be needed to build, an efficient tool whit relevant application in the acoustic monitoring and conservation of biodiversity.
3Automatic large-scale classification of bird sounds is strongly improved by unsupervised feature learning; Dan Stowell and Mark D Plumbley; 2014; PeerJ
4Automatic classification of a taxon-rich community recorded in the wild; Potamitis and Ilyas; 2014; PloS on

The paper is a novel application for the sticky HDP-HMM, focused on correctly identifying the number of components in bird and whale song across a variety of datasets. It's nice to see the model applied to an interesting dataset. My main issues with the paper have to do with structure and the choice of representation used in the model. Namely:
The organization of the paper could be significantly improved. There is a lot of repetitive introduction that adds little to the paper. The first and last two sentences of the abstract could be cut. Many other parts of the abstract basically repeat the introduction. The second paragraph of section 2.3 also repeats your introduction - by now we know what you're doing. I think most people reading this will have no idea what Kershenbaum (2014) is. The description of the data should go in the experiments section. "Different hypotheses for the songs were emitted" in the introduction is odd phrasing. Figure 4 should be the first figure and go in the introduction. Figure 5 should be in the methods section. A summary of Table 1 should be in the experiments section. Generally the writing could be tightened quite a bit, which would make space for these figures. The description of the HDP-HMM, which mostly follows the existing literature, is well done.
Some general questions about the methods used:
If you're interested in scalable inference, why use Gibbs sampling? Why not the beam sampler (van Gael 2008), which at least recently was the state of the art for MCMC inference in the HDP-HMM? More generally, why use MCMC at all? For very large datasets, most of the Bayesian ML community has converged on stochastic variational inference as the most practical method (eg Wang, Paisley and Blei 2011).
If your interest is mainly in the number of clusters, how would you address the fact that DP mixture models are known not to be consistent for estimating the true number of clusters (Miller and Harrison 2013)?
MFCC features are calibrated to the human auditory system, not bird or whale auditory systems. In your data, do you calibrate the MFCC scale to be closer to the auditory systems of the animals that generated the song?
And a final suggestion for future work, which could use the results presented here as a baseline:
Given the success of LSTMs in speech recognition in recent years, it may be the case that deep learned representations are superior to linear features (like the means of each cluster in an HDP-HMM) for animal song as well. Have you considered a hybrid model, similar to recent work combining autoencoders and graphical models (Johnson, Duvenaud, Wiltschko, Datta and Adams 2016)?


EXTRACTIVE QUESTION ANSWERING
The reading comprehension task, that asks questions about a given evidence document, is a central problem in natural language understanding. Recent formulations of this task have typically focused on answer selection from a set of candidates pre-defined manually or through the use of an external NLP pipeline. However, Rajpurkar et al. (2016) recently released the SQUAD dataset in which the answers can be arbitrary strings from the supplied text. In this paper, we focus on this answer extraction task, presenting a novel model architecture that efficiently builds fixed length representations of all spans in the evidence document with a recurrent network. We show that scoring explicit span representations significantly improves performance over other approaches that factor the prediction into separate predictions about words or start and end markers. Our approach improves upon the best published results of Wang & Jiang (2016) by 5% and decreases the error of Rajpurkar et al.’s baseline by > 50%.
1 INTRODUCTION
A primary goal of natural language processing is to develop systems that can answer questions about the contents of documents. The reading comprehension task is of practical interest – we want computers to be able to read the world’s text and then answer our questions – and, since we believe it requires deep language understanding, it has also become a flagship task in NLP research.
A number of reading comprehension datasets have been developed that focus on answer selection from a small set of alternatives defined by annotators (Richardson et al., 2013) or existing NLP pipelines that cannot be trained end-to-end (Hill et al., 2016; Hermann et al., 2015). Subsequently, the models proposed for this task have tended to make use of the limited set of candidates, basing their predictions on mention-level attention weights (Hermann et al., 2015), or centering classifiers (Chen et al., 2016), or network memories (Hill et al., 2016) on candidate locations.
Recently, Rajpurkar et al. (2016) released the less restricted SQUAD dataset1 that does not place any constraints on the set of allowed answers, other than that they should be drawn from the evidence document. Rajpurkar et al. proposed a baseline system that chooses answers from the constituents identified by an existing syntactic parser. This allows them to prune the O(N2) answer candidates in each document of length N , but it also effectively renders 20.7% of all questions unanswerable.
Subsequent work by Wang & Jiang (2016) significantly improve upon this baseline by using an endto-end neural network architecture to identify answer spans by labeling either individual words, or the start and end of the answer span. Both of these methods do not make independence assumptions about substructures, but they are susceptible to search errors due to greedy training and decoding.
In contrast, here we argue that it is beneficial to simplify the decoding procedure by enumerating all possible answer spans. By explicitly representing each answer span, our model can be globally normalized during training and decoded exactly during evaluation. A naive approach to building the O(N2) spans of up to length N would require a network that is cubic in size with respect
∗Work completed during internship at Google, New York. 1http://stanford-qa.com
to the passage length, and such a network would be untrainable. To overcome this, we present a novel neural architecture called RASOR that builds fixed-length span representations, reusing recurrent computations for shared substructures. We demonstrate that directly classifying each of the competing spans, and training with global normalization over all possible spans, leads to a significant increase in performance. In our experiments, we show an increase in performance over Wang & Jiang (2016) of 5% in terms of exact match to a reference answer, and 3.6% in terms of predicted answer F1 with respect to the reference. On both of these metrics, we close the gap between Rajpurkar et al.’s baseline and the human-performance upper-bound by > 50%.
2 EXTRACTIVE QUESTION ANSWERING
2.1 TASK DEFINITION
Extractive question answering systems take as input a question q = {q0, . . . , qn} and a passage of text p = {p0, . . . , pm} from which they predict a single answer span a = 〈astart , aend〉, represented as a pair of indices into p. Machine learned extractive question answering systems, such as the one presented here, learn a predictor function f(q,p)→ a from a training dataset of 〈q,p,a〉 triples.
2.2 RELATED WORK
For the SQUAD dataset, the original paper from Rajpurkar et al. (2016) implemented a linear model with sparse features based on n-grams and part-of-speech tags present in the question and the candidate answer. Other than lexical features, they also used syntactic information in the form of dependency paths to extract more general features. They set a strong baseline for following work and also presented an in depth analysis, showing that lexical and syntactic features contribute most strongly to their model’s performance. Subsequent work by Wang & Jiang (2016) use an end-to-end neural network method that uses a Match-LSTM to model the question and the passage, and uses pointer networks (Vinyals et al., 2015) to extract the answer span from the passage. This model resorts to greedy decoding and falls short in terms of performance compared to our model (see Section 5 for more detail). While we only compare to published baselines, there are other unpublished competitive systems on the SQUAD leaderboard, as listed in footnote 5.
A task that is closely related to extractive question answering is the Cloze task (Taylor, 1953), in which the goal is to predict a concealed span from a declarative sentence given a passage of supporting text. Recently, Hermann et al. (2015) presented a Cloze dataset in which the task is to predict the correct entity in an incomplete sentence given an abstractive summary of a news article. Hermann et al. also present various neural architectures to solve the problem. Although this dataset is large and varied in domain, recent analysis by Chen et al. (2016) shows that simple models can achieve close to the human upper bound. As noted by the authors of the SQUAD paper, the annotated answers in the SQUAD dataset are often spans that include non-entities and can be longer phrases, unlike the Cloze datasets, thus making the task more challenging.
Another, more traditional line of work has focused on extractive question answering on sentences, where the task is to extract a sentence from a document, given a question. Relevant datasets include datasets from the annual TREC evaluations (Voorhees & Tice, 2000) and WikiQA (Yang et al., 2015), where the latter dataset specifically focused on Wikipedia passages. There has been a line of interesting recent publications using neural architectures, focused on this variety of extractive question answering (Tymoshenko et al., 2016; Wang et al., 2016, inter alia). These methods model the question and a candidate answer sentence, but do not focus on possible candidate answer spans that may contain the answer to the given question. In this work, we focus on the more challenging problem of extracting the precise answer span.
3 MODEL
We propose a model architecture called RASOR2 illustrated in Figure 1, that explicitly computes embedding representations for candidate answer spans. In most structured prediction problems (e.g. sequence labeling or parsing), the number of possible output structures is exponential in the input
2An abbreviation for Recurrent Span Representations, pronounced as razor.
length, and computing representations for every candidate is prohibitively expensive. However, we exploit the simplicity of our task, where we can trivially and tractably enumerate all candidates. This facilitates an expressive model that computes joint representations of every answer span, that can be globally normalized during learning.
In order to compute these span representations, we must aggregate information from the passage and the question for every answer candidate. For the example in Figure 1, RASOR computes an embedding for the candidate answer spans: fixed to, fixed to the, to the, etc. A naive approach for these aggregations would require a network that is cubic in size with respect to the passage length. Instead, our model reduces this to a quadratic size by reusing recurrent computations for shared substructures (i.e. common passage words) from different spans.
Since the choice of answer span depends on the original question, we must incorporate this information into the computation of the span representation. We model this by augmenting the passage word embeddings with additional embedding representations of the question.
In this section, we motivate and describe the architecture for RASOR in a top-down manner.
3.1 SCORING ANSWER SPANS
The goal of our extractive question answering system is to predict the single best answer span among all candidates from the passage p, denoted as A(p). Therefore, we define a probability distribution over all possible answer spans given the question q and passage p, and the predictor function finds the answer span with the maximum likelihood:
f(q,p) := argmax a∈A(p)
P (a | q,p) (1)
One might be tempted to introduce independence assumptions that would enable cheaper decoding. For example, this distribution can be modeled as (1) a product of conditionally independent distributions (binary) for every word or (2) a product of conditionally independent distributions (over words) for the start and end indices of the answer span. However, we show in Section 5.2 that such independence assumptions hurt the accuracy of the model, and instead we only assume a fixed-length representation ha of each candidate span that is scored and normalized with a softmax layer (Span score and Softmax in Figure 1):
sa = wa · FFNN(ha) a ∈ A(p) (2)
P (a | q,p) = exp(sa)∑ a′∈A(p) exp(sa′)
a ∈ A(p) (3)
where FFNN(·) denotes a fully connected feed-forward neural network that provides a non-linear mapping of its input embedding, and wa denotes a learned vector for scoring the last layer of the feed-forward neural network.
3.2 RASOR: RECURRENT SPAN REPRESENTATION
The previously defined probability distribution depends on the answer span representations, ha. When computing ha, we assume access to representations of individual passage words that have been augmented with a representation of the question. We denote these question-focused passage word embeddings as {p∗1, . . . , p∗m} and describe their creation in Section 3.3. In order to reuse computation for shared substructures, we use a bidirectional LSTM (Hochreiter & Schmidhuber, 1997) to encode the left and right context of every p∗i (Passage-level BiLSTM in Figure 1). This allows us to simply concatenate the bidirectional LSTM (BiLSTM) outputs at the endpoints of a span to jointly encode its inside and outside information (Span embedding in Figure 1):
{p∗′1 , . . . , p∗′m} = BILSTM({p∗1, . . . , p∗m}) (4)
ha = [p ∗′ astart , p ∗′ aend
] 〈astart , aend〉 ∈ A(p) (5) where BILSTM(·) denotes a BiLSTM over its input embedding sequence and p∗′i is the concatenation of forward and backward outputs at time-step i. While the visualization in Figure 1 shows a single layer BiLSTM for simplicity, we use a multi-layer BiLSTM in our experiments. The concatenated output of each layer is used as input for the subsequent layer, allowing the upper layers to depend on the entire passage.
3.3 QUESTION-FOCUSED PASSAGE WORD EMBEDDING
Computing the question-focused passage word embeddings {p∗1, . . . , p∗m} requires integrating question information into the passage. The architecture for this integration is flexible and likely depends on the nature of the dataset. For the SQUAD dataset, we find that both passage-aligned and passageindependent question representations are effective at incorporating this contextual information, and experiments will show that their benefits are complementary. To incorporate these question representations, we simply concatenate them with the passage word embeddings (Question-focused passage word embedding in Figure 1).
We use fixed pretrained embeddings to represent question and passage words. Therefore, in the following discussion, notation for the words are interchangeable with their embedding representations.
Question-independent passage word embedding The first component simply looks up the pretrained word embedding for the passage word, pi.
Passage-aligned question representation In this dataset, the question-passage pairs often contain large lexical overlap or similarity near the correct answer span. To encourage the model to exploit these similarities, we include a fixed-length representation of the question based on soft alignments with the passage word. The alignments are computed via neural attention (Bahdanau et al., 2014), and we use the variant proposed by Parikh et al. (2016), where attention scores are dot products between non-linear mappings of word embeddings.
sij = FFNN(pi) · FFNN(qj) 1 ≤ j ≤ n (6) aij = exp(sij)∑n k=1 exp(sik) 1 ≤ j ≤ n (7)
qaligni = n∑ j=1 aijqj (8)
Passage-independent question representation We also include a representation of the question that does not depend on the passage and is shared for all passage words.
Similar to the previous question representation, an attention score is computed via a dot-product, except the question word is compared to a universal learned embedding rather any particular passage word. Additionally, we incorporate contextual information with a BiLSTM before aggregating the outputs using this attention mechanism.
The goal is to generate a coarse-grained summary of the question that depends on word order. Formally, the passage-independent question representation qindep is computed as follows:
{q′1, . . . , q′n} = BILSTM(q) (9) sj = wq · FFNN(q′j) 1 ≤ j ≤ n (10)
aj = exp(sj)∑n k=1 exp(sk)
1 ≤ j ≤ n (11)
qindep = n∑ j=1 ajq ′ j (12)
where wq denotes a learned vector for scoring the last layer of the feed-forward neural network.
This representation is a bidirectional generalization of the question representation recently proposed by Li et al. (2016) for a different question-answering task.
Given the above three components, the complete question-focused passage word embedding for pi is their concatenation: p∗i = [pi, q align i , q indep].
3.4 LEARNING
Given the above model specification, learning is straightforward. We simply maximize the loglikelihood of the correct answer candidates and backpropagate the errors end-to-end.
4 EXPERIMENTAL SETUP
We represent each of the words in the question and document using 300 dimensional GloVe embeddings trained on a corpus of 840bn words (Pennington et al., 2014). These embeddings cover 200k words and all out of vocabulary (OOV) words are projected onto one of 1m randomly initialized 300d embeddings. We couple the input and forget gates in our LSTMs, as described in Greff et al. (2016), and we use a single dropout mask to apply dropout across all LSTM time-steps as proposed by Gal & Ghahramani (2016). Hidden layers in the feed-forward neural networks use rectified linear units (Nair & Hinton, 2010). Answer candidates are limited to spans with at most 30 words.
To choose the final model configuration, we ran grid searches over: the dimensionality of the LSTM hidden states (25, 50, 100, 200); the number of stacked LSTM layers (1, 2, 3); the width (50, 100, 150, 200) and depth (1, 2) of the feed-forward neural networks; the dropout rate (0, 0.1, 0.2); and the decay multiplier (0.9, 0.95, 1.0) with which we multiply the learning rate every 10k steps. The best model uses a single 150d hidden layer in all feed-forward neural networks; 50d LSTM states; two-layer BiLSTMs for the span encoder and the passage-independent question representation; dropout of 0.1 throughout; and a learning rate decay of 5% every 10k steps.
All models are implemented using TensorFlow3 and trained on the SQUAD training set using the ADAM (Kingma & Ba, 2015) optimizer with a mini-batch size of 4 and trained using 10 asynchronous training threads on a single machine.
3www.tensorflow.org
5 RESULTS
We train on the 80k (question, passage, answer span) triples in the SQUAD training set and report results on the 10k examples in the SQUAD development set. Due to copyright restrictions, we are currently not able to upload our models to Codalab4, which is required to run on the blind SQUAD test set, but we are working with Rajpurkar et al. to remedy this, and this paper will be updated with test numbers as soon as possible.
All results are calculated using the official SQUAD evaluation script, which reports exact answer match and F1 overlap of the unigrams between the predicted answer and the closest labeled answer from the 3 reference answers given in the SQUAD development set.
5.1 COMPARISONS TO OTHER WORK
Our model with recurrent span representations (RASOR) is compared to all previously published systems 5. Rajpurkar et al. (2016) published a logistic regression baseline as well as human performance on the SQUAD task. The logistic regression baseline uses the output of an existing syntactic parser both as a constraint on the set of allowed answer spans, and as a method of creating sparse features for an answer-centric scoring model. Despite not having access to any external representation of linguistic structure, RASOR achieves an error reduction of more than 50% over this baseline, both in terms of exact match and F1, relative to the human performance upper bound.
More closely related to RASOR is the boundary model with Match-LSTMs and Pointer Networks by Wang & Jiang (2016). Their model similarly uses recurrent networks to learn embeddings of each passage word in the context of the question, and it can also capture interactions between endpoints, since the end index probability distribution is conditioned on the start index. However, both training and evaluation are greedy, making their system susceptible to search errors when decoding. In contrast, RASOR can efficiently and explicitly model the quadratic number of possible answers, which leads to a 14% error reduction over the best performing Match-LSTM model.
We also ensemble RASOR with a baseline model described in Section 5.2 that independently predicts endpoints rather than spans (Endpoints prediction in Table 2b). By simply computing the product of the output probabilities, this ensemble further increases performance to 68.2% exact-match. We examine the causes of this improvement in Section 6.
Since we do not have access to the test set, we also present 5-fold cross validation experiments to demonstrate that our dev-set results are not an outcome of overfitting through hyper-parameter selection. In this 5-fold setting, we create 5 pseudo dev/test splits from the SQUAD development set.6 We choose hyper-parameters on the basis of the pseudo dev set, and report performance on the disjoint pseudo test set. Each of the pseudo dev sets led us to choose the same optimal model
4http://stanford-qa.com 5As of submission, other unpublished systems are shown on the SQUAD leaderboard, including MatchLSTM with Ans-Ptr (Boundary+Ensemble), Co-attention, r-net, Match-LSTM with Bi-Ans-Ptr (Boundary), Coattention old, Dynamic Chunk Reader, Dynamic Chunk Ranker with Convolution layer, Attentive Chunker.
6We split by Wikipedia page ID and use 4 5 as a development set and 1 5 as a test set.
hyper-parameters from a grid of 59 settings, as well as very similar training stopping points. We compute the mean and standard deviation of both evaluation metrics for these optimal models on the pseudo test set, resulting in a 66.0± 1.0 exact match and 74.5± 0.9 F1. These results show that our hyper-parameter selection procedure is not overfitting on the 10k SQUAD development set, and we subsequently expect that our model’s performance will translate to the SQUAD test set.
5.2 MODEL VARIATIONS
We investigate two main questions in the following ablations and comparisons. (1) How important are the two methods of representing the question described in Section 3.3? (2) What is the impact of learning a loss function that accurately reflects the span prediction task?
Question representations Table 2a shows the performance of RASOR when either of the two question representations described in Section 3.3 is removed. The passage-aligned question representation is crucial, since lexically similar regions of the passage provide strong signal for relevant answer spans. If the question is only integrated through the inclusion of a passage-independent representation, performance drops drastically. The passage-independent question representation over the BiLSTM is less important, but it still accounts for over 3% exact match and F1. The input of both of these components is analyzed qualitatively in Section 6.
Learning objectives Given a fixed architecture that is capable of encoding the input questionpassage pairs, there are many ways of setting up a learning objective to encourage the model to predict the correct span. In Table 2b, we provide comparisons of some alternatives (learned end-toend) given only the passage-level BiLSTM from RASOR. In order to provide clean comparisons, we restrict the alternatives to objectives that are trained and evaluated with exact decoding.
The simplest alternative is to consider this task as binary classification for every word (Membership prediction in Table 2b). In this baseline, we optimize the logistic loss for binary labels indicating whether passage words belong to the correct answer span. At prediction time, a valid span can be recovered in linear time by finding the maximum contiguous sum of scores.
Li et al. (2016) proposed a sequence-labeling scheme that is similar to the above baseline (BIO sequence prediction in Table 2b). We follow their proposed model and learn a conditional random field (CRF) layer after the passage-level BiLSTM to model transitions between the different labels. At prediction time, a valid span can be recovered in linear time using Viterbi decoding, with hard transition constraints to enforce a single contiguous output.
We also consider a model that independently predicts the two endpoints of the answer span (Endpoints prediction in Table 2b). This model uses the softmax loss over passage words during learning. When decoding, we only need to enforce the constraint that the start index is no greater than the end index. Without the interactions between the endpoints, this can be computed in linear time. Note that this model has the same expressivity as RASOR if the span-level FFNN were removed.
Lastly, we compare with a model using the same architecture as RASOR but is trained with a binary logistic loss rather than a softmax loss over spans (Span prediction w/ logistic loss in Table 2b).
The trend in Table 2b shows that the model is better at leveraging the supervision as the learning objective more accurately reflects the fundamental task at hand: determining the best answer span.
First, we observe general improvements when using labels that closely align with the task. For example, the labels for membership prediction simply happens to provide single contiguous spans in the supervision. The model must consider far more possible answers than it needs to (the power set of all words). The same problem holds for BIO sequence prediction– the model must do additional work to learn the semantics of the BIO tags. On the other hand, in RASOR, the semantics of an answer span is naturally encoded by the set of labels.
Second, we observe the importance of allowing interactions between the endpoints using the spanlevel FFNN. RASOR outperforms the endpoint prediction model by 1.1 in exact match, The interaction between endpoints enables RASOR to enforce consistency across its two substructures. While this does not provide improvements for predicting the correct region of the answer (captured by the F1 metric, which drops by 0.2), it is more likely to predict a clean answer span that matches human judgment exactly (captured by the exact-match metric).
6 ANALYSIS
Figure 2 shows how the performances of RASOR and the endpoint predictor introduced in Section 5.2 degrade as the lengths of their predictions increase. The endpoint predictor underpredicts single word answer spans, while overpredicting answer spans with more than 8 words.
Since the endpoints predictor does not explicitly model the interaction between the start and end of any given answer span, it is susceptible to choosing the span start and end points from separate answer candidates. For example, consider the following endpoints prediction that is most different in length from a correct span classification. Here, the span classifier correctly answers the question ‘Where did the Meuse flow before the flood?’ with ‘North Sea’ but the endpoints prediction is:
‘south of today’s line Merwede-Oude Maas to the North Sea and formed an archipelago-like estuary with Waal and Lek. This system of numerous bays, estuary-like extended rivers, many islands and constant changes of the coastline, is hard to imagine today. From 1421 to 1904, the Meuse and Waal merged further upstream at Gorinchem to form Merwede. For flood protection reasons, the Meuse was separated from the Waal through a lock and diverted into a new outlet called ”Bergse Maas”, then Amer and then flows into the former bay Hollands Diep’
In this prediction, we can see that both the start ‘south of . . . ’ and the end ‘. . . Hollands Diep’ have a reasonable answer type. However, the endpoints predictor has failed to model the fact that they cannot resonably be part of the same answer, a common error case. The endpoints predictor predicts 514 answers with > 25 more words than the gold answer, but the span classifier never does this.
Figure 3 shows attention masks for both of RASOR’s question representations. The passageindependent question representation pays most attention to the words that could attach to the answer in the passage (‘brought’, ‘against’) or describe the answer category (‘people’). Meanwhile, the passage-aligned question representation pays attention to similar words. The top predictions for both examples are all valid syntactic constituents, and they all have the correct semantic category. However, RASOR assigns almost as much probability mass to it’s incorrect third prediction ‘British’ as it does to the top scoring correct prediction ‘Egyptian’. This showcases a common failure case for RASOR, where it can find an answer of the correct type close to a phrase that overlaps with the question – but it cannot accurately represent the semantic dependency on that phrase.
A significant architectural difference from other neural models for the SQUAD dataset, such as Wang & Jiang (2016), is the use of the question-independent passage representation (Equation 12). Table 3 shows examples in the development set where the model paid the most attention to a single word in the question. The attention mechanism tends to seek words in the question that indicate the answer type, e.g. ‘language’ from the question: ‘What language did the Court of Justic accept ...’ This pattern provides insight for the necessity of using both question representations, since the answer type information is orthogonal to passage alignment information.
7 CONCLUSION
We have shown a novel approach for perform extractive question answering on the SQUAD dataset by explicitly representing and scoring answer span candidates. The core of our model relies on a recurrent network that enables shared computation for the shared substructure across span candidates. We explore different methods of encoding the passage and question, showing the benefits of including both passage-independent and passage-aligned question representations. While we show that this encoding method is beneficial for the task, this is orthogonal to the core contribution of efficiently computing span representation. In future work, we plan to explore alternate architectures that provide input to the recurrent span representations.

The authors proposed RASOR to address the problem of finding the best answer span according to a given question. The focus of the paper is mainly on how to model the relationship between question and the answer spans. The idea proposed by this paper is reasonable, but not ground breaking. The analysis is interesting and potentially useful. I would hope the authors can go extra miles to analyze different choices of boundary prediction models and make a more convincing case for the necessity of modeling the score of the span globally.
The main idea behind RASOR is to globally normalize and rank the scores of the possible answer spans. RASOR is able to achieve this by first modeling the hidden vectors of all words with LSTMs. Then, the representation of a text span is formed by concatenating the corresponding hidden vectors of the start and the end word of the corresponding chunk. The approach is reasonable, but not earth shattering. Also, the table 6 shows that the improvement over end-prediction point is not very large.
I appreciate the fact that authors conduct several analysis experiments as some of them are quite interesting. For example, it seems that question independent representation is also very import to the performance. In addition to the current analysis, I also want to get a clear idea on what makes the current model be better than the Match-LSTM. Is it hyper-parameter tuning? Or it is due to the use of the question independent representation?
Another good thing about the proposed model is that it is relatively simple, so there is a chance that the proposed techniques can be combined with other newly proposed ones.


UNDERSTANDING INTERMEDIATE LAYERS USING LINEAR CLASSIFIER PROBES
Neural network models have a reputation for being black boxes. We propose a new method to better understand the roles and dynamics of the intermediate layers. This has direct consequences on the design of such models and it enables the expert to be able to justify certain heuristics (such as adding auxiliary losses in middle layers). Our method uses linear classifiers, referred to as “probes”, where a probe can only use the hidden units of a given intermediate layer as discriminating features. Moreover, these probes cannot affect the training phase of a model, and they are generally added after training. They allow the user to visualize the state of the model at multiple steps of training. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems.
1 INTRODUCTION
The recent history of deep neural networks features an impressive number of new methods and technological improvements to allow the training of deeper and more powerful networks.
Despite this, models still have a reputation for being black boxes. Neural networks are criticized for their lack of interpretability, which is a tradeoff that we accept because of their amazing performance on many tasks. Efforts have been made to identify the role played by each layer, but it can be hard to find a meaning to individual layers.
There are good arguments to support the claim that the first layers of a convolution network for image recognition contain filters that are relatively “general”, in the sense that they would work great even if we switched to an entirely different dataset of images. The last layers are specific to the dataset being used, and have to be retrained when using a different dataset. In Yosinski et al. (2014) the authors try to pinpoint the layer at which this transition occurs, but they show that the exact transition is spread across multiple layers.
In this paper, we introduce the concept of linear classifier probe, referred to as a “probe” for short when the context is clear. We start from the concept of Shannon entropy, which is the classic way to describe the information contents of a random variable. We then seek to apply that concept to understand the roles of the intermediate layers of a neural network, to measure how much information is gained at every layer (answer : technically, none). We argue that it fails to apply, and so we propose an alternative framework to ask the same question again. This time around, we ask what would be the performance of an optimal linear classifier if it was trained on the inputs of a given layer from our model. We demonstrate how this powerful concept can be very useful to understand the dynamics involved in a deep neural network during training and after.
2 INFORMATION THEORY
It was a great discovery when Claude Shannon repurposed the notion of entropy to represent information contents in a formal way. It laid the foundations for the discipline of information theory. We would refer the reader to first chapters of MacKay (2003) for a good exposition on the matter.
Naturally, we would like to ask some questions about the information contents of the many layers of convolutional neural networks.
• What happens when we add more layers? • Where does information flow in a neural network with multiple branches? • Does having multiple auxiliary losses help? (e.g. Inception model)
Intuitively, for a training sample xi with its associated label yi, a deep model is getting closer to the correct answer in the higher layers. It starts with the difficult job of classifying xi, which becomes easier as the higher layers distill xi into a representation that is easier to classify. One might be tempted to say that this means that the higher layers have more information about the ground truth, but this would be incorrect.
Here there is a mismatch between two different concepts of information. The notion of entropy fails to capture the essence of those questions. This is illustrated in a formal way by the Data Processing Inequality. It states that, for a set of three random variables satisfying the dependency
X → Y → Z
then we have that I(X;Z) ≤ I(X;Y )
where I(X,Y ) is the mutual information.
Intuitively, this means that the deterministic transformations performed by the many layers of a deep neural network are not adding more information. In the best case, they preserve information and affect only the representation. But in almost all situations, they lose some information in the process.
If we distill this further, we can think of the serious mismatch between the two following ideas :
• Part of the genius of the notion of entropy is that is distills the essence of information to a quantity that does not depend on the particular representation.
• A deep neural network is a series of simple deterministic transformations that affect the representation so that the final layer can be fed to a linear classifier.
The former ignores the representation of data, while the latter is an expert in finding good representations. A deaf painter is working on a visual masterpiece to offer to a blind musician who plays music for him.
We need a conceptual tool to analyze neural networks in a way that corresponds better to our intuitive notion of information. The role of data representation is important, but we would also argue that we have to think about this issue as it relates to computational complexity. A linear classifier is basically the simplest form of classifier that is neither trivial nor degenerate.
We define a new notion of information that depends on our ability to classify features of a given layer with an optimal linear classifier. Then we have a conceptual tool to ask new questions and to get potentially interesting answers.
We end this section with a conceptual example in Figure 1. If X contains an image of the savannah, and Y ∈ {0, 1} refers to whether it contains a lion or not, then none of the subsequent layers are truly more informative than X itself. The raw bits from the picture file contain everything.
3 LINEAR CLASSIFIER PROBES
In section 3.1 we present the main concept of this paper. We illustrate the concept in section 3.3. We then present a basic experiment in section 3.4. In section 3.6 we modify a very deep network in two different ways and we show how probes allow us to visualize the consequences (sometimes disastrous) of our design choices.
3.1 PROBES
As we discussed the previous section, there is indeed a good reason to use many deterministic layers, and it is because they perform useful transformations to the data with the goal of ultimately fitting a linear classifier at the very end. That is the purpose of the many layers. They are a tool to transform data into a form to be fed to a boring linear classifier.
With this in mind, it is natural to ask if that transformation is sudden or progressive, and whether the intermediate layers already have a representation that is immediately useful to a linear classifier. We refer the reader to Figure 2 for a diagram of probes being inserted in the usual deep neural network.
The conceptual framework that we propose is one where the intuitive notion of information is equivalent with immediate suitability for a linear classifier (instead of being related to entropy).
Just to be absolutely clear about what we call a linear classifier, we mean a function
f : H → [0, 1]D
h 7→ softmax (Wh+ b) .
where h ∈ H are the features of some hidden layer, [0, 1]D is the space of one-hot encodings of the D target classes, and (W, b) are the probe weights and biases to be learned so as to minimize the usual cross-entropy loss.
Over the course of training a model, the parameters of the model change. However, probes only make sense when we refer to a given training step. We can talk about the probes at iteration n of training, when the model parameters are θn. These parameters are not affected by the probes. We prevent backpropagation through the model either by stopping the gradient flow (done with tf.stop gradient in tensorflow), or simply by specifying that the only variables to be updated are the probe parameters, while we keep θn frozen.
3.1.1 TRAINING THE PROBES
For the purposes of this paper, we train the probes up to convergence with fixed model parameters, and we report the prediction error on the training set.
It is absolutely possible to train the probes simulatenously while training the model itself. This is a good approach if we consider about how long it can take to train the model. However, this creates a potential problem if we optimize the loss of the model more quickly than the loss of the probes. This can present a skewed view of the actual situation that we would have if we trained the probes until convergence before updating the model parameters. If we accept this trade off, then we can train the probes at the same time as the model.
In some situations, the probes might overfit the training set, so we may want to do early stopping on the validation set and report the performance for the probes on the test set. This is what we do in section 3.4 with the simple MNIST convnet.
We are still unsure if one of those variations should be preferred in general, and right now they all seem acceptable so long as we interpret the probe measurements properly.
Note that training those probes represents a convex optimization problem. In practice, this does mean guarantee that they are easy to train. However, it is reassuring because it means that probes taken at time θn can be used as initialization for probes at time θn+1.
We use cross-entropy as probe loss because all models studied here used cross-entropy. Other alternative losses could be justified in other settings.
3.2 PROBES ON BIFURCATING TOY MODEL
Here we show a hypothetical example in which a model contains a bifurcation with two paths that later recombine. We are interested in knowing whether those two branches are useful, or whether one is potentially redundant or useless.
X
con cat
concat
probe prediction error0.75
0.60
0.45
Ŷ
For example, the two different branches might contain convolutional layers with different dimensions. They may have a different number of sublayers, or one might represent a skip connection. We assume that the branches are combined through concatenation of their features, so that nothing is lost.
For this hypothetical situation, we indicate the probe prediction errors on the graphical model. The upper path has a prediction error of 0.75, the lower path has 0.60, and their combination has 0.45. Small errors are preferred. Although the upper path has “less information” than the lower path, we can see here that it is not redundant information, because when we concatenate the features of the two branches we get a prediction error of 0.45 < 0.60.
If the concatenated layer had a prediction error of 0.60 instead of 0.45, then we could declare that the above branch did nothing useful. It may have nonzero weights, but it’s still useless.
Naturally, this kind of conclusion might be entirely wrong. It might be the case that the branch above contains very meaningful features, and they simply happen to be useless to a linear classifier applied right there. The idea of using linear classification probes to understand the roles of different branches is suggested as a heuristic instead of a hard rule. Moreover, if the probes are not optimized perfectly, the conclusions drawn can be misleading.
Note that we are reporting here the prediction errors, and it might be the case that the loss is indeed lower when we concatenate the two branches, but for some reason it could fail to apply to the prediction error.
0 5 10 15 20 25 30 35 linear probe at layer k
0.0
0.1
0.2
0.3
0.4
0.5
o p ti
m a l p re
d ic
ti o n e
rr o r
Figure 3: Toy experiment described in section 3.3, with linearly separable data (two labels), an untrained MLP with 32 layers, and probes at every layer. We report the prediction error for every probe, where 0.50 would be the performence of a coin flip and 0.00 would be ideal. Note that the layer 0 here corresponds to the raw data, and the probes are indeed able to classify it perfectly. As expected, performance degrades when applying random transformations. If many more layers were present, it would be hard to imagine how the final layer (with the model loss) can get any useful signal to backpropagate.
3.3 PROBES ON UNTRAINED MODEL
We start with a toy example to illustrate what kind of plots we expect from probes. We use a 32- layer MLP with 128 hidden units. All the layers are fully-connected and we use LeakyReLU(0.5) as activation function.
We will run the same experiment 100 times, with a different toy dataset each time. The goal is to use a data distribution (X,Y ) where X ∈ R128 is drawn N (0, I) and where Y ∈ {−1, 1} in linearly separable (i.e. super easy to classify with a one-layer neural network). To do this, we just pick a w ∈ R128 for each experiment, and let the label yn be the sign of xTnw. We initialize this 32-layer MLP using glorot normal initialization, we do not perform any training on the model, and we add one probe at every layer. We optimize the probes with RMSProp and a sufficiently small learning rate.
In Figure 3, we show the prediction error rate for every probe, averaged over the 100 experiments. The graph includes a probe applied directly on the inputs X , where we naturally have an error rate that is essentially zero (to be expected by the way we constructed our data), and which serves as a kind of sanity check. Given that we have only two possible labels, we also show a dotted horizontal line at 0.50, which is essentially the prediction error that we would get by flipping a coin. We can see that the prediction error rate climbs up towards 0.50 as we go deeper in the MLP (with untrained parameters).
This illustrates the idea that the input signal is getting mangled by the successive layers, so much that it becomes rather useless by the time we reach the final layer. We checked the mean activation norm of the hidden units at layer 32 to be sure that numerical underflow was not the cause for the degradation. Note that this situation could be avoided by using orthogonal weights.
One of the popular explanation for training difficulties in very deep models is that of the exploding/vanishing (Hochreiter, 1991; Bengio et al., 1993). Here we would like to offer another complementary explanation, based on the observations from Figure 3. That is, at the beginning of training, the usefulness of layers decays as we go deeper, reaching the point where the deeper layers are utterly useless. The values contained in the last layer are then used in the final softmax classifier, and the loss backpropagates the values of the derivatives. Since that derivative is based on garbage activations, the backpropagated quantities are also garbage, which means that the weights are all going to be updated based on garbage. The weights stay bad, and we fail to train the model. The authors like to refer to that phenomenon as garbage forwardprop, garbage backprop, in reference to the popular concept of garbage in, garbage out in computer science.
3.4 PROBES ON MNIST CONVNET
In this section we run the MNIST convolutional model provided by the tensorflow github repo (tensorflow/models/image/mnist/convolutional.py) We selected that model for reproducibility and to demonstrate how to easily peek into popular models by using probes.
We start by sketching the model in Figure 4. We report the results at the beginning and the end of training on Figure 5. One of the interesting dynamics to be observed there is how useful the first layers are, despite the fact that the model is completely untrained. Random projections can be useful to classify data, and this has been studied by others (Jarrett et al., 2009).
3.5 PROBES ON INCEPTION V3
We have performed an experiment using the Inception v3 model on the ImageNet dataset (Szegedy et al., 2015; Russakovsky et al., 2015). This is very similar to what is presented in section 3.4, but on a much larger scale. Due to the challenge presented by this experiment, we were not able to do everything that we had hoped. We have chosen to put those results in the appendix section A.2.
Certain layers of the Inception v3 model have approximately one million features. With 1000 classes, this means that some probes can take even more storage space than the whole model itself. In these cases, one of the creative solutions was to try to use only a random subset of the features. This is discussed in the appendix section A.1.
3.6 AUXILIARY LOSS BRANCHES AND SKIP CONNECTIONS
Here we investigate two ways to modify a deep model in order to facilitate training. Our goal is not to convince the reader that they should implement these suggestions in their own models. Rather, we want to demonstrate the usefulness of the linear classifier probes as a way to better understand what is happening in their deep networks.
In both cases we use a toy model with 128 fully-connected layers with 128 hidden units in each layer. We train on MNIST, and we use Glorot initialization along with leaky ReLUs.
We choose this model because we wanted a pathologically deep model without getting involved in architecture details. The model is pathological in the sense that smaller models can easily be designed to achieve better performance, but also in the sense that the model is so deep that it is very hard to train it with gradient descent methods. From our experiments, the maximal depth where things start to break down was depth 64, hence the choice here of using depth 128.
In the first scenario, we add one linear classifier at every 16 layers. These classifiers contribute to the loss minimization. They are not probes. This is very similar to what happens in the famous Inception model where “auxiliary heads” are used (Szegedy et al., 2015). This is illustrated in Figure 6a, and it works nicely. The untrainable model is now made trainable through a judicious use of auxiliary classifier losses. The results are shown in Figure 7.
In the second scenario, we look at adding a bridge (a skip connection) between layer 0 and layer 64. This means that the input features to layer 64 are obtained by concatenating the output of layer 63 with the features of layer 0. The idea here is that we might observe that the model would effectively train a submodel of depth 64, using the skip connection, and shift gears later to use the whole depth of 128 layers. This is illustrated in Figure 6b, and the results are shown in Figure 8. It does not work as expected, but the failure of this approach is visualized very nicely with probes and serves as a great example of their usefulness in diagnosing problems with models.
In both cases, there are two interesting observations that can be made with probes. We refer readers to https://youtu.be/x8j4ZHCR2FI for the full videos associated to Figures 5, 7 and 8.
Firstly, at the beginning of training, we can see how the raw data is directly useful to perform linear classification, and how this degrades as more layers are added. In the case of the skip connection in Figure 8, this has the effect of creating two bumps. This is because the layer 64 also has the input data as direct parent, so it can fit a probe to that signal.
Secondly, the prediction error goes down in all probes during training, but it does so in a way that starts with the parents before it spreads to their descendants. This is even more apparent on the full video (instead of the 3 frames provided here). This is a ripple effect, where the prediction error in Figure 6b is visually spreading like a wave from the left of the plot to the right.
4 DISCUSSION AND FUTURE WORK
We have presented more toy models or simple models instead of larger models such as Inception v3. In the appendix section A.2 we show an experiment on Inception v3, which proved to be more challenging than expected. Future work in this domain would involve performing better experiments on a larger scale than small MNIST convnets, but still within a manageable size so we can properly train all the probes. This would allow us to produce nice videos showing many training steps in sequence.
We have received many comments from people who thought about using multi-layer probes. This can be seen as a natural extension of the linear classifier probes. One downside to this idea is that we lose the convexity property of the probes. It might be worth pursuing in a particular setting, but as of
now we feel that it is premature to start using multi-layer probes. This also leads to the convoluted idea of having a regular probe inside a multi-layer probe.
5 CONCLUSION
In this paper we introduced the concept of the linear classifier probe as a conceptual tool to better understand the dynamics inside a neural network and the role played by the individual intermediate layers. We are now able to ask new questions and explore new areas. We have demonstrated how these probes can be used to identify certain problematic behaviors in models that might not be apparent when we traditionally have access to only the prediction loss and error.
We hope that the notions presented in this paper can contribute to the understanding of deep neural networks and guide the intuition of researchers that design them.
ACKNOWLEDGMENTS
Yoshua Bengio is a senior CIFAR Fellow. The authors would like to acknowledge the support of the following agencies for research funding and computing support: NSERC, FQRNT, Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR.
A APPENDIX
A.1 PROPOSAL : TRAIN PROBES USING ONLY SUBSETS OF FEATURES
One of the challenges to train on the Inception v3 model is that many of the layers have more than 200, 000 features. This is even worse in the first convolution layers before the pooling operations, where we have around a million features. With 1000 output classes, a probe using 200, 000 features has a weight matrix taking almost 1GB of storage.
When using stochastic gradient descent, we require space to store the gradients, and if we use momentum this ends up taking three times the memory on the GPU. This is even worse for RMSProp. Normally this might be acceptable for a model of reasonable size, but this turns into almost 4GB overhead per probe.
We do not have to put a probe at every layer. We can also train probes independently. We can put probe parameters on the CPU instead of the GPU, if necessary. But when the act of training probes increases the complexity of the experiment beyond a certain point, the researcher might decide that they are not worth the trouble.
We propose the following solution : for a given probe, use a fixed random subset of features instead of the whole set of features.
With certain assumptions about the independence of the features and their shared role in predicting the correct class, we can make certain claims about how few features are actually required to assess the prediction error of a probe. We thank Yaroslav Bulatov for suggesting this approach.
We ran an experiment in which we used data X ∼ N (0, ID) where D = 100, 000 is the number of features. We used K = 1000 classes and we generated the ground truth using a matrix W of shape (D,K). To obtain the class of a given x, we simply multiply xTW and take the argmax over the K components of the result.
x ∼ N (0, ID) y = argmax k=1..K
( xTW [:, k] ) We selected a matrix W by drawing all its individual coefficients from a univariate gaussian.
Instead of using D = 100, 000 features, we used instead only 1000 features picked at random. We trained a linear classifier on those features and, experimentally, it was relatively easy to achieve a 4% error rate on our first try. With all the features, we could achieve a 0% error rate, so 4 % might not look great. We have to keep in mind that we have K = 1000 classes so random guesses yield an error rate of 99.9%.
This can reduce the storage cost for a probe from 1GB down to 10MB. The former is hard to justify, and the latter is almost negligible.
A.2 PROBES ON INCEPTION V3
We are interested in putting linear classifier probes in the popular Inception v3 model, training on the ImageNet dataset. We used the tensorflow implementation available online (tensorflow/models/inception/inception) and ran it on one GPU for 2 weeks.
As described in section A.1, one of the challenges is that the number of features can be prohibitively large, and we have to consider taking only a subset of the features. In this particular experiment, we have had the most success by taking 1000 random features for each probe. This gives certain layers an unfair advantage if they start with 4000 features and we kept 1000, whereas in other cases the probe insertion point has 426, 320 features and we keep 1000. There was no simple “fair” solution. That being said, 13 out of the 17 probes have more than 100, 000 features, and 11 of those probes have more than 200, 000 features, so things were relatively comparable.
We put linear classifier probes at certain strategic layers. We represent this using boxes in the following Figure 9. The prediction error of the probe given by the last layer of each box is illustrated by coloring the box. Red is bad (high prediction error) and green/blue is good (low prediction error).
We would have liked to have a video to show the evolution of this during training, but this experiment had to be scaled back due to the large computational demands. We show here the prediction errors at three moments of training. These correspond roughly to the beginning of training, then after a few days, and finally after a week.
Inception v3

This paper proposes to use a linear classifier as the probe for the informativeness of the hidden activations from different neural network layers. The training of the linear classifier does not affect the training of the neural network.
The paper is well motivated for investigating how much useful information (or how good the representations are) for each layer. The observations in this paper agrees with existing insights, such as, 1) (Fig 5a) too many random layers are harmful. 2) (Fig 5b) training is helpful. 3) (Fig 7) lower layers converge faster than higher layer. 4) (Fig 8) too deep network is hard to train, and skip link can remedy this problem.
However, this paper has following problems:
1. It is not sufficiently justified why the linear classifier is a good probe. It is not crystal clear why good intermediate features need to show high linear classification accuracy. More theoretical analysis and/or intuition will be helpful.
2. This paper does not provide much insight on how to design better networks based on the observations. Designing a better network is also the best way to justify the usefulness of the analysis.
Overall, this paper is tackling an interesting problem, but the technique (the linear classifier as the probe) is not novel and more importantly need to be better justified. Moreover, it is important to show how to design better neural networks using the observations in this paper.