## Learning Joint Acoustic-Phonetic Word Embeddings

## Mohamed El-Geish

Voicea geish@voicea.ai

## Abstract

Most speech recognition tasks pertain to mapping words across two modalities: acoustic and orthographic. In this work, we suggest learning encoders that map variable-length - acoustic or phonetic - sequences that represent words into fixed-dimensional vectors in a shared latent space; such that the distance between two word vectors represents how closely the two words sound. Instead of directly learning the distances between word vectors, we employ weak supervision and model a binary classification task to predict whether two inputs, one of each modality, represent the same word given a distance threshold. We explore various deep-learning models, bimodal contrastive losses, and techniques for mining hard negative examples such as the semi-supervised technique of self-labeling. Our best model achieves an F 1 score of 0.95 for the binary classification task.

## 1 Introduction

The proliferation of voice-first applications and devices generated unprecedented demand to improve a plethora of speech recognition tasks. In this work, we propose a building block for speech recognition tasks like automatic speech recognition (ASR), keyword spotting, and query-by-example search. The outcome is an embedding model that maps variable-length - acoustic or phonetic - sequences that represent words into fixed-dimensional vectors in a shared latent space, which connects audio and phonetic modalities together, and encompasses a distributed representation of words, such that the distance between two word vectors represents how closely the two words sound: The more similar words sound, in either modality, the closer they end up in the shared vector space. Deep learning is befitting to learn pairwise relationships and joint embeddings, which have become the cornerstone of many machine learning applications [4, 38, 14, 26]. One of the applications we can reformulate is using the learned joint embeddings is ASR hypotheses reranking to reduce word error rate [2, 28].

The approach we took to learn distances between word embeddings in the vector space can be described as a weakly supervised task: Instead of training using ground-truth pairwise distances, we produced ground-truth data of word pairs labeled as either similar-sounding (distance of 0) or different-sounding (distance of 1) - making it a binary classification problem. At inference time, the model predicts real-valued distances that can be turned into labels given a distance threshold.

Despite the emergence of many successful embedding models, learning them is relatively poorly understood [46]. We experimented with various techniques to mine hard examples for training. Training using words picked at random from a corpus to be dissimilar to one another impedes learning; the model needs to be trained using hard examples of words that sound slightly different as examples of distant words [38, 47]. Another facet of weak supervision is creating ground-truth data using heuristics applied to the results of ASR systems and unreliable, non-expert transcribers.

## 2 Related Work

Thanks to recent natural language processing (NLP) research [32, 37, 1], the use of word embeddings to represent the semantic relationships between words has been prevailing. Similarly, in the field of speech recognition, the use of acoustic word embeddings has improved many tasks like keyword spotting [5], ASR systems [2], query-by-example search [25, 24, 41], and other word discrimination tasks [40, 21]; it also enabled attempts at unsupervised speech processing [20, 9] and representing semantic relationships between words [7, 8]. The aforementioned speech recognition research learned models from the acoustic representation of words - a single view, which requires side information to map that view into text. Infusing cross-modal (acoustic and orthographic) representations of words into the embedding model showed improvements in a plethora of applications; for example, [16] jointly learned to embed acoustic sequences and their respective character sequences into a common vector space using deep bidirectional long short-term memory (LSTM) embedding models and multiview contrastive losses. In [10], the model separately learns acoustic and orthographic embedding spaces, and then attempts to align them using adversarial training; it employs a bidirectional LSTM encoder and a unidirectional LSTM decoder to perform the cross-modal alignment in an unsupervised fashion; the performance of such model is comparable to its supervised counterpart.

One of the limitations of mapping acoustic representations to their corresponding character sequences is confusing homographs (words that share the same spelling but not the same pronunciation) - they may miss the subtle phonetic variations within words and/or across dialects. Multiple approaches attempted to mitigate such limitation. For example, [27] models phonetic information by training using both word- and frame- level criteria; however, it doesn't learn from phonetic labels so the learned embeddings cannot precisely represent the phonetic spelling of words. In [45], the objective is to learn word embeddings from audio segments and use side information (phonetic labels) to train an acoustic model, which can be used for ASR systems - a desideratum similar to ours; it trains the model using a Siamese neural network and explores multiple loss functions. In [9], the objective is to find vector representations that precisely describe represent the sequential phonetic structures of audio segments; in addition, [16] considers directly training using phonetic-spelling supervision as a future direction. It's worth mentioning that we discovered the latter after embarking on this endeavor, which validated our decision of using phonetic spelling instead of orthography.

In [2, 40, 16], the models are trained using matched (similar-sounding) and mismatched (differentsounding) pairs of words; the mismatched words are drawn randomly, which makes minimizing losses like triplet and contrastive loss a challenge [47]. In [10], adversarial training is used to align the acoustic and orthographic vector spaces. We use three different techniques, which we detail below, to mine for hard negative examples mining.

## 3 Dataset and Representations

The raw data consist of 25k short, single-channel recordings and their respective transcripts. The recordings were captured at a sample rate of 16kHz and encoded using pulse-code modulation with 16-bit precision. Transcripts do not include word alignments and may include errors. The dataset is proprietary and growing. To obtain the phonetic spelling of words, we used the LibriSpeech lexicon [34]; it contains over 200k words and their pronunciations; out-of-vocabulary (OOV) words are excluded from the acoustic dataset as they lack ground-truth phonetic labels.

In order to obtain word alignments, we force-aligned each transcript in the raw dataset with recognition hypotheses generated by an ensemble of ASR systems. When the forced alignment failed due to an insertion or a deletion error, we considered the affected words too noisy to include. When the human-selected transcript for a word aligned successfully with an ASR hypothesis, we labeled the pair (the audio segment and its phonetic label) as similar-sounding (distance of 0). Using the set of positive pairs, we mined unique pairs of ASR hypotheses that substituted the human-selected transcript for the same audio segment and labeled them as hard negative examples (distance of 1). The restriction of starting from positive pairs reduced the overall number of examples; however, it drastically increased the quality of the labels as we minimized the number of false negatives.

We employed another technique to mine more of the hard negative examples: We started from the hard negative examples we had created above and computed their phonetic-edit distance (between the human-selected transcript and the ASR's hypothesis for the same audio segment); then, we picked ones that scored below a maximum distance threshold (0.7 was satisfactory) and grouped them by their respective human-selected transcripts; finally, we synthesized additional unique hard negative examples for ones that share the same human-selected transcript. The result of this process increased the number of negative examples from 73,845 to 439,679; it also helped balance the dataset,

which included 393,623 positive examples. We removed stop words from the dataset as they are not interesting for downstream tasks; words shorter than 0.2 seconds long were also removed as manual inspection deemed them mostly mislabeled. The processed dataset consisted of 654,224 examples that were split, with class-label stratification, into train/dev/test datasets of sizes 621,905/12,692/19,627 respectively.The three sets contained 352,769/11,775/17,632 unique audio segments, respectively.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00493v1-figure-bdea102b-eeae-4428-b23b-4ccee62a7f56.jpg)

Figure 1: Distributions of word durations (left) and frequencies (right) in the dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00493v1-figure-024b5adf-8c8b-4933-9dac-376aa858c324.jpg)

The third technique we used to mine even more hard negative examples is a semi-supervised learning technique that's attributed to Scudder for his work circa 1965: self-labeling [39]. Every r epochs during training, we allow the dataset generator to augment the training dataset with newly minted hard negative examples. We encode the phonetic spelling of the training set's lexicon and build a k -dimensional tree for the resultant phonetic embeddings. Then we sample, at random, from the unique audio segments in the training set such that the sample's size is proportional to the number of epochs. Constraining the contributions of the less accurate models at early epochs reduces the chances of polluting the training set with easy-to-predict negative examples and hogging the memory. For each audio segment in the sample, we compute an embedding and find the closest phonetic neighbors within a maximum distance that reflects similarity (predicted a positive label). For each neighbor, we look up the true label of the word pair (audio segment and its phonetic label); if the model made a new mistake (the true label was negative and the pair is unique), we add the example to the training set as a hard negative. We observed a healthy growth in the training set's size thanks to self-labeling; it reached a 41% relative growth (to a total of 876,771 training examples) at one point.

We represented audio as mel-spectograms using standard parameters given a sample rate of 16KHz: 400 samples (25ms) for the length of the Fast Fourier Transform (FFT) window; a hop of 200 samples between successive frames, and 64 mel bands. The input audio signal was centered and padded with zeros (silence) to fit in a window of 2 seconds to fix the size of the model's input; the result is a tensor of size 64 × 161 × 1 (single channel). Audio examples were normalized individually using cepstral mean and variance normalization [44] and then normalized across examples to have zero mean and unit variance for each mel band.

We represented phonetic spelling of words as a sequence of one-hot encoded vectors; the longest word in the LibriSpeech lexicon has 20 phones and there are 69 unique phones in the ARPAbet flavor used by the lexicon. We indicate an empty phone using a sentinel value (making the length of the alphabet 70). The result is a matrix of size 70 × 20 .

## 4 Methods

In this work, we follow an approach similar to [16] in the sense that we model the task at hand as a weakly supervised, bimodal binary classification task; and along the way, we learn acoustic and phonetic encoders that map words into a shared vector space such that the distance between two word vectors represents how closely the two words sound. We train a Siamese neural network [4] that feeds forward the acoustic and phonetic representations of a pair of words through a series of transformations to encode the two inputs into glyph[lscript] 2 -normalized vectors (embeddings), then outputs the distance between the two embeddings. Our objective is to minimize the contrastive loss [13], which

allows us to learn encoders that map similar inputs to nearby points and dissimilar inputs to distant points in the shared vector space. To describe the objective formally, let ( x , x a p ) be the input word pair (acoustic and phonetic representations, respectively) and y be its true binary label such that y = 0 when the two representations, acoustic and phonetic, are of the same word; otherwise, y = 1 . The model learns two functions, f ( x a ) and g x ( p ) , that map the inputs into glyph[lscript] 2 -normalized embeddings; for each prediction, we compute the distance D ( f ( x a ) , g x ( p )) , or D for brevity, between the outputs of both functions using a distance function such as the Euclidean or cosine distance. Since the embeddings are glyph[lscript] 2 -normalized, we think of the two distance functions as mostly interchangeable. Given N training examples, we minimize the following function:

$$\mathcal { L } = \frac { 1 } { N } \sum _ { i } ^ { N } \left [ \, ( 1 - y ^ { ( i ) } ) \, ( \mathcal { D } ^ { ( i ) } ) ^ { 2 } + y ^ { ( i ) } \, \max ( 0, \, m - \mathcal { D } ^ { ( i ) } ) ^ { 2 } \, \right ]$$

where m&gt; 0 is a margin parameter that controls when dissimilar pairs contribute to the loss function: only when D &lt; m . Unless otherwise specified, we used m = 1 for the experiments below.

## 5 Experiments and Results

Evaluating embeddings depends heavily on the downstream task; in this work, we picked F 1 score (the harmonic mean of precision and recall) as the metric of choice for the bimodal binary classification task. The binary labels at test time were calculated at a distance threshold of 0 5 . , which approximates the observed break-even point when m = 1 in our experiments. In order to compute the break-even point for the test set, we generated all unique pairs of acoustic and phonetic inputs in the set approximately, 192.6 million word pairs. The F 1 score of our best-performing model is 0 95 . .

(a) Development set loss before self-labeling to augment training data (red line) and after (gray line) - learning use to stagnate at much earlier epochs (x-axis).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00493v1-figure-33b42638-2ada-4d3a-8342-49b6e5872083.jpg)

(b) Confusion matrix of our best-performing model.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00493v1-figure-cc188866-ea6f-44ef-b9d5-a5bcb804b49a.jpg)

Projecting a sample of embeddings derived from the test set using a t-SNE [29] model results in a reasonable clustering of words. More interestingly, the model found phonetic analogies such as "cat" to "cool" is like "pat" to "pool", etc. We also inspected a sample of 50 classification mistakes the model made. A few patterns emerged: the model failed to predict a distance that reflects similarity between the acoustic and phonetic embeddings when the audio was too noisy, too faint (far field), or the speech was accented. Cross talk and reverb were also problematic but not as common. Audio preprocessing to clean up the signal may be helpful in such cases. Also, training using n-grams and acquiring more data from speakers with accents and different acoustic environment may boost performance as we expect distributions of words to be non-IID (independent and identically distributed).

We experimented with and manually tuned various hyperparameter choices for input representations, architecture, contrastive losses, batch size, number of hidden units, number of epochs, embedding size, dimensions of filters in convolutional neural networks (CNN), single vs. bidirectional LSTM cells, Euclidean- vs. cosine- based distance functions, etc. The sparse representation of phonetic input led us to believe that the architecture for its encoder should be different from that of the acoustic one; however, empirical evidence suggests that mirroring the same architecture for both encoders,

Table 1: Summary of notable experiments and their results for training and testing, respectively.

|   # | Notable Experiment Details                                                                                                                                                                                                                                   | F 1 Scores      |
|-----|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|
|   1 | CNN ( 3 × 3 × 32 ); dense layer; 256-D embedding; batch size = 32                                                                                                                                                                                            | 0 . 97 , 0 . 91 |
|   2 | CNN ( 3 × 3 × 32 ) -> ( 3 × 3 × 64 ); dense layer; 256 -D embedding; batch size = 32                                                                                                                                                                         | 0 . 99 , 0 . 93 |
|   3 | Same as #2 but for a dropout with a rate of 0 . 5 after the first hidden layer                                                                                                                                                                               | 0 . 99 , 0 . 94 |
|   4 | Same as #3 but with another dropout of 0 . 5 after the second hidden layer                                                                                                                                                                                   | 0 . 95 , 0 . 91 |
|   5 | Same as #3 but with margin = the phonetic-edit distance for the pair                                                                                                                                                                                         | 0 . 96 , 0 . 91 |
|   6 | Same as #3 but with incoming weights constrained to a maximum norm of 3                                                                                                                                                                                      | 0 . 99 , 0 . 94 |
|   7 | 2 unidirectional LSTM layers with 128 hidden units; dense layer; 256 -D embedding; batch size = 32 ; 24 epochs (in 99 hours)                                                                                                                                 | 0 . 91 , 0 . 87 |
|   8 | 2 Bidirectional LSTM layers with 512 hidden units and a dropout of 0 . 4 in between; a dropout of 0 . 2 for the acoustic input; 512 -D embedding; 28 epochs (in 47 hours)                                                                                    | 0 . 95 , 0 . 91 |
|   9 | CNN with 2 blocks [( 3 × 3 × 64 ) -> ( 2 × 2 ) max pooling]; two dense layers with 512 hidden units and a dropout of 0.4 in between; a dropout of 0.2 for the acoustic input; 512 -D embedding; cosine distance; batch size = 128 ; 64 epochs (in 4.8 hours) | 0 . 99 , 0.95   |
|  10 | Same as #9 but with additional dropout of 0 . 4 between convolutional layers as well; trained for much longer (142 epochs in 19 hours)                                                                                                                       | 0 . 96 , 0 . 93 |

with the exception of a dropout for the acoustic input layer, yields better results. Table 1 summarizes notable experiments and their results.

We use the glyph[lscript] 2 -normalized output of each encoder's last layer as the learned embedding hence we don't use an activation function to those layers. Unless otherwise specified, we constrained incoming weights to a maximum norm of 3 for layers with dropout regularization to allow for large learning rates without the risk of the weights ballooning [43]. We used the Adam optimizer [22] with initial learning rates tuned for different architectures ( 0 001 . for CNN and 0 0001 . for LSTM); we reduced the learning rate by a factor of 2 when learning stagnates for 2 epochs. Since the addition of self-labeling to our experiments, we observed a much steeper acceleration in learning in early epochs. To balance the learning rate decay with the addition of new data on-the-fly during training, we increase the learning rate - by the same factor to a maximum of 0 001 . -when new hard negative examples are mined to give the model a better chance at learning from the newly minted examples.

For LSTM, we used tanh for activation; otherwise, we used rectified linear units (ReLU) [33], with the exception of the model's and the encoder's output layers. Network weight were initialized using He initialization [15] when ReLU was used; otherwise, we use the Xavier initialization method [12].

## 6 Conclusion and Future Work

We presented techniques for learning functions that map acoustic and phonetic representations of words into fixed-dimensional vectors in a shared latent space, which are extremely useful in a plethora of speech recognition tasks. We experimented with many modeling techniques, hyperparameters, and neural network architectures to learn the joint embeddings; our best model is a Siamese CNN that feeds forward acoustic and phonetic inputs and brings together, in the shared vector space, similar-sounding words while keeping apart different-sounding ones. We used binary classification as a surrogate task to learn the embeddings at the last layer of each encoder. The choice of training examples cannot be cannot be overstated: we use three different techniques, including self-labeling, to mine hard negative examples for the contrastive loss function so that it can learn the subtleties requisite to discriminate between input words.

One of the areas to explore in future work is training multiple models for multiple word duration buckets to minimize extraneous padding. We'd also like to explore other loss functions detailed in [46, 16].

## Acknowledgments

We'd like to thank Ahmad Abdulkader for his mentorship and guidance; and the authors of related work and software tools [6, 36, 31, 17, 23, 18, 30, 35, 3, 42, 19, 11] for allowing us to build on top of what they've created.

## References

| [1]   | Ahmad Abdulkader, A Lakshmiratan, and J Zhang. 'Introducing DeepText: Facebook's Text Understanding Engine'. In: Facebook Code (2016).                                                                                                                                                                                                             |
|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [2]   | Samy Bengio and Georg Heigold. 'Word Embeddings for Speech Recognition'. In: Fifteenth Annual Conference of the International Speech Communication Association . 2014.                                                                                                                                                                             |
| [3]   | Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python . 1st. O'Reilly Media, Inc., 2009. ISBN: 9780596516499.                                                                                                                                                                                                         |
| [4]   | Jane Bromley et al. 'Signature Verification Using a "Siamese" Time Delay Neural Network'. In: Advances in neural information processing systems . 1994, pp. 737-744.                                                                                                                                                                               |
| [5]   | Guoguo Chen, Carolina Parada, and Tara N Sainath. 'Query-by-Example Keyword Spotting Using Long Short-Term Memory Networks'. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2015, pp. 5236-5240.                                                                                               |
| [6]   | François Chollet et al. Keras . https://keras.io . 2015.                                                                                                                                                                                                                                                                                           |
| [7]   | Yu-An Chung and James Glass. 'Learning Word Embeddings from Speech'. In: (2017).                                                                                                                                                                                                                                                                   |
| [8]   | Yu-An Chung and James Glass. 'Speech2vec: A Sequence-to-Sequence Framework for Learn- ing Word Embeddings from Speech'. In: INTERSPEECH . 2018.                                                                                                                                                                                                    |
| [9]   | Yu-An Chung et al. 'Audio Word2Vec: Unsupervised Learning of Audio Segment Representa- tions Using Sequence-to-Sequence Autoencoder'. In: (2016).                                                                                                                                                                                                  |
| [10]  | Yu-An Chung et al. 'Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces'. In: Advances in Neural Information Processing Systems 31 . Ed. by S. Bengio et al. Curran Associates, Inc., 2018, pp. 7354-7364. URL: http://papers.nips.cc/paper/ 7965-unsupervised-cross-modal-alignment-of-speech-and-text-embedding- spaces.pdf . |
| [11]  | FFmpeg . 2016. URL: http://http://www.ffmpeg.org/ (visited on 12/16/2016).                                                                                                                                                                                                                                                                         |
| [12]  | Xavier Glorot and Yoshua Bengio. 'Understanding the difficulty of training deep feedforward neural networks'. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics . 2010, pp. 249-256.                                                                                                            |
| [13]  | Raia Hadsell, Sumit Chopra, and Yann LeCun. 'Dimensionality Reduction by Learning an Invariant Mapping'. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) . Vol. 2. IEEE. 2006, pp. 1735-1742.                                                                                                       |
| [14]  | David Harwath and James R Glass. 'Learning Word-Like Units from Joint Audio-Visual Analysis'. In: (2017).                                                                                                                                                                                                                                          |
| [15]  | Kaiming He et al. 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification'. In: Proceedings of the IEEE international conference on computer vision . 2015, pp. 1026-1034.                                                                                                                                    |
| [16]  | Wanjia He, Weiran Wang, and Karen Livescu. 'Multi-View Recurrent Neural Acoustic Word Embeddings'. In: International Conference on Learning Representations (2017).                                                                                                                                                                                |
| [17]  | John D Hunter. 'Matplotlib: A 2D graphics environment'. In: Computing in science & engineering 9.3 (2007), p. 90.                                                                                                                                                                                                                                  |
| [18]  | Eric Jones, Travis Oliphant, and Pearu Peterson. ' { SciPy } : Open source scientific tools for { Python } '. In: (2014).                                                                                                                                                                                                                          |
| [19]  | Kaldi ASR . 2016. URL: http://kaldi-asr.org/ .                                                                                                                                                                                                                                                                                                     |
| [20]  | Herman Kamper, Aren Jansen, and Sharon Goldwater. 'A Segmental Framework for Fully- Unsupervised Large-Vocabulary Speech Recognition'. In: Computer Speech &Language 46 (2017), pp. 154-174.                                                                                                                                                       |

| [21]   | Herman Kamper, Weiran Wang, and Karen Livescu. 'Deep Convolutional Acoustic Word Embeddings Using Word-Pair Side Information'. In: Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on . IEEE. 2016, pp. 4950-4954.    |
|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [22]   | Diederik P Kingma and Jimmy Ba. 'Adam: A method for stochastic optimization'. In: (2014).                                                                                                                                                                  |
| [23]   | Thomas Kluyver et al. 'Jupyter Notebooks - a publishing format for reproducible computa- tional workflows'. In: Positioning and Power in Academic Publishing: Players, Agents and Agendas . Ed. by F. Loizides and B. Schmidt. IOS Press. 2016, pp. 87-90. |
| [24]   | Keith Levin, Aren Jansen, and Benjamin Van Durme. 'Segmental Acoustic Indexing for Zero Resource Keyword Search'. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2015, pp. 5828-5832.                  |
| [25]   | Keith Levin et al. 'Fixed-Dimensional Acoustic Embeddings of Variable-Length Segments in Low-Resource Settings'. In: Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on . IEEE. 2013, pp. 410-415.                               |
| [26]   | Yingming Li, Ming Yang, and Zhongfei Mark Zhang. 'A Survey of Multi-View Representation Learning'. In: IEEE Transactions on Knowledge and Data Engineering (2018).                                                                                         |
| [27]   | Hyungjun Lim et al. 'Learning Acoustic Word Embeddings with Phonetically Associated Triplet Network'. In: (2018).                                                                                                                                          |
| [28]   | Yukun Ma, Erik Cambria, and Benjamin Bigot. 'ASR Hypothesis Reranking Using Prior- Informed Restricted Boltzmann Machine'. In: International Conference on Computational Linguistics and Intelligent Text Processing . Springer. 2017, pp. 503-514.        |
| [29]   | Laurens van der Maaten and Geoffrey Hinton. 'Visualizing data using t-SNE'. In: Journal of machine learning research 9.Nov (2008), pp. 2579-2605.                                                                                                          |
| [30]   | Brian McFee et al. librosa/librosa: 0.6.3 . Feb. 2019. DOI: 10.5281/zenodo.2564164 . URL: https://doi.org/10.5281/zenodo.2564164 .                                                                                                                         |
| [31]   | Wes McKinney et al. 'Data structures for statistical computing in python'. In: Proceedings of the 9th Python in Science Conference . Vol. 445. Austin, TX. 2010, pp. 51-56.                                                                                |
| [32]   | Tomas Mikolov et al. 'Distributed Representations of Words and Phrases and Their Composi- tionality'. In: Advances in neural information processing systems . 2013, pp. 3111-3119.                                                                         |
| [33]   | Vinod Nair and Geoffrey E Hinton. 'Rectified linear units improve restricted boltzmann machines'. In: Proceedings of the 27th international conference on machine learning (ICML- 10) . 2010, pp. 807-814.                                                 |
| [34]   | V. Panayotov et al. 'Librispeech: An ASR Corpus Based on Public Domain Audio Books'. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) Apr. 2015, pp. 5206-5210. DOI: 10.1109/ICASSP.2015.7178964 .               |
| [35]   | Adam Paszke et al. 'Automatic differentiation in PyTorch'. In: (2017).                                                                                                                                                                                     |
| [36]   | F. Pedregosa et al. 'Scikit-learn: Machine Learning in Python'. In: Journal of Machine Learning Research 12 (2011), pp. 2825-2830.                                                                                                                         |
| [37]   | Jeffrey Pennington, Richard Socher, and Christopher Manning. 'GloVe: Global Vectors for Word Representation'. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) . 2014, pp. 1532-1543.                    |
| [38]   | Florian Schroff, Dmitry Kalenichenko, and James Philbin. 'Facenet: A Unified Embedding for Face Recognition and Clustering'. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2015, pp. 815-823.                        |
| [39]   | H Scudder. 'Probability of error of some adaptive pattern-recognition machines'. In: IEEE Transactions on Information Theory 11.3 (1965), pp. 363-371.                                                                                                     |
| [40]   | S. Settle and K. Livescu. 'Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches'. In: 2016 IEEE Spoken Language Technology Workshop (SLT) Dec. 2016, pp. 503-510.                                                            |
| [41]   | Shane Settle et al. 'Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings'. In: (2017).                                                                                                                                             |
| [42]   | SoX - Sound eXchange . 2015. URL: http : / / sox . sourceforge . net/ (visited on                                                                                                                                                                          |

| [43]   | Nitish Srivastava et al. 'Dropout: a simple way to prevent neural networks from overfitting'. In: The Journal of Machine Learning Research 15.1 (2014), pp. 1929-1958.                                                       |
|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [44]   | Ole Morten Strand and Andreas Egeberg. 'Cepstral Mean and Variance Normalization in the Model Domain'. In: COST278 and ISCA Tutorial and Research Workshop (ITRW) on Robustness Issues in Conversational Interaction . 2004. |
| [45]   | Gabriel Synnaeve, Thomas Schatz, and Emmanuel Dupoux. 'Phonetics embedding learning with side information'. In: 2014 IEEE Spoken Language Technology Workshop (SLT) . IEEE. 2014, pp. 106-111.                               |
| [46]   | Evgeniya Ustinova and Victor Lempitsky. 'Learning Deep Embeddings with Histogram Loss'. In: Advances in Neural Information Processing Systems 29 . Ed. by D. D. Lee et al. Curran Associates, Inc., 2016, pp. 4170-4178.     |
| [47]   | Chao-Yuan Wu et al. 'Sampling Matters in Deep Embedding Learning'. In: Proceedings of the IEEE International Conference on Computer Vision . 2017, pp. 2840-2848.                                                            |
Mohamed El-Geish
Learning Joint Acoustic-Phonetic Word Embeddings
Most speech recognition tasks pertain to mapping words across two modalities:
acoustic and orthographic. In this work, we suggest learning encoders that map
variable-length, acoustic or phonetic, sequences that represent words into
fixed-dimensional vectors in a shared latent space; such that the distance
between two word vectors represents how closely the two words sound. Instead of
directly learning the distances between word vectors, we employ weak
supervision and model a binary classification task to predict whether two
inputs, one of each modality, represent the same word given a distance
threshold. We explore various deep-learning models, bimodal contrastive losses,
and techniques for mining hard negative examples such as the semi-supervised
technique of self-labeling. Our best model achieves an $F_1$ score of 0.95 for
the binary classification task.