Phonemes are defined by their relationship to words: changing a phoneme changes the word. Learning a phoneme inventory with lit-tle supervision has been a longstanding challenge with important applications to underresourced speech technology. In this paper, we bridge the gap between the linguistic and statistical definition of phonemes and propose a novel neural discrete representation learning model for self-supervised learning of phoneme inventory with raw speech and word labels. Given the availability of phoneme segmentation and some mild conditions, we prove that the phoneme inventory learned by our approach converges to the true one with an exponentially low error rate. Moreover, in experiments on TIMIT and Mboshi benchmarks, our approach consistently learns a better phoneme-level representation and achieves a lower error rate in a zero-resource phoneme recognition task than previous state-of-the-art self-supervised representation learning algorithms. Thanks to recent developments in self-supervised speech representation learning (van den Oord et al., 2017, 2019; Chorowski et al., 2019; Baevski et al., 2020), there is new hope for the development of speech processing systems without the need for full textual transcriptions. Supervised speech processing systems for tasks such as automatic speech recognition (ASR) rely on a large amount of textual transcriptions, but self-supervised systems can be applied to under-resourced languages in which such annotation is either scarce or unavailable. A key task of the self-supervised system is to learn a discrete representation. While it is possible to dis-cretize the speech solely on the basis of its acoustic properties, a more desirable discrete representation would serve as a bridge from the continuous acoustic signal toward higher-level linguistic structures such as syntax and semantics. Such a representation would make it possible to repurpose algorithms developed for written languages so that they could be used for unwritten languages in tasks such as speech translation and spoken language understanding. Words are the obvious choice for a discrete, semantic-driven speech representation, but a practical speech understanding system needs at least thousands of words; learning them in an unsupervised manner may be challenging. Phonemes may be a more learnable representation. According to the standard linguistic definition, phonemes are closely linked to words: Definition 1. (Linguistic definition of phonemes (Swadesh, 1934)) Phonemes are the smallest units in speech such that given a correct native word, the replacement of one or more phonemes by other phonemes (capable of occurring in the same position) results in a native word other than that intended, or a native-like nonsense word. For example, the sentences he th inks and he s inks differ by exactly one phoneme but have very different meaning. The optimal compactness of a phoneme inventory as specified in the definition leads to three advantages. First, learning phonemes requires lower sample complexity than learning words since the number of distinct phonemes is much smaller than the number of distinct words in a language. Second, the phonemes are much more abundant and more balanced in classes than words within a speech corpus, which makes sample complexity less of an issue when learning phonemes. Third, phonemes are more generalizable in the sense that knowing the phoneme inventory allows the learner to memorize previously unseen words as sequences of phonemes, and, having memorized them, to begin seeking clues to their meaning. Motivated by the semantic-driven definition of phonemes, we formulate the problem of learning a phoneme inventory as a self-supervised learning problem, where a small amount of semantic su-8027 pervision is available. The required supervision specifies which acoustic segments are instances of the same word, and which are instances of different words. Such supervision might be acquired in a naturalistic setting by asking native speakers to name objects in a set of standardized images, as is commonly done in primary education classrooms, or by asking for the translations of common words in a second language, a common baseline approach in dialectology and historical linguistics (Swadesh, 1952). Our contributions are threefold: (1) we propose a computationally tractable definition of phoneme that is almost equivalent to the linguistic definition. (2) We propose a finite-sample objective function for learning phoneme-level units and prove that when the phoneme segmentation is available and under mild conditions, the empirical risk minimizer (ERM) of this objective will find the correct phoneme inventory with exponentially low error rate. (3) We propose a novel neural network called information quantizer to optimize the proposed objective, which achieve state-of-the-art results in the phoneme inventory discovery task on the TIMIT and low-resourced Mboshi benchmarks with much less training data than previous approaches. Due to the challenge of learning phonemes, early works on unsupervised speech representation learning (Park and Glass, 2005; Lee and Glass, 2012; Ondel et al., 2016) focus on learning speech segments sharing similar acoustic properties, or phones , without taking into account the meaning of the speech they are part of. There are two main approaches in this direction. One approach is to learn discrete phone-like units without any textual labels by modeling phone labels of the speech segments as latent variables. In particular, (Park and Glass, 2005; Jansen et al., 2010) first detect segments with recurring patterns in the speech corpus followed by graph clustering using the similarity graph formed by the segments. (Lee and Glass, 2012; Ondel et al., 2016; Kamper et al., 2016) develop probabilistic graphical models to jointly segment and cluster speech into phone-like segments. An extension to the latent variable approach is to introduce additional latent variables such as speaker identity (Ondel et al., 2019) or language identity (Yusuf et al., 2020) and develop mechanisms to disentangle these variables. With the advance of deep learning, neural network models have also been proposed to learn unsupervised phone-level representation either by first learning a continuous representation (Chung et al., 2019; Feng et al., 2019; Nguyen et al., 2020) followed by off-line clustering, or by learning a discrete representation end-to-end with Gumbel softmax (Eloff et al., 2019b; Baevski et al., 2020) or vector-quantized variational autoencoder (VQ-VAE) (van den Oord et al., 2017; Chorowski et al., 2019; Baevski et al., 2019). However, codebooks learned by the neural approaches tend to be much larger than the number of phonemes (Baevski et al., 2020), leading to low scores in standard phoneme discovery metrics. The second approach utilizes weak supervision such as noisy phone labels predicted by a supervised, multilingual ASR system trained on other languages. Along this direction, early works (Schultz and Waibel, 1998; Lf et al., 2009; Swietojanski et al., 2012) have showed that phonetic knowledge gained from one language can be leveraged to develop ASR systems for another language using an HMM-based or DNN-HMM hybrid approach. Instead of using phone labels, (Stuker et al., 2003) explores the use of articulatory features as supervision for the multilingual ASR. Recently, ( Zelasko et al., 2020a,b; Feng et al., 2021a) systematically study the performance of zero-shot crosslingual ASR on 13 languages trained with international phonetic alphabet (IPA) tokens and found that the system tends to perform poorly on unseen languages. Instead, (Feng et al., 2021b) is able to discover phone-like units by clustering bottleneck features (BNF) from a factorized time-delay neural network (TDNN-f) trained with phone labels predicted by a crosslingual ASR (Feng et al., 2021a). Several works have since shifted focus toward the more challenging phoneme discovery problem by formulating it as a self-supervised learning problem where the semantics of the speech are known, such as from translation, phoneme-level language models or other sensory modalities such as vision. (Jansen, 2013) has studied the use of pairwise word identity labels for training phoneme discovery models based on Gaussian mixture models (GMM); (Harwath and Glass, 2019) analyzes the hidden layers of a two-branch neural network trained to retrieve spoken captions with semantically related images and finds strong correlation between segment representation and 8028 phoneme boundaries. (Harwath et al., 2020) adds hierarchical vector quantization (VQ) layers in the same retrieval network and is able to find a much smaller codebook than the unsupervised neural approach (Baevski et al., 2020), and achieve high correlation with the phoneme inventory. (Godard et al., 2018; Boito et al., 2019) has studied the possibility of learning semantic units using an attention-based speech-to-text translation system, though the units appear to correlate more with words. Works on unsupervised speech recognition (Chen et al., 2019) attempt to learn to recognize phonemes by leveraging the semantic information from a phoneme language model unpaired with the speech, typically by matching the empirical prior and posterior distributions of phonemes either using cross entropy (Yeh et al., 2019) or adversarial loss (Chen et al., 2019; Baevski et al., 2021). Such models, however, have a slightly different objective as they assume knowledge about the phoneme inventory of the language and instead tries to find the alignment between the speech and phonemes, rather than induce the phoneme inventory from scratch. We use capital letters to denote random variables and lower-case letters to represent samples of random variables. We use PX := P { X = x } to denote both probability mass and density functions of random variable X , depending on whether it is continuous or discrete. Further, denote PY | X ( y | x ) := P { Y = y | X = x } as the true conditional probability distribution of random variable Y = y given random variable X = x . The linguistic definition of phonemes can be rephrased as follows. Define X to be the set of all physical acoustic segments that can ever be produced as instances of the phonemes of a given language. Definition 1 can be phrased as follows: Two sequences of segments x = [ x 1 ,    , x T ] and x (cid:48) = [ x 1: t  1 , x (cid:48) t , x t +1: T ] , differing only in that x (cid:48) t (cid:54) = x t , are instances of different words, y (cid:48) (cid:54) = y , if and only if x (cid:48) t and x t are instances of different phonemes. In order to design effective algorithms, we will work with a relaxation of this definition, which we call the statistical definition of phonemes. Definition 2. (Statistical definition of phonemes) Let X be the set of all speech segments in a language, and let X be a random vector taking values in X and Y be a random variable representing the word of which X is one segment. The phoneme inventory of a language is the minimal partition Z = { Z 1 ,    , ZK } of X (i.e., X =  Kk =1 Z k , Z j  Z k =  ,  1  j, k  K ), such that if a speech segment pair ( x, x (cid:48) )  X 2 satisfies ( x, x (cid:48) )  Z 2 k for some k  { 1 ,    , K } , then their conditional distributions satisfy PY | X = x = PY | X = x (cid:48) . In other words, given only the knowledge that two acoustic sequences contain instances of the same phoneme, the resulting conditional distributions across possible word labels are the same. The fundamental intuition of Definition 2 is that different phonemes have different distributions across the words of the language. Two instances of the same phoneme, x and x (cid:48) , might have different likelihoods PX = x | Y and PX = x (cid:48) | Y , e.g., because of allophony; but their posteriors PY | X = x and PY | X = x (cid:48) cannot be different without violating Definition 1. The relationship between Definition 1 and Definition 2 is given by the following proposition, whose proof is in Appendix A.3. Proposition 1. Let Z =  Kk =1 Z k be a partition of X . If, for all possible { PY | X = x s } s (cid:54) = t , for any spoken word x = [ x 1 ,    , x T ] , and for any segment pairs ( x t , x (cid:48) t )  Z 2 k , k  { 1 ,    , K } , changing x t 8029 Figure 2: Network architecture of information quantizer to x (cid:48) t does not alter the identity of the word, i.e., arg max y PY | X 1: T ( y | x 1: t  1 , x (cid:48) t , x t +1: T ) = arg max y PY | X 1: T ( y | x ) , (2) but for any segment pairs x t  Z k , x (cid:48)(cid:48) t  Z l for k (cid:54) = l , changing x t to x (cid:48) t alters the identity of the word, i.e., arg max y PY | X 1: T ( y | x 1: t  1 , x (cid:48)(cid:48) t , x t +1: T ) (cid:54) = arg max y PY | X 1: T ( y | x ) , (3) then Z is a phoneme inventory from Definition 2. Define the phoneme assignment function z : X  { 1 ,    , K } such that z ( x ) = k if x  Z k . Suppose a segment X is randomly chosen from X with probability distribution PX and its phoneme label is another random variable Z := z ( X ) , then by Definition 2, for any pair x, x (cid:48)  X such that z ( x ) = z ( x (cid:48) ) , we have PY | X = x = PY | X = x (cid:48) = PY | Z = z ( x ) . The phoneme inventory is thereby completely characterized by the phoneme label function z (  ) as well as the set of distributions associated with each class PY | Z . Let z (  ) be the phoneme assignment function from Definition 2 and assume the size of the phoneme inventory is known to be K . Given a training set D = { ( x ( i ) , y ( i ) ) } ni =1 , where each x ( i ) is an acoustic segment extracted from a spoken word, and each y ( i )  Y is the corresponding word label, a semantic-driven phoneme discovery (SPD) system tries to find an assignment function that minimizes the token error rate (TER) : PTER ( z ) := min    P { z ( X ) (cid:54) =  ( z ( X )) } , (4) where  is the set of all permutations of length K , which is used because the problem is unsupervised and z (  ) is not available during training. An assignment function  z is said to achieve exact discovery if PTER ( z ) = 0 . It can be easily shown that TER is equivalent to standard evaluation metrics for phoneme discovery such as normalized mutual information (NMI) (Yusuf et al., 2020; Harwath et al., 2020; Feng et al., 2021b) and token F1 (Dun-bar et al., 2017), as presented in Appendix A.2. Thus, to provide guarantees for NMI and token F1, it suffices to provide a guarantee for TER. We solve the SPD problem using a novel type of neural network called an information quantizer (IQ), depicted in Figure 2. An IQ ( , q )    QK consists of four main components: A pre-segmentation network, a speech encoder e  1 (  ) , a word posterior c  2 (  ) and a quantizer q :  | Y |  C = { Q 1 ,    , QK } , where [  1 ,  2 ] =  and C is the distribution codebook and Q k 's are called the code distributions of q . IQ performs phoneme discovery in three stages. The pre-segmentation stage takes a raw speech waveform as input and extracts phoneme-level segments x = [ x 1 ,    , x T ] in a self-supervised fashion (Kreuk et al., 2020). Afterwards, in the joint distribution learning stage, the speech encoder extracts phoneme-level representations e  1 ( x ) = [ e  1 ( x 1 ) ,    , e  1 ( x T )] before passing them into the word posterior network to estimate the distribution of word labels, Y , given the presence in the word of acoustic phonetic segment X = x : P Y | X = x t = c  2 ( e  1 ( x t )) , 1  t  T. (5) 8030 Note that it is crucial that no recurrent connection exists between segments since our goal is to learn the probability of a word label given the presence of one phoneme segment. Finally, in the quantization stage, the quantizer creates the phoneme inventory by assigning each segment x t an integer index via codeword assignment function  z ( x t ) such that  z ( x t ) = k if q ( P Y | X = x t ) = Q k . The loss function that IQ minimizes has two goals: learn a good estimator for the conditional distribution PY | X and learn a good quantization function q (  ) . The first goal is achieved by minimizing the cross entropy loss: LCE ( P n ,  ) :=  1 n n (cid:88) i =1 log P Y | X ( y ( i ) | x ( i ) ) , (6) where P n is the empirical joint distribution. The second goal is achieved by minimizing the KL-divergence between the estimated conditional distribution before and after quantization: LQ (  P n , , q ) := 1 n n (cid:88) i =1 DKL ( P Y | X = x ( i ) || q ( P Y | X = x ( i ) )) , (7) where  P n := 1 n n (cid:88) i =1  x ( i ) P Y | X = x ( i ) is the smoothed version of the empirical distribution. The final loss function of IQ for SPD is then: LIQ ( P n , , q ) := LCE ( P n ,  ) +  LQ (cid:16)  P n , , q (cid:17) , ( P 1 ) where  > 0 is some hyperparameter set to approximately 1 for most experiments. Further, we restrict q to be nearest-neighbor so that: q ( P ) = arg min Q k :1  k  KDKL ( P || Q k ) . (8) This restriction does not increase the loss ( P 1 ) and serves as a regularization during phoneme discovery, as shown in Appendix A.3. We show that when the phoneme segmentation is available and under mild assumption, IQ is able to achieve exact discovery of phoneme inventory. First, let us state the main assumptions of the paper. Assumption 1. (boundedness of the density ratio) There exist universal constants C l < C u such that     ,  q  QK ,  ( x, y )  X  Y , log PY | X ( y | x ) P Y | X ( y | x )  [ C l , C u ] , log PY | X ( y | x ) q ( P Y | X ( y | x ))  [ C l , C u ] . Assumption 2. (log-smoothness of the density ratio) There exists  > 0 such that   1 ,  2   , x, y  X  Y , (cid:12)(cid:12)(cid:12)(cid:12) log P  1 Y | X ( y | x ) P  2 Y | X ( y | x ) (cid:12)(cid:12)(cid:12)(cid:12)   (cid:107)  1   2 (cid:107) . Assumption 3. (realizability) There exists a nonempty subset     such that P Y | X = PY | X ,      . Assumption 4. The true prior of the phoneme inventory is known to be PZ ( z ) = 1 K , 1  z  K . The first two assumptions are similar to the ones in (Tsai et al., 2020). Assumption 3 assumes that the true probability measure is within the function class, which combined with Assumption 1 requires the true distribution to share the same support as the estimated one. However, such assumption can be relaxed so that DKL ( P   Y | X || PY | X )  ,       for some small enough  > 0 , which does not affect the essential idea behind our analysis and can be achieved by some rich class of universal ap-proximators such as neural networks (Hornik et al., 1989). The last assumption ensures the inventory to be identifiable by assuming knowledge of the prior of the phoneme inventory. before giving some intuitive explanation. Theorem 1. Given Assumption 1-4, let the information quantizer ( ,  q ) with assignment function  z be an empirical risk minimizer (ERM) of ( P 1 ): LIQ ( P n ,  ,  q ) = min    ,q  QKLIQ ( P n , , q ) . For any   (0 , 1] , with probability at least 1   , the cluster assignment function  z of the ERM information quantizer  q achieves PTER ( z ) = 0 if the sample size n satisfies: x for some  > 0 and DJS ( P || Q ) := 12 DKL (cid:16) P || P + Q 2 (cid:17) + 12 DKL (cid:16) Q || P + Q 2 (cid:17) is the Jensen-Shannon divergence. The bound in Theorem 1 captures two main factors determining the sample complexity of exact phoneme discovery: the first factor is how close the word distributions of phonemes are from each other as measured by their Jensen-Shannon (JS) divergence, and the second factor is how hard it is for the training data to cover all the phonemes. The theorem works essentially because ( P 1 ) can be viewed as an approximation of the mutual information between the codeword  z ( X ) and word type Y , I ( z ( X ); Y ) . Suppose P  Y | X  PY | X and let H ( | ) denotes conditional entropy, we have: LIQ ( P n ,  ,  q )  H ( Y | X ) + DKL ( PY | X ||  q ( PY | X ))   I ( X ; Y ) + DKL ( PY | X ||  q ( PY | X )) =  I ( z ( X ); Y ) , which is minimized if  q ( PY | X ) = PY | z ( X ) . Datasets We construct four training datasets consisting of spoken words only. The vocabulary set with | Y | = 224 is selected from head words of noun phrases from the Flickr30kEntities dataset (Hodosh et al., 2010) that appear at least 500 times. For the Flickr audio word dataset, spoken words in the vocabulary are extracted from Flickr audio dataset (Harwath and Glass, 2015). For the Librispeech and TIMIT word dataset with | Y | = 224 , spoken words are extracted from Librispeech (Vassil et al., 2015) 460-hour train-clean TIMIT  Token F1  NMI  Boundary F1 (Yusuf et al., 2020) -40.1  0 . subset, resulting in a dataset of about 6 hours and 0.1 hours; for Librispeech and TIMIT word dataset with | Y | = 524 and | Y | = 824 , we supplement the dataset with the speech for the top 300 frequent words and top 600 frequent words respectively (ex-cluding the visual words) in Librispeech, resulting in datasets of about 15 and 21 hours. For Mboshi dataset, we found only about 20 actual words occur more than 100 times, so instead we use n -grams with either n  3 (all except uniand bi-grams) or n  2 (all except unigrams) that occur more than 100 times as words, resulting in a vocabulary size of 161 and 377 respectively. Note that the amount of labeled data we need is much lower than previous works (Yusuf et al., 2020): around 30 hours, (Feng et al., 2021b): around 600 hours) and the vocabulary size used is much smaller than the total vocabulary size in the language. More details of the sets can be found in Appendix B. We also test our 8032 models on two standard phoneme discovery benchmarks, which contain whole-sentence utterances with many words unseen during training. The first dataset is TIMIT (Garofolo et al., 1993), an English corpus consisting of about 5 hours speech and Mboshi (Godard et al., 2017), which contains about 2.4 hours speech from a low-resource language. For both datasets, we follow the split in (Yusuf et al., 2020), (Feng et al., 2021b) Baselines For phoneme discovery from segmented words, we compare our model (IQ) to four baselines. The first two baselines use continuous representation: the CPC+k-means model performs k-means clustering on the segment-level CPC features, and the k-means model performs k-means clustering after the model is trained on the word recognition task. The last two baselines use discrete representations: the Gumbel variational information bottleneck (Alemi et al., 2017) (Gumbel VIB) is a neural model with a Gumbel softmax (Jang et al., 2016) layer to approximate the codebook assignment function z (  ) , and we set  = 0 . 001 and decay the temperature of the Gumbel softmax from 1 to 0 . 1 linearly for the first 300000 steps, keeping it at 0 . 1 afterwards, which works best in our experiments; the deterministic information bottleneck (DIB), a generalization of (Strouse and Schwab, 2016) for continuous feature variable X , which assumes the same deterministic relation between speech X and codebook unit Z as ours, but optimizes the models in a pipeline fashion (first the speech encoder and then the quantizer) by performing clustering on the learned conditional distributions. The CPC features used are trained in a self-supervised fashion on the 960-hour LibriSpeech dataset and released by (Nguyen et al., 2020). All models share the same speech encoder as IQ. For the whole-sentence datasets, we compare our models to three phoneme discovery systems, namely, the unsupervised H-SHMM trained with multilingual speech (Yusuf et al., 2020), the ResDAVEnet-VQ (Harwath et al., 2020) with visual supervision and the TDNN-f system by (Feng et al., 2021b) trained with multilingual speech. To study how well our model performs in extreme low-resource speech recognition compared to other neural speech representation learning models, we compare our models to wav2vec (Schneider et al., 2019), wav2vec 2.0 (Baevski et al., 2020) (small, trained on the 960-hour LibriSpeech), vq-wav2vec with Gumbel softmax and k-means as discretiza-tion strategies (Baevski et al., 2019), CPC (van den Oord et al., 2019) and VQ-CPC (van Niekerk et al., 2020), using the pretrained models released by the authors. Implementation details of the baselines and our models are in Appendix C. Evaluation metrics Standard metrics are used such as NMI and boundary F1 for the quality of codebook and segmentation respectively with the same implementation as in prior works (Yusuf et al., 2020; Feng et al., 2021b). In addition, token F1 (Dunbar et al., 2017) is also reported. To examine the benefit of using our discovered phoneme inventory for low-resource speech recognition, we also evaluate using equivalent phone error rate (equiv. PER: Ondel et al. 2019). This metric can be viewed as a proxy for phone error rate (PER) applicable beyond supervised speech recognizers. The results on visual word-only test sets of Flickr audio and Librispeech are shown in Table 1. On both datasets, IQ outperforms both Gumbel VIB and DIB in terms of all metrics, especially on Flickr audio, which has more phonemes than Librispeech and a larger test set. Moreover, the performance of IQ is very robust to the codebook size, achieving good results even when the codebook size is very different from the size of the true phoneme inventory, suggesting our theory may be able to work with a relaxed Assumption 4. 6.2 Sentence-level Phoneme Discovery The results on TIMIT and Mboshi are shown in Table 2 and Table 3a respectively. On TIMIT, our model is able to outperform the visually grounded baseline (Harwath et al., 2020) for all training vocabulary, and all three baselines for | Y | = 524 and | Y | = 824 with and without gold segmentation in terms of all three metrics. Further, we also empirically verify the sample complexity bound in Theorem 1 as IQ performs better in Token F1 and NMI as the training vocabulary size get larger, which generally increases the JS divergence. On Mboshi, IQ with CPC feature consistently outpeforms (Feng et al., 2021b) in token F1 and boundary F1, and IQ with CPC+BNF features consistently outperform (Feng et al., 2021b) in all three metrics under various level of word supervision. The performance of our model on Mboshi compared with other neural self-supervised models are shown in Table 3b. We found that IQ outperforms the best self-supervised model, CPC+k-means in equiv. PER by 34% and 20% absolute with and without gold segmentation respectively and 12% absolute in terms of boundary F1, suggesting that IQ is able to learn consistent phoneme-like sequence useful for zero-resource or extremely low-resource speech recognition. Effect of segmentation and codebook size The use of unsupervised phoneme segmentation deteriorates the NMI by about 18% and 28% absolute on TIMIT and Mboshi respectively for our models since the distributional property of phonemes does not apply exactly to non-phoneme segments. On the other hand, in Appendix F we show that the quality of codeword assignments by IQs is very robust against varying codebook size, after experimenting with codebook size from 30 to 70 on TIMIT and Mboshi. Multilingual and word supervision are complimentary In all vocabulary sizes, concatenating the multilingual BNF from (Feng et al., 2021b) to the CPC output representation from the segmental speech encoder in Figure 2 significantly improves token F1 and NMI to allow our best models to outperform baselines in all three metrics. IQ codebook resembles true phonemes From Figure 3b, we observe that the codeword assignments by IQ correlates well with the actual phonemes, but tends to confuse the most between phonemes within the same manner class, such as nasals /n/ and /m/. This is also confirmed by the t-SNE plot in Figure 3a, where the embeddings of most manner classes are well-clustered, except for related manner classes such as affricate and fricative, or glide and vowel. Further, from the examples shown in Figure 4, we can see that IQ is not only better at grouping segments of the same 8034 phonemes but also at detecting segment boundaries than the baselines. Also, across different examples, IQ assign the same codes to phonemes such as /a/ (31) and /s/ (7) more consistently than other models do. Please check Appendix G for more speech examples. Limitation While our theory predicts that with gold segmentation, the TER of IQ is asymptotically zero, in practice TER is nonzero due to the violation of Assumption 4, i.e., the phonemes are not uniformly distributed for languages such as Mboshi. As a result, the model often discards information of the rare phonemes by merging them into a more frequent phoneme cluster. Evidently, from Figure 5, where we use ABX accuracy (Munson and Gardner, 1950) to score how reliable the IQ codebook can identify segments of the same phoneme, we observe a strong correlation is observed between ABX accuracy and the frequency of the phonemes. Motivated by the linguistic definition of phonemes, we propose information quantizer (IQ), a new neural network model for self-supervised phoneme discovery that can take advantage of word-level supervision. We demonstrate in two ways that word-level supervision is beneficial for phoneme inventory discovery: theoretically, we prove that IQ can achieve zero token error rate asymptotically with the help of word labels; empirically, we show that IQ out-performs various speech-only algorithms in phoneme discovery tasks under both simulated (En-glish) and realistic (Mboshi) low-resource settings. In the future, we would like to apply the discovered phoneme inventory to develop better low-resource speech technologies such speech translation and speech synthesis systems.
abstain abstain objective abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective abstain abstain objective objective objective other other other other other other other other other other other other other other other other other other other method method other abstain abstain other method abstain other abstain abstain other abstain other method abstain abstain other abstain abstain abstain other abstain abstain other abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain method method method method abstain method abstain method abstain abstain method abstain abstain method method method abstain method abstain abstain abstain result result result abstain result result abstain abstain result result result abstain result abstain abstain abstain abstain result objective objective objective