Answers.Body: <p>I would slightly disagree with Ryan's answer: the fundamental frequency is mainly specific to a <em>speaker</em>. Sounds are defined by other frequency patterns.</p>
  <p>Vowels, for example, have two bands of energy in different frequency ranges, called <a href="https://en.wikipedia.org/wiki/Formant" rel="nofollow noreferrer"><em>formants</em></a>. Depending on the speaker, these will be in different actual frequency ranges, but the general pattern will remain the same. They are linked to the fundamental frequency <em>F0</em> in that it is multiples of <em>F0</em> which can be seen in a <a href="https://en.wikipedia.org/wiki/Spectrogram" rel="nofollow noreferrer">spectrogram</a>.</p>
  <p>There is obviously  lot of variation in the exact frequencies involved because of circumstances. Apart from different speakers with different physiology, having a cold also changes your frequency output. For speech recognition you need various mechanisms to map a sequence of frequency values to phonemes and then words. When I was studying phonetics in the 1990s, the upcoming tool was <a href="https://en.wikipedia.org/wiki/Hidden_Markov_model" rel="nofollow noreferrer">Hidden Markov Models</a>, and neural networks were being explored as well, but were still in their infancy.</p>
  <p>I'm not sure what currently is used for speech recognition, as I haven't done any phonetics for years, but I would expect NNs to be more prevalent these days.</p>
  <p>To summarise: there are some features in the frequency pattern of an utterance which give you an indication of the (language) sounds produced, whereas others reflect the physiology of the vocal tract (and are thus speaker-specific). But they are not unique, as there is no &quot;voice print&quot;, which was some time ago touted as a finger print equivalent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T08:25:01.447Z
Answers.LastActivityDate: 2023-07-07T08:25:01.447Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>A sound byte can be decomposed into a set of features which are distinguishable by a classifier. The most important feature extracted is the fundamental frequency. This is the lowest frequency function from approximating the windowed waveform with Fourier series. It is fairly unique for each phoneme. In addition, each phoneme can be represented as a series of &quot;frequency bins&quot; where the intensity is the height of each bin. These features are not only unique to each phoneme, but also to each speaker. Additionally, statical moments can be calculated for each window and used as features.</p>
  <p>A Recurrent Neural Network (there are many techniques) can be used to agglomerate features into a recognition model. The idea is to learn the probability of one set of features following another set in time.</p>
  <p>For an example, a 20 second sound byte of a speaker can be broken into 20 1-second windows of speech. Each window represents a phoneme or some part of speech (depending on window-size, speaker speed, etc). A Fourier transform is applied on a window to derive a feature set. The feature set is fed into an RNN to predict the speaker (or text-to-speech, TTS). There are many inter-related applications here.</p>
  <p>Hope this was helpful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-06T20:01:39.443Z
Answers.Text: "A sound byte can be decomposed into a set of features which are distinguishable by a classifier."   What If like to know is, which classifiers are relevant to distinguishing between different _speakers_ and which are relevant to distinguishing between different _messages_?  e.g. If had a recording of a person with a British accent accent saying the word "water" and another recording of a person with an American accent saying, which features of the data would recognize the different accents and which ones would "recognize" (for lack of a better term) that it's the same word being spoken?
Answers.UserDisplayName: Mikayla Eckel Cifrese
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-06T23:21:48.660Z
Answers.Text: Typically there are (at least) two stages. The first stage picks up features like phonemes and syllables. The second stage identifies specific words and picks up things like which homophone was meant (e.g. they're or their).
Answers.UserDisplayName: Bruce Adams
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T16:41:09.127Z
Answers.Text: @MikaylaEckelCifrese the same models can be used for both. RNNs, LSTMs, transformers, etc can all be used for either task. The difference is in the target for training which comes down to calculating loss for updating the network. Is the network "output" a vector that represents a speaker or a word. This is training dependent. I would recommend becoming more familiar with ML in general before tackling this specific task.
Answers.UserDisplayName: Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T19:16:11.260Z
Answers.Text: @Ryan, but about when the software needs to recognize both speaker specific and content specific information?  Also, I don't currently have any intention of trying to make a speech recognition system, as that's definitely beyond my current skill level -- I'm just curious about how such software works at a high level.
Answers.UserDisplayName: Mikayla Eckel Cifrese
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-12T20:50:13.440Z
Answers.Text: @MikaylaEckelCifrese, Many features can be re-used for both tasks, however most models in practice will have common feature extraction layers and then have models that perform specific tasks. There might be several layers on top of the base extraction for one task, and separate layers for another task. The software has multiple models with some common layers.
Answers.UserDisplayName: Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-06T18:53:53.793Z
Answers.LastActivityDate: 2023-07-06T18:53:53.793Z
Answers.UserDisplayName: Ryan
Body: <p>Some background: I'm an EE major and data science minor, so I have a basic understanding of machine learning - I had a one semester course on it where we covered some of the most commonly used algorithms such as Naive Bayes' and K-nearest-neighbors, as well as the basics of how artificial neural networks work.  I've also had EE classes on signal processing, so I'm familiar with the Fourier transform and the like.  So I have a general idea of how different algorithms can, for lack of a better term, &quot;pick up on&quot; patterns in the training data and make predictions on the testing data, which will hopefully be reasonably accurate.  And I have a general idea of how signals can be represented as discrete binary sequences which could then be used to train an ML model.</p>
  <p>What I want to know is, what sorts of patterns, in the mathematical sense, in audio recordings of speech specifically, do these algorithms find?  e.g., What allows the Alexa app to both recognize when I'm speaking vs when it's my grandma speaking, and yet still understand the same commands from both of us?  What, mathematically, makes, e.g., a recording of me saying &quot;Alexa play music&quot; both distinguishable from a recording of a different person saying the same command, and yet still recognizably contain the same information of the command?  In other words, how do the algorithms extract and match the speaker-specific and content-specific information from the speech signals?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-05T22:18:07.087Z
Comments.Text: Is there a reason you find this odd? After all, you understand yourself, your grandmother, etc.
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-06T05:33:33.193Z
Comments.Text: @Dave, it's not that I find it _odd_ -- it's just that I don't understand _how_ it works and I want to understand.   Heck, I don't understand how our brains are able to do it either, but I don't really know anything about neurology beyond the basic facts most people know.  I do know the basics of machine learning well enough that I'm hoping I can get a high level understanding of the parts of the math behind speech recognition that distinguish between speaker specific and content specific information.  I can't think of a particular reason I'm curious about it -- I just am.
Comments.UserDisplayName: Mikayla Eckel Cifrese
Tags: <machine-learning><math><algorithm><audio-processing><speech-recognition>
Title: How is speech recognition software able to distinguish between different speakers and yet still understand them all?
UserDisplayName: Mikayla Eckel Cifrese
ThreadText: # How is speech recognition software able to distinguish between different speakers and yet still understand them all?
  
  (Asked by: username_0 on 2023-07-05)
  
  Some background: I'm an EE major and data science minor, so I have a basic understanding of machine learning - I had a one semester course on it where we covered some of the most commonly used algorithms such as Naive Bayes' and K-nearest-neighbors, as well as the basics of how artificial neural networks work. I've also had EE classes on signal processing, so I'm familiar with the Fourier transform and the like. So I have a general idea of how different algorithms can, for lack of a better term, "pick up on" patterns in the training data and make predictions on the testing data, which will hopefully be reasonably accurate. And I have a general idea of how signals can be represented as discrete binary sequences which could then be used to train an ML model.
  
  What I want to know is, what sorts of patterns, in the mathematical sense, in audio recordings of speech specifically, do these algorithms find? e.g., What allows the Alexa app to both recognize when I'm speaking vs when it's my grandma speaking, and yet still understand the same commands from both of us? What, mathematically, makes, e.g., a recording of me saying "Alexa play music" both distinguishable from a recording of a different person saying the same command, and yet still recognizably contain the same information of the command? In other words, how do the algorithms extract and match the speaker-specific and content-specific information from the speech signals?
  
  **Tags:** machine-learning, math, algorithm, audio-processing, speech-recognition
  > **Comments:**
  > **username_1**: Is there a reason you find this odd? After all, you understand yourself, your grandmother, etc.  
  > **username_0**: @Dave, it's not that I find it _odd_ -- it's just that I don't understand _how_ it works and I want to understand.   Heck, I don't understand how our brains are able to do it either, but I don't really know anything about neurology beyond the basic facts most people know.  I do know the basics of machine learning well enough that I'm hoping I can get a high level understanding of the parts of the math behind speech recognition that distinguish between speaker specific and content specific information.  I can't think of a particular reason I'm curious about it -- I just am.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-07. Score: 2
  
  I would slightly disagree with Ryan's answer: the fundamental frequency is mainly specific to a *speaker*. Sounds are defined by other frequency patterns.
  
  Vowels, for example, have two bands of energy in different frequency ranges, called [*formants*](https://en.wikipedia.org/wiki/Formant). Depending on the speaker, these will be in different actual frequency ranges, but the general pattern will remain the same. They are linked to the fundamental frequency *F0* in that it is multiples of *F0* which can be seen in a [spectrogram](https://en.wikipedia.org/wiki/Spectrogram).
  
  There is obviously lot of variation in the exact frequencies involved because of circumstances. Apart from different speakers with different physiology, having a cold also changes your frequency output. For speech recognition you need various mechanisms to map a sequence of frequency values to phonemes and then words. When I was studying phonetics in the 1990s, the upcoming tool was [Hidden Markov Models](https://en.wikipedia.org/wiki/Hidden_Markov_model), and neural networks were being explored as well, but were still in their infancy.
  
  I'm not sure what currently is used for speech recognition, as I haven't done any phonetics for years, but I would expect NNs to be more prevalent these days.
  
  To summarise: there are some features in the frequency pattern of an utterance which give you an indication of the (language) sounds produced, whereas others reflect the physiology of the vocal tract (and are thus speaker-specific). But they are not unique, as there is no "voice print", which was some time ago touted as a finger print equivalent.
  
  
  ---
  
  ### Answer by username_3 on 2023-07-06. Score: 1
  
  A sound byte can be decomposed into a set of features which are distinguishable by a classifier. The most important feature extracted is the fundamental frequency. This is the lowest frequency function from approximating the windowed waveform with Fourier series. It is fairly unique for each phoneme. In addition, each phoneme can be represented as a series of "frequency bins" where the intensity is the height of each bin. These features are not only unique to each phoneme, but also to each speaker. Additionally, statical moments can be calculated for each window and used as features.
  
  A Recurrent Neural Network (there are many techniques) can be used to agglomerate features into a recognition model. The idea is to learn the probability of one set of features following another set in time.
  
  For an example, a 20 second sound byte of a speaker can be broken into 20 1-second windows of speech. Each window represents a phoneme or some part of speech (depending on window-size, speaker speed, etc). A Fourier transform is applied on a window to derive a feature set. The feature set is fed into an RNN to predict the speaker (or text-to-speech, TTS). There are many inter-related applications here.
  
  Hope this was helpful.
  
  > **Comments:**
  > **username_0**: "A sound byte can be decomposed into a set of features which are distinguishable by a classifier."   What If like to know is, which classifiers are relevant to distinguishing between different _speakers_ and which are relevant to distinguishing between different _messages_?  e.g. If had a recording of a person with a British accent accent saying the word "water" and another recording of a person with an American accent saying, which features of the data would recognize the different accents and which ones would "recognize" (for lack of a better term) that it's the same word being spoken?  
  > **username_4**: Typically there are (at least) two stages. The first stage picks up features like phonemes and syllables. The second stage identifies specific words and picks up things like which homophone was meant (e.g. they're or their).  
  > **username_3**: @MikaylaEckelCifrese the same models can be used for both. RNNs, LSTMs, transformers, etc can all be used for either task. The difference is in the target for training which comes down to calculating loss for updating the network. Is the network "output" a vector that represents a speaker or a word. This is training dependent. I would recommend becoming more familiar with ML in general before tackling this specific task.  
  > **username_0**: @Ryan, but about when the software needs to recognize both speaker specific and content specific information?  Also, I don't currently have any intention of trying to make a speech recognition system, as that's definitely beyond my current skill level -- I'm just curious about how such software works at a high level.  
  > **username_3**: @MikaylaEckelCifrese, Many features can be re-used for both tasks, however most models in practice will have common feature extraction layers and then have models that perform specific tasks. There might be several layers on top of the base extraction for one task, and separate layers for another task. The software has multiple models with some common layers.  
  
  ---
  
  