Body: <p>I'm studying the series of Wav2Vec papers, in particular, the <a href="https://arxiv.org/pdf/1910.05453.pdf" rel="nofollow noreferrer">vq-wav2vec</a> and <a href="https://arxiv.org/pdf/2006.11477.pdf" rel="nofollow noreferrer">wav2vec 2.0</a>, and have a problem understanding some details about the quantization procedure.</p>
  <p>The broader context is this: they use raw audio and first convert it to &quot;features&quot; <span class="math-container">$z$</span> via a convolutional network. Then they project any feature <span class="math-container">$z$</span> to a &quot;quantized&quot; element <span class="math-container">$\hat{z}$</span> from a given finite codebook (or concatenation of finitely many finite codebooks). To find <span class="math-container">$\hat{z}$</span>, they compute scores <span class="math-container">$l_j$</span> for each codebook entry <span class="math-container">$v_j$</span>, convert these scores to Gumbel-Softmax probabilities <span class="math-container">$p_j$</span> (using a formula which is not deterministic, the formula involves random choices of some numbers from some distribution) and then use these probabilities <span class="math-container">$p_j$</span> to choose <span class="math-container">$\hat{z}$</span>. Further stages of the pre-training pipeline are trained to predict <span class="math-container">$\hat{z}'s$</span> by either predicting &quot;future&quot; from the &quot;past&quot;, or &quot;reconstructing masked segments&quot;.</p>
  <p>My question is this is about this sentence:</p>
  <blockquote>
  <p>During the forward pass, <span class="math-container">$i = \text{argmax}_j p_j$</span> and in the
  backward pass, the true gradient of the Gumbel-Softmax outputs is
  used.</p>
  </blockquote>
  <ul>
  <li>I have trouble seeing what exactly is happening in the loss function and back-propagation. Could someone please help me to break this down into details?</li>
  </ul>
  <hr />
  <p>My mental attempts to make sense out of it (I'm using the notation <span class="math-container">$\hat{z}$</span> for quantized vectors, in the second paper they use <span class="math-container">$q$</span>)</p>
  <p>(1) I would say that during the forward pass, in the Gumbel-Softmax, random variables from the Gumbel-distribution <span class="math-container">$n_j$</span> are sampled every time (for every training example) to compute the Gumbel-softmax probabilities <span class="math-container">$p_j$</span>.</p>
  <p>(1a) In the back-propagation, these <span class="math-container">$n_j$</span>'s are kept constant, and <span class="math-container">$p_j$</span> is treated as a function of <span class="math-container">$l_j's$</span> only.</p>
  <p>(2) The loss function has 2 parts here, Contrastive loss and Diversity loss.</p>
  <p>(2a) Based on the description, I would say that in the contrastive loss, the &quot;sampled&quot; vectors <span class="math-container">$\hat{z}_j$</span> are used, and probabilities never appear (even not in back-propagation of this part of the loss).</p>
  <p>(2b) I would believe that in the gradient of the Diversity loss, which only uses probabilities <span class="math-container">$p_{g,v}$</span>, that here the gradient or the loss actually <strong>is</strong> used, as this is responsible for maximizing the entropy. This part of the gradient probably does not use the sampled values <span class="math-container">$\hat{z}_j$</span>.</p>
  <p>Is this approximately correct?</p>
  <p>If yes, then I still fail to understand what exactly is happening in the vq-wav2vec paper. The sentence</p>
  <blockquote>
  <p>During the forward pass, <span class="math-container">$i = \text{argmax}_j p_j$</span> and in the
  backward pass, the true gradient of the Gumbel-Softmax outputs is
  used.</p>
  </blockquote>
  <p>is there as well, but I cannot see any part of the loss function (in this paper) where the probabilities are explicitly used (such as the diversity loss).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T13:26:13.823Z
Comments.Text: Hello. Although the second question is clearly related to the first one, I would suggest that you ask it in a separate post so that people can [focus on 1 question/problem at a time](https://meta.stackexchange.com/a/39224).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T19:33:55.147Z
Comments.Text: Thanks @nbro I only left the second question, as it is kind of clear to me now that the codebook is learned.
Comments.UserDisplayName: Peter Franek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T19:38:59.517Z
Comments.Text: Ok, thanks. But can you be more specific about what your question now is? What's confusing about this statement that you want to be clarified? In other words, try to ask a more specific question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T19:58:49.140Z
Comments.Text: That;s harder but will try :-)
Comments.UserDisplayName: Peter Franek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-22T20:19:04.000Z
Comments.Text: @nbro Can you take a look now pls? I provided more of my thoughts, although the clarity/readability is probably worse than before.
Comments.UserDisplayName: Peter Franek
Tags: <deep-learning><papers><audio-processing>
Title: Understanding gumbel-softmax backpropagation in Wav2Vec papers
UserDisplayName: Peter Franek
ThreadText: # Understanding gumbel-softmax backpropagation in Wav2Vec papers
  
  (Asked by: username_0 on 2021-10-19)
  
  I'm studying the series of Wav2Vec papers, in particular, the [vq-wav2vec](https://arxiv.org/pdf/1910.05453.pdf) and [wav2vec 2.0](https://arxiv.org/pdf/2006.11477.pdf), and have a problem understanding some details about the quantization procedure.
  
  The broader context is this: they use raw audio and first convert it to "features" $z$ via a convolutional network. Then they project any feature $z$ to a "quantized" element $\hat{z}$ from a given finite codebook (or concatenation of finitely many finite codebooks). To find $\hat{z}$, they compute scores $l\_j$ for each codebook entry $v\_j$, convert these scores to Gumbel-Softmax probabilities $p\_j$ (using a formula which is not deterministic, the formula involves random choices of some numbers from some distribution) and then use these probabilities $p\_j$ to choose $\hat{z}$. Further stages of the pre-training pipeline are trained to predict $\hat{z}'s$ by either predicting "future" from the "past", or "reconstructing masked segments".
  
  My question is this is about this sentence:
  
  > During the forward pass, $i = \text{argmax}\_j p\_j$ and in the
  > backward pass, the true gradient of the Gumbel-Softmax outputs is
  > used.
  
  * I have trouble seeing what exactly is happening in the loss function and back-propagation. Could someone please help me to break this down into details?
  
  ---
  
  My mental attempts to make sense out of it (I'm using the notation $\hat{z}$ for quantized vectors, in the second paper they use $q$)
  
  (1) I would say that during the forward pass, in the Gumbel-Softmax, random variables from the Gumbel-distribution $n\_j$ are sampled every time (for every training example) to compute the Gumbel-softmax probabilities $p\_j$.
  
  (1a) In the back-propagation, these $n\_j$'s are kept constant, and $p\_j$ is treated as a function of $l\_j's$ only.
  
  (2) The loss function has 2 parts here, Contrastive loss and Diversity loss.
  
  (2a) Based on the description, I would say that in the contrastive loss, the "sampled" vectors $\hat{z}\_j$ are used, and probabilities never appear (even not in back-propagation of this part of the loss).
  
  (2b) I would believe that in the gradient of the Diversity loss, which only uses probabilities $p\_{g,v}$, that here the gradient or the loss actually **is** used, as this is responsible for maximizing the entropy. This part of the gradient probably does not use the sampled values $\hat{z}\_j$.
  
  Is this approximately correct?
  
  If yes, then I still fail to understand what exactly is happening in the vq-wav2vec paper. The sentence
  
  > During the forward pass, $i = \text{argmax}\_j p\_j$ and in the
  > backward pass, the true gradient of the Gumbel-Softmax outputs is
  > used.
  
  is there as well, but I cannot see any part of the loss function (in this paper) where the probabilities are explicitly used (such as the diversity loss).
  
  **Tags:** deep-learning, papers, audio-processing
  > **Comments:**
  > **username_1**: Hello. Although the second question is clearly related to the first one, I would suggest that you ask it in a separate post so that people can [focus on 1 question/problem at a time](https://meta.stackexchange.com/a/39224).  
  > **username_0**: Thanks @nbro I only left the second question, as it is kind of clear to me now that the codebook is learned.  
  > **username_1**: Ok, thanks. But can you be more specific about what your question now is? What's confusing about this statement that you want to be clarified? In other words, try to ask a more specific question.  
  > **username_0**: That;s harder but will try :-)  
  > **username_0**: @nbro Can you take a look now pls? I provided more of my thoughts, although the clarity/readability is probably worse than before.  
  
  ---
  
  ## 0 answers
  
  