Answers.Body: <p>This is to get the gradient to &quot;skip&quot; the quantization part.</p>
  <p>The trick implements the red arrow in the <a href="https://arxiv.org/abs/1711.00937" rel="nofollow noreferrer">original paper</a>'s diagram:</p>
  <p><a href="https://i.sstatic.net/HmVaG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HmVaG.png" alt="VQ-VAE architecture, with red arrow signifying gradient skipping from quantized code vector to encoder output" /></a></p>
  <h3>Simplified example: Rounding</h3>
  <p>Let's simplify this a bit and imagine we want to use rounding in our architecture:</p>
  <pre><code>import torch
  x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  r = z.sum()
  </code></pre>
  <p>The graph (<code>torchviz.make_dot</code>) looks like this:</p>
  <p><a href="https://i.sstatic.net/WscOq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WscOq.png" alt="enter image description here" /></a></p>
  <p>We can look at the output:</p>
  <pre><code>r 
  # tensor(6., grad_fn=&lt;SumBackward0&gt;)
  </code></pre>
  <p>All is looking good. However, when we try to compute the gradient, we get a tensor of zeroes:</p>
  <pre><code>r.backward()
  x.grad
  # tensor([0., 0.])
  </code></pre>
  <p>This makes sense: the rounding function has derivative zero almost everywhere:</p>
  <p><a href="https://i.sstatic.net/xyWXH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xyWXH.png" alt="enter image description here" /></a></p>
  <p>However, it also means we cannot train our network.</p>
  <p>To circumvent that, we could simply tell the gradient to skip the rounding network. To do that, we use <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.detach.html" rel="nofollow noreferrer"><code>detach</code></a>, a function that tells PyTorch to detach a vector from the computational graph.</p>
  <pre><code>x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  z = y + (z - y).detach() # Detach everything between z and y, including z
  r = z.sum()
  </code></pre>
  <p>We still get the same answer for <span class="math-container">$r$</span></p>
  <pre><code>r 
  # tensor(6., grad_fn=&lt;SumBackward0&gt;)
  </code></pre>
  <p>But we now also get reasonable gradients</p>
  <pre><code>r.backward()
  x.grad
  # tensor([2., 2.])
  </code></pre>
  <p>Thanks to the fact that the gradient now &quot;skips&quot; the rounding part:</p>
  <p><a href="https://i.sstatic.net/vGU4O.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vGU4O.png" alt="enter image description here" /></a></p>
  <h3>Back to VQ-VAE</h3>
  <p>In VQ-VAE, we are replacing the output of the encoder (the <code>input</code> to the <a href="http://www.mqasem.net/vectorquantization/vq.html" rel="nofollow noreferrer">vector quantization</a> layer) with the <strong>code vector</strong>. In some sense, this similar to &quot;rounding&quot;, only this time we're &quot;rounding&quot; the encoder's output to the nearest code vector.</p>
  <p><a href="https://i.sstatic.net/vcN7P.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vcN7P.png" alt="enter image description here" /></a></p>
  <p>We therefore run into the same problem. Here's what the authors say about it</p>
  <blockquote>
  <p>Note that there is no real gradient defined for [the quantization], however we approximate the gradient similar to the straight-through estimator and just copy gradients from decoder input <span class="math-container">$z_q(x)$</span> to encoder output <span class="math-container">$z_e(x)$</span></p>
  </blockquote>
  <p>which is precisely what</p>
  <pre><code>quantized = inputs + (quantized - inputs).detach()
  </code></pre>
  <p>does in the notebook.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-02T10:33:27.800Z
Answers.LastActivityDate: 2021-12-02T15:23:08.937Z
Answers.LastEditDate: 2021-12-02T15:23:08.937Z
Answers.UserDisplayName: Dahn
Body: <p>The VQ-VAE implimentation:<a href="https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb" rel="nofollow noreferrer">https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb</a></p>
  <pre><code>quantized = inputs + (quantized - inputs).detach()
  </code></pre>
  <p>Why are we subtracting and adding input to quantized result?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-11T19:02:51.263Z
Comments.Text: Numerical stability.
Comments.UserDisplayName: Abhishek Verma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-12T20:09:04.843Z
Comments.Text: @AbhishekVerma it would be really helpful if you could please explain this with a little detail....kind of a newbie here.
Comments.UserDisplayName: Vortex
Tags: <variational-autoencoder>
Title: In VQ-VAE code what does this line of code signify?
UserDisplayName: Vortex
ThreadText: # In VQ-VAE code what does this line of code signify?
  
  (Asked by: username_0 on 2021-03-11)
  
  The VQ-VAE implimentation:https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb
  quantized = inputs + (quantized - inputs).detach()
  
  Why are we subtracting and adding input to quantized result?
  
  **Tags:** variational-autoencoder
  > **Comments:**
  > **username_1**: Numerical stability.  
  > **username_0**: @AbhishekVerma it would be really helpful if you could please explain this with a little detail....kind of a newbie here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-12-02. Score: 4
  
  This is to get the gradient to "skip" the quantization part.
  The trick implements the red arrow in the original paper's diagram:
  
  Simplified example: Rounding
  Let's simplify this a bit and imagine we want to use rounding in our architecture:
  import torch
  x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  r = z.sum()
  
  The graph (torchviz.make_dot) looks like this:
  
  We can look at the output:
  r 
  # tensor(6., grad_fn=)
  
  All is looking good. However, when we try to compute the gradient, we get a tensor of zeroes:
  r.backward()
  x.grad
  # tensor([0., 0.])
  
  This makes sense: the rounding function has derivative zero almost everywhere:
  
  However, it also means we cannot train our network.
  To circumvent that, we could simply tell the gradient to skip the rounding network. To do that, we use detach, a function that tells PyTorch to detach a vector from the computational graph.
  x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  z = y + (z - y).detach() # Detach everything between z and y, including z
  r = z.sum()
  
  We still get the same answer for $r$
  r 
  # tensor(6., grad_fn=)
  
  But we now also get reasonable gradients
  r.backward()
  x.grad
  # tensor([2., 2.])
  
  Thanks to the fact that the gradient now "skips" the rounding part:
  
  Back to VQ-VAE
  In VQ-VAE, we are replacing the output of the encoder (the input to the vector quantization layer) with the code vector. In some sense, this similar to "rounding", only this time we're "rounding" the encoder's output to the nearest code vector.
  
  We therefore run into the same problem. Here's what the authors say about it
  
  Note that there is no real gradient defined for [the quantization], however we approximate the gradient similar to the straight-through estimator and just copy gradients from decoder input $z_q(x)$ to encoder output $z_e(x)$
  
  which is precisely what
  quantized = inputs + (quantized - inputs).detach()
  
  does in the notebook.
  
  
  ---
  
  