Answers.Body: <p>This is to get the gradient to &quot;skip&quot; the quantization part.</p>
  <p>The trick implements the red arrow in the <a href="https://arxiv.org/abs/1711.00937" rel="nofollow noreferrer">original paper</a>'s diagram:</p>
  <p><a href="https://i.sstatic.net/HmVaG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HmVaG.png" alt="VQ-VAE architecture, with red arrow signifying gradient skipping from quantized code vector to encoder output" /></a></p>
  <h3>Simplified example: Rounding</h3>
  <p>Let's simplify this a bit and imagine we want to use rounding in our architecture:</p>
  <pre><code>import torch
  x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  r = z.sum()
  </code></pre>
  <p>The graph (<code>torchviz.make_dot</code>) looks like this:</p>
  <p><a href="https://i.sstatic.net/WscOq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WscOq.png" alt="enter image description here" /></a></p>
  <p>We can look at the output:</p>
  <pre><code>r 
  # tensor(6., grad_fn=&lt;SumBackward0&gt;)
  </code></pre>
  <p>All is looking good. However, when we try to compute the gradient, we get a tensor of zeroes:</p>
  <pre><code>r.backward()
  x.grad
  # tensor([0., 0.])
  </code></pre>
  <p>This makes sense: the rounding function has derivative zero almost everywhere:</p>
  <p><a href="https://i.sstatic.net/xyWXH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xyWXH.png" alt="enter image description here" /></a></p>
  <p>However, it also means we cannot train our network.</p>
  <p>To circumvent that, we could simply tell the gradient to skip the rounding network. To do that, we use <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.detach.html" rel="nofollow noreferrer"><code>detach</code></a>, a function that tells PyTorch to detach a vector from the computational graph.</p>
  <pre><code>x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  z = y + (z - y).detach() # Detach everything between z and y, including z
  r = z.sum()
  </code></pre>
  <p>We still get the same answer for <span class="math-container">$r$</span></p>
  <pre><code>r 
  # tensor(6., grad_fn=&lt;SumBackward0&gt;)
  </code></pre>
  <p>But we now also get reasonable gradients</p>
  <pre><code>r.backward()
  x.grad
  # tensor([2., 2.])
  </code></pre>
  <p>Thanks to the fact that the gradient now &quot;skips&quot; the rounding part:</p>
  <p><a href="https://i.sstatic.net/vGU4O.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vGU4O.png" alt="enter image description here" /></a></p>
  <h3>Back to VQ-VAE</h3>
  <p>In VQ-VAE, we are replacing the output of the encoder (the <code>input</code> to the <a href="http://www.mqasem.net/vectorquantization/vq.html" rel="nofollow noreferrer">vector quantization</a> layer) with the <strong>code vector</strong>. In some sense, this similar to &quot;rounding&quot;, only this time we're &quot;rounding&quot; the encoder's output to the nearest code vector.</p>
  <p><a href="https://i.sstatic.net/vcN7P.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vcN7P.png" alt="enter image description here" /></a></p>
  <p>We therefore run into the same problem. Here's what the authors say about it</p>
  <blockquote>
  <p>Note that there is no real gradient defined for [the quantization], however we approximate the gradient similar to the straight-through estimator and just copy gradients from decoder input <span class="math-container">$z_q(x)$</span> to encoder output <span class="math-container">$z_e(x)$</span></p>
  </blockquote>
  <p>which is precisely what</p>
  <pre><code>quantized = inputs + (quantized - inputs).detach()
  </code></pre>
  <p>does in the notebook.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-02T10:33:27.800Z
Answers.LastActivityDate: 2021-12-02T15:23:08.937Z
Answers.LastEditDate: 2021-12-02T15:23:08.937Z
Answers.UserDisplayName: Dahn
Body: <p>The VQ-VAE implimentation:<a href="https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb" rel="nofollow noreferrer">https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb</a></p>
  <pre><code>quantized = inputs + (quantized - inputs).detach()
  </code></pre>
  <p>Why are we subtracting and adding input to quantized result?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-11T19:02:51.263Z
Comments.Text: Numerical stability.
Comments.UserDisplayName: Abhishek Verma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-12T20:09:04.843Z
Comments.Text: @AbhishekVerma it would be really helpful if you could please explain this with a little detail....kind of a newbie here.
Comments.UserDisplayName: Vortex
Tags: <variational-autoencoder>
Title: In VQ-VAE code what does this line of code signify?
UserDisplayName: Vortex
ThreadText: # In VQ-VAE code what does this line of code signify?
  
  (Asked by: username_0 on 2021-03-11)
  
  The VQ-VAE implimentation:<https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb>
  
  ```
  quantized = inputs + (quantized - inputs).detach()
  
  ```
  
  Why are we subtracting and adding input to quantized result?
  
  **Tags:** variational-autoencoder
  
  > **Comments:**
  > **username_1**: Numerical stability.  
  > **username_0**: @username_1 it would be really helpful if you could please explain this with a little detail....kind of a newbie here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-12-02. Score: 4
  
  This is to get the gradient to "skip" the quantization part.
  
  The trick implements the red arrow in the [original paper](https://arxiv.org/abs/1711.00937)'s diagram:
  
  [![VQ-VAE architecture, with red arrow signifying gradient skipping from quantized code vector to encoder output](https://i.sstatic.net/HmVaG.png)](https://i.sstatic.net/HmVaG.png)
  
  ### Simplified example: Rounding
  
  Let's simplify this a bit and imagine we want to use rounding in our architecture:
  
  ```
  import torch
  x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  r = z.sum()
  
  ```
  
  The graph (`torchviz.make_dot`) looks like this:
  
  [![enter image description here](https://i.sstatic.net/WscOq.png)](https://i.sstatic.net/WscOq.png)
  
  We can look at the output:
  
  ```
  r 
  # tensor(6., grad_fn=<SumBackward0>)
  
  ```
  
  All is looking good. However, when we try to compute the gradient, we get a tensor of zeroes:
  
  ```
  r.backward()
  x.grad
  # tensor([0., 0.])
  
  ```
  
  This makes sense: the rounding function has derivative zero almost everywhere:
  
  [![enter image description here](https://i.sstatic.net/xyWXH.png)](https://i.sstatic.net/xyWXH.png)
  
  However, it also means we cannot train our network.
  
  To circumvent that, we could simply tell the gradient to skip the rounding network. To do that, we use [`detach`](https://pytorch.org/docs/stable/generated/torch.Tensor.detach.html), a function that tells PyTorch to detach a vector from the computational graph.
  
  ```
  x = torch.tensor([1.1, 2.1], requires_grad=True)
  y = 2*x
  z = torch.round(y)
  z = y + (z - y).detach() # Detach everything between z and y, including z
  r = z.sum()
  
  ```
  
  We still get the same answer for $r$
  
  ```
  r 
  # tensor(6., grad_fn=<SumBackward0>)
  
  ```
  
  But we now also get reasonable gradients
  
  ```
  r.backward()
  x.grad
  # tensor([2., 2.])
  
  ```
  
  Thanks to the fact that the gradient now "skips" the rounding part:
  
  [![enter image description here](https://i.sstatic.net/vGU4O.png)](https://i.sstatic.net/vGU4O.png)
  
  ### Back to VQ-VAE
  
  In VQ-VAE, we are replacing the output of the encoder (the `input` to the [vector quantization](http://www.mqasem.net/vectorquantization/vq.html) layer) with the **code vector**. In some sense, this similar to "rounding", only this time we're "rounding" the encoder's output to the nearest code vector.
  
  [![enter image description here](https://i.sstatic.net/vcN7P.png)](https://i.sstatic.net/vcN7P.png)
  
  We therefore run into the same problem. Here's what the authors say about it
  
  > Note that there is no real gradient defined for [the quantization], however we approximate the gradient similar to the straight-through estimator and just copy gradients from decoder input $z\_q(x)$ to encoder output $z\_e(x)$
  
  which is precisely what
  
  ```
  quantized = inputs + (quantized - inputs).detach()
  
  ```
  
  does in the notebook.
  
  
  ---
  
  