Answers.Body: <p><strong>Note:</strong> This answer just uses images and pixels as an analogy for simplicity. It's not formal.</p>
  <p>Some notes about <strong>VQ-VAE</strong>:</p>
  <ol>
  <li>In the paper, they used <strong>PixelCNN</strong> to learn the prior.
  Remember, <strong>PixelCNN</strong> is a good choice for images of discrete values (e.g. 256 value, 3 channels).</li>
  <li>The discrete latent variables are just the indices of the
  embedding vectors. For example, you can put your embedding vectors
  in an array.</li>
  </ol>
  <p>For a single input, the number of the output channels of the encoder before quantization equals the dimensionality of the embedding vectors. For an analogy, the output of the encoder is like an image but with a number of channels that equals the dimensions in the embedding space. With that, each pixel is a vector in the space of the embedding vectors. The quantization is done on each pixel by mapping each pixel to the nearest embedding vector <span class="math-container">$e$</span>.</p>
  <p>Now, each pixel can be represented by a single integer number which equals the index of its nearest embedding (what you call the discrete latent variables). Thus, you have two representations for the quantized output, one is a large tensor with many channels (embedding vectors), and another is a simple &quot;discrete image&quot; with one channel (discrete latent variables).</p>
  <p>The one-channel &quot;discrete images&quot; are used to train <strong>PixelCNN</strong>. This is great because they are small-size images. while the large tensor with the embedding vectors is used for the decoder as it holds the information needed for reconstruction.</p>
  <p>So, the discrete latent variables are used later to learn the prior and the embedding vectors are used as inputs for the decoder.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T12:11:24.683Z
Answers.LastActivityDate: 2023-02-01T14:48:04.227Z
Answers.LastEditDate: 2023-02-01T14:48:04.227Z
Answers.UserDisplayName: Badran
Body: <p>I'm trying to implement the <a href="https://arxiv.org/abs/1711.00937" rel="nofollow noreferrer">VQ-VAE model</a>. In there, a continuous variable <span class="math-container">$x$</span> is encoded in an array <span class="math-container">$z$</span> of discrete latent variables <span class="math-container">$z_i$</span> that are mapped each to an embedding vector <span class="math-container">$e_i$</span>. These vectors can be used to generate an <span class="math-container">$\hat{x}$</span> that approximates <span class="math-container">$x$</span>. </p>
  
  <p>In order to obtain a reasonable generative model <span class="math-container">$p_\theta(x)=\int p_\theta(x|z)p(z)$</span>, one needs to learn the prior distribution of the code <span class="math-container">$z$</span>. However, it is not clear in this paper, or its <a href="https://arxiv.org/abs/1906.00446" rel="nofollow noreferrer">second version</a>, what should be the input of the network that learns the prior. Is it <span class="math-container">$z=[z_i]$</span> or <span class="math-container">$e=[e_i]$</span>? The paper seems to indicate that it is <span class="math-container">$z$</span>, but if that's the case, I don't understand how I should encode <span class="math-container">$z$</span> properly. For example, a sample of <span class="math-container">$z$</span> might be an <span class="math-container">$n\times n$</span> matrix with discrete values between <span class="math-container">$0$</span> and <span class="math-container">$511$</span>. It is not reasonable to me to use a one-hot encoding, nor to simply use the discrete numbers as if they were continuous, given that there is no defined order for them. On the other hand, using <span class="math-container">$e$</span> doesn't have this problem since it represents a matrix with continuous entries, but then the required network would be much bigger.</p>
  
  <p>So, what should be the input for the prior model? <span class="math-container">$z$</span> or <span class="math-container">$e$</span>? If it is <span class="math-container">$z$</span>, how should I represent it? If it is <span class="math-container">$e$</span>, how should I implement the network?</p>
  
Tags: <machine-learning><generative-model><variational-autoencoder>
Title: What is the input for the prior model of VQ-VAE?
UserDisplayName: Diego Gomez
ThreadText: # What is the input for the prior model of VQ-VAE?
  
  (Asked by: username_0 on 2019-12-22)
  
  I'm trying to implement the [VQ-VAE model](https://arxiv.org/abs/1711.00937). In there, a continuous variable $x$ is encoded in an array $z$ of discrete latent variables $z\_i$ that are mapped each to an embedding vector $e\_i$. These vectors can be used to generate an $\hat{x}$ that approximates $x$.
  
  In order to obtain a reasonable generative model $p\_\theta(x)=\int p\_\theta(x|z)p(z)$, one needs to learn the prior distribution of the code $z$. However, it is not clear in this paper, or its [second version](https://arxiv.org/abs/1906.00446), what should be the input of the network that learns the prior. Is it $z=[z\_i]$ or $e=[e\_i]$? The paper seems to indicate that it is $z$, but if that's the case, I don't understand how I should encode $z$ properly. For example, a sample of $z$ might be an $n\times n$ matrix with discrete values between $0$ and $511$. It is not reasonable to me to use a one-hot encoding, nor to simply use the discrete numbers as if they were continuous, given that there is no defined order for them. On the other hand, using $e$ doesn't have this problem since it represents a matrix with continuous entries, but then the required network would be much bigger.
  
  So, what should be the input for the prior model? $z$ or $e$? If it is $z$, how should I represent it? If it is $e$, how should I implement the network?
  
  **Tags:** machine-learning, generative-model, variational-autoencoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-29. Score: 1
  
  **Note:** This answer just uses images and pixels as an analogy for simplicity. It's not formal.
  
  Some notes about **VQ-VAE**:
  
  1. In the paper, they used **PixelCNN** to learn the prior.
     Remember, **PixelCNN** is a good choice for images of discrete values (e.g. 256 value, 3 channels).
  2. The discrete latent variables are just the indices of the
     embedding vectors. For example, you can put your embedding vectors
     in an array.
  
  For a single input, the number of the output channels of the encoder before quantization equals the dimensionality of the embedding vectors. For an analogy, the output of the encoder is like an image but with a number of channels that equals the dimensions in the embedding space. With that, each pixel is a vector in the space of the embedding vectors. The quantization is done on each pixel by mapping each pixel to the nearest embedding vector $e$.
  
  Now, each pixel can be represented by a single integer number which equals the index of its nearest embedding (what you call the discrete latent variables). Thus, you have two representations for the quantized output, one is a large tensor with many channels (embedding vectors), and another is a simple "discrete image" with one channel (discrete latent variables).
  
  The one-channel "discrete images" are used to train **PixelCNN**. This is great because they are small-size images. while the large tensor with the embedding vectors is used for the decoder as it holds the information needed for reconstruction.
  
  So, the discrete latent variables are used later to learn the prior and the embedding vectors are used as inputs for the decoder.
  
  
  ---
  
  