Answers.Body: <p>One of the core contributions presented in the paper consists of understanding at a deeper level the objective function used in Beta VAE and improving it</p>
  
  <p>More specifically, the authors started from Beta VAE OF</p>
  
  <p><span class="math-container">$$\frac{1}{N} \sum_{i=1}^{N}\left[\mathbb{E}_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-\beta K L\left(q\left(z | x^{(i)}\right) \| p(z)\right)\right]$$</span></p>
  
  <p>which consists of the classical reconstruction loss and the regularization term which steers the Latent Code PDF towards a target PDF.</p>
  
  <p>Developing the second term</p>
  
  <p><span class="math-container">$$\mathbb{E}_{p_{\text{data}}(x)}[K L(q(z | x) \| p(z))]=I(x ; z)+K L(q(z) \| p(z))$$</span></p>
  
  <p>they observed there are 2 terms which push in different directions:</p>
  
  <p><span class="math-container">$I(x,z)$</span> is the mutual information between the input data and the code, and we do not want to penalize this as in fact we want this to be as high as possible, so that the decode contains as much information as possible about the input
  <span class="math-container">$KL(q(z) || p(z))$</span> is the KL divergence term which if penalized pushes the latent code PDF <span class="math-container">$q(z)$</span> towards the prior <span class="math-container">$p(z)$</span> which we choose as factorized
  So penalizing the second term in Beta VAE OF means also penalizing the mutual information hence ultimately loosing reconstruction capability</p>
  
  <p>The authors then propose a new OF which fixes this</p>
  
  <p><span class="math-container">$$\frac{1}{N} \sum_{i=1}^{N}\left[\mathbb{E}_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-K L\left(q\left(z | x^{(i)}\right)|| p(z)\right)\right] -\gamma K L(q(z) \| \bar{q}(z))$$</span></p>
  
  <p>In the rest of the paper they run experiments and show they have improved the disentanglement of the representation (according to the new metric they propose in the paper) while preserving reconstruction hence improving the reconstruction vs disentanglement tradeoff with respect to the Beta VAE paper</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T23:01:53.473Z
Answers.LastActivityDate: 2020-06-01T14:36:25.597Z
Answers.LastEditDate: 2020-06-01T14:36:25.597Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>Considering the paper <a href="https://arxiv.org/abs/1802.05983" rel="nofollow noreferrer">Disentangling by Factorising</a>, in addition to introducing a new model for Disentangled Representation Learning, FactorVAE (see figure), what is the main theoretical contribution provided by the paper? </p>
  
  <p><a href="https://i.sstatic.net/qvcTU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qvcTU.png" alt="FactorVAE Architecture"></a></p>
  
Tags: <neural-networks><deep-learning><papers><variational-autoencoder><disentangled-representation>
Title: What is the main contribution of the paper Disentangling by Factorising?
UserDisplayName: Nicola Bernini
ThreadText: # What is the main contribution of the paper Disentangling by Factorising?
  
  (Asked by: username_0 on 2020-05-31)
  
  Considering the paper [Disentangling by Factorising](https://arxiv.org/abs/1802.05983), in addition to introducing a new model for Disentangled Representation Learning, FactorVAE (see figure), what is the main theoretical contribution provided by the paper?
  
  [![FactorVAE Architecture](https://i.sstatic.net/qvcTU.png)](https://i.sstatic.net/qvcTU.png)
  
  **Tags:** neural-networks, deep-learning, papers, variational-autoencoder, disentangled-representation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-31. Score: 1
  
  One of the core contributions presented in the paper consists of understanding at a deeper level the objective function used in Beta VAE and improving it
  
  More specifically, the authors started from Beta VAE OF
  
  $$\frac{1}{N} \sum\_{i=1}^{N}\left[\mathbb{E}\_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-\beta K L\left(q\left(z | x^{(i)}\right) \| p(z)\right)\right]$$
  
  which consists of the classical reconstruction loss and the regularization term which steers the Latent Code PDF towards a target PDF.
  
  Developing the second term
  
  $$\mathbb{E}\_{p\_{\text{data}}(x)}[K L(q(z | x) \| p(z))]=I(x ; z)+K L(q(z) \| p(z))$$
  
  they observed there are 2 terms which push in different directions:
  
  $I(x,z)$ is the mutual information between the input data and the code, and we do not want to penalize this as in fact we want this to be as high as possible, so that the decode contains as much information as possible about the input
  $KL(q(z) || p(z))$ is the KL divergence term which if penalized pushes the latent code PDF $q(z)$ towards the prior $p(z)$ which we choose as factorized
  So penalizing the second term in Beta VAE OF means also penalizing the mutual information hence ultimately loosing reconstruction capability
  
  The authors then propose a new OF which fixes this
  
  $$\frac{1}{N} \sum\_{i=1}^{N}\left[\mathbb{E}\_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-K L\left(q\left(z | x^{(i)}\right)|| p(z)\right)\right] -\gamma K L(q(z) \| \bar{q}(z))$$
  
  In the rest of the paper they run experiments and show they have improved the disentanglement of the representation (according to the new metric they propose in the paper) while preserving reconstruction hence improving the reconstruction vs disentanglement tradeoff with respect to the Beta VAE paper
  
  
  ---
  
  