Body: <p>Suppose <span class="math-container">$G_{\phi}:\mathcal{Z}\rightarrow \mathcal{X}$</span> is a generator (neural network, non-invertible) that can sample from some distribution <span class="math-container">$\pi$</span> on <span class="math-container">$\mathcal{X}$</span>. That is, <span class="math-container">$G_{\phi}(z)\sim \pi$</span> when <span class="math-container">$z\sim \mathcal{N}(0,I)$</span>. Let <span class="math-container">$\phi+\delta_{\phi}$</span> represent a (small) perturbation of the parameters of <span class="math-container">$G_{\phi}$</span> and let <span class="math-container">$G_{\phi+\delta_{\phi}}(z)\sim \pi'$</span> when <span class="math-container">$z\sim \mathcal{N}(0,I)$</span>.</p>
  <p>Are there any results that quantify or bound <span class="math-container">$\mathcal{D}(\pi,\pi')$</span> in terms of <span class="math-container">$\delta_{\phi}$</span>, where <span class="math-container">$\mathcal{D}$</span> is a distance measure for distributions (let's say KL-divergence, or the Wasserstein-1 distance)?</p>
  <p>Basically, I want to know what kind of geometry is induced on the space of distributions by the Euclidian geometry on the parameter space of a generative adversarial network.</p>
  <p>To explain further, let's consider a parametric family of distributions <span class="math-container">$p_{\phi}$</span>, where <span class="math-container">$\phi\in\Phi$</span> (some parameter space). It is a fairly well-known result in statistics that <span class="math-container">$\text{KL}(p_{\phi}||p_{\phi+\delta_{\phi}})\approx \frac{1}{2}\delta_{\phi}^\top F_{\phi} \delta_{\phi}$</span>, where <span class="math-container">$F_{\phi}$</span> is the Fisher information matrix. When the family <span class="math-container">$p_{\phi}$</span> is generated by a GAN with parameter <span class="math-container">$\phi$</span> (in which case we don't know <span class="math-container">$p_{\phi}$</span> in closed-form), can we have an analogous result?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-02T18:11:50.557Z
Comments.Text: I'm not really sure if I understood your question correctly, but here is some ideas. Let's say we have a 32x32 RGB image. It gives us $32^2\cdot 256^3$ dimensional space. But it turns out, that we can encode an image using latent features. When we talk about a distribution space in terms of GAN, we mean a lower-dimensional latent features manifold embedded in that higher-dimensional space. StyleGAN2 has done excellent research on this topic.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-02T18:18:02.797Z
Comments.Text: You can also take a look at this [paper](https://arxiv.org/abs/1803.01541). The authors propose 50% dropdown layers that are used to generate images with some noise to make the model explore the underlaying manifold.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-02T19:05:27.630Z
Comments.Text: Please see the clarification I addded. Hope that clarifies my question.
Comments.UserDisplayName: Subho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-02T22:04:33.160Z
Comments.Text: What do you mean by generating a probability distribution $p_{\phi}$ by a GAN? The GAN itself (namely the generator) is a probability distribution from which we can sample data (images).
Comments.UserDisplayName: Aray Karjauv
Tags: <generative-adversarial-networks>
Title: How does the output distribution of a GAN change if the parameters are slightly purturbed?
UserDisplayName: Subho
ThreadText: # How does the output distribution of a GAN change if the parameters are slightly purturbed?
  
  (Asked by: username_0 on 2021-04-01)
  
  Suppose $G\_{\phi}:\mathcal{Z}\rightarrow \mathcal{X}$ is a generator (neural network, non-invertible) that can sample from some distribution $\pi$ on $\mathcal{X}$. That is, $G\_{\phi}(z)\sim \pi$ when $z\sim \mathcal{N}(0,I)$. Let $\phi+\delta\_{\phi}$ represent a (small) perturbation of the parameters of $G\_{\phi}$ and let $G\_{\phi+\delta\_{\phi}}(z)\sim \pi'$ when $z\sim \mathcal{N}(0,I)$.
  
  Are there any results that quantify or bound $\mathcal{D}(\pi,\pi')$ in terms of $\delta\_{\phi}$, where $\mathcal{D}$ is a distance measure for distributions (let's say KL-divergence, or the Wasserstein-1 distance)?
  
  Basically, I want to know what kind of geometry is induced on the space of distributions by the Euclidian geometry on the parameter space of a generative adversarial network.
  
  To explain further, let's consider a parametric family of distributions $p\_{\phi}$, where $\phi\in\Phi$ (some parameter space). It is a fairly well-known result in statistics that $\text{KL}(p\_{\phi}||p\_{\phi+\delta\_{\phi}})\approx \frac{1}{2}\delta\_{\phi}^\top F\_{\phi} \delta\_{\phi}$, where $F\_{\phi}$ is the Fisher information matrix. When the family $p\_{\phi}$ is generated by a GAN with parameter $\phi$ (in which case we don't know $p\_{\phi}$ in closed-form), can we have an analogous result?
  
  **Tags:** generative-adversarial-networks
  > **Comments:**
  > **username_1**: I'm not really sure if I understood your question correctly, but here is some ideas. Let's say we have a 32x32 RGB image. It gives us $32^2\cdot 256^3$ dimensional space. But it turns out, that we can encode an image using latent features. When we talk about a distribution space in terms of GAN, we mean a lower-dimensional latent features manifold embedded in that higher-dimensional space. StyleGAN2 has done excellent research on this topic.  
  > **username_1**: You can also take a look at this [paper](https://arxiv.org/abs/1803.01541). The authors propose 50% dropdown layers that are used to generate images with some noise to make the model explore the underlaying manifold.  
  > **username_0**: Please see the clarification I addded. Hope that clarifies my question.  
  > **username_1**: What do you mean by generating a probability distribution $p_{\phi}$ by a GAN? The GAN itself (namely the generator) is a probability distribution from which we can sample data (images).  
  
  ---
  
  ## 0 answers
  
  