Answers.Body: <p>The reason is indeed the difference between <strong>traditional</strong> variational inference and the <strong>amortized</strong> variational inference.</p>
  <p>In traditional VI, the objective is to solving the following problem for every datapoint <span class="math-container">$x^{(i)}$</span>
  <span class="math-container">$$\min _{q(z)} \mathrm{KL}\left(q(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$</span>
  Pay attention that if we write out explicitly the variational parameter  <span class="math-container">$\phi$</span>, it should depend on i-th datapoint (hence we call them <em>local parameters</em>), i.e. we are in fact solving N problems provided N is the number of datapoints.
  <span class="math-container">$$\min _{\phi_i} \mathrm{KL}\left(q_{\phi_i}(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$</span></p>
  <p>while for amortized VI, we try to find a <em>global variational parameter</em> that serves for the inference of all the datapoints, i.e.
  <span class="math-container">$$\min _{\phi} \sum_{i=1}^N \operatorname{KL}\left(q_{\phi}\left(z \mid x^{(i)}\right) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$</span></p>
  <p><a href="https://arxiv.org/abs/2209.10888" rel="nofollow noreferrer">This paper</a> explains the difference with better notations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-19T14:53:07.333Z
Answers.LastActivityDate: 2023-12-19T14:53:07.333Z
Answers.UserDisplayName: sicheng mao
Body: <p>In page 21 <a href="https://drive.google.com/file/d/1JV-Rsz1MECZWWtvdXjxnt03HOxiGWPYy/view" rel="nofollow noreferrer">here</a>, it states:</p>
  <blockquote>
  <p><strong>General Idea of Amortization</strong>: if same inference problem needs to be solved many times, can we parameterize a neural network to solve it?</p>
  <p><strong>Our case</strong>: for all <span class="math-container">$x^{(i)}$</span> we want to solve:
  <span class="math-container">$$
  \min _{q(z)} \mathrm{KL}\left(q(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right.
  $$</span>
  <strong>Amortized formulation</strong>:
  <span class="math-container">$$
  \min _{\phi} \sum \operatorname{KL}\left(q_{\phi}\left(z \mid x^{(i)}\right) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)
  $$</span></p>
  </blockquote>
  <p>One thing I am trying to wrap my mind around is <span class="math-container">$q(z)$</span> in the 1st formulation vs <span class="math-container">$q_{\phi}(z \mid x^{(i)})$</span> in the second.</p>
  <p>Why is one a conditioned on <span class="math-container">$x^{(i)}$</span> and the other is not? I know in the 1st formulation, we are trying to find a different <span class="math-container">$q(z)$</span> for each datapoint.</p>
  <p>I also recall that in VAEs,which uses amortized inference we consider <span class="math-container">$q(z)$</span> to be aggregated posterior, like</p>
  <p><span class="math-container">$$q_{\phi}(z)=\int q_{\phi}(x, z) d x \quad \text { Marginal of } q_{\phi}(x, z) \text { on } z$$</span></p>
  <p><span class="math-container">$$q_{\phi}(x, z) \equiv p_{\mathcal{D}}(x) q_{\phi}(z \mid x)$$</span></p>
  <p>(formulas taken from <a href="https://arxiv.org/pdf/1706.02262.pdf" rel="nofollow noreferrer">here</a>)</p>
  
Tags: <variational-autoencoder><notation><variational-inference>
Title: Why do we use $q_{\phi}(z \mid x^{(i)})$ in the objective function of amortized variational inference, while sometimes we use $q(z)$?
UserDisplayName: a12345
ThreadText: # Why do we use $q_{\phi}(z \mid x^{(i)})$ in the objective function of amortized variational inference, while sometimes we use $q(z)$?
  
  (Asked by: username_0 on 2021-12-17)
  
  In page 21 [here](https://drive.google.com/file/d/1JV-Rsz1MECZWWtvdXjxnt03HOxiGWPYy/view), it states:
  
  > **General Idea of Amortization**: if same inference problem needs to be solved many times, can we parameterize a neural network to solve it?
  >
  > **Our case**: for all $x^{(i)}$ we want to solve:
  > $$
  > \min \_{q(z)} \mathrm{KL}\left(q(z) \| p\_{\theta}\left(z \mid x^{(i)}\right)\right.
  > $$
  > **Amortized formulation**:
  > $$
  > \min \_{\phi} \sum \operatorname{KL}\left(q\_{\phi}\left(z \mid x^{(i)}\right) \| p\_{\theta}\left(z \mid x^{(i)}\right)\right)
  > $$
  
  One thing I am trying to wrap my mind around is $q(z)$ in the 1st formulation vs $q\_{\phi}(z \mid x^{(i)})$ in the second.
  
  Why is one a conditioned on $x^{(i)}$ and the other is not? I know in the 1st formulation, we are trying to find a different $q(z)$ for each datapoint.
  
  I also recall that in VAEs,which uses amortized inference we consider $q(z)$ to be aggregated posterior, like
  
  $$q\_{\phi}(z)=\int q\_{\phi}(x, z) d x \quad \text { Marginal of } q\_{\phi}(x, z) \text { on } z$$
  
  $$q\_{\phi}(x, z) \equiv p\_{\mathcal{D}}(x) q\_{\phi}(z \mid x)$$
  
  (formulas taken from [here](https://arxiv.org/pdf/1706.02262.pdf))
  
  **Tags:** variational-autoencoder, notation, variational-inference
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-19. Score: 1
  
  The reason is indeed the difference between **traditional** variational inference and the **amortized** variational inference.
  
  In traditional VI, the objective is to solving the following problem for every datapoint $x^{(i)}$
  $$\min \_{q(z)} \mathrm{KL}\left(q(z) \| p\_{\theta}\left(z \mid x^{(i)}\right)\right)$$
  Pay attention that if we write out explicitly the variational parameter $\phi$, it should depend on i-th datapoint (hence we call them *local parameters*), i.e. we are in fact solving N problems provided N is the number of datapoints.
  $$\min \_{\phi\_i} \mathrm{KL}\left(q\_{\phi\_i}(z) \| p\_{\theta}\left(z \mid x^{(i)}\right)\right)$$
  
  while for amortized VI, we try to find a *global variational parameter* that serves for the inference of all the datapoints, i.e.
  $$\min \_{\phi} \sum\_{i=1}^N \operatorname{KL}\left(q\_{\phi}\left(z \mid x^{(i)}\right) \| p\_{\theta}\left(z \mid x^{(i)}\right)\right)$$
  
  [This paper](https://arxiv.org/abs/2209.10888) explains the difference with better notations.
  
  
  ---
  
  