Body: <p>I don't quite understand why, in <em>Conditional Variational Autoencoder (CVAE)</em>, we concatenate a conditioning vector two times, at encoder and decoder respectively.
  After we concatenate it once at the beginning, isn't the latent distribution going to already incorporate knowledge about the label?</p>
  <p>I know that, from a practical perspective we need to concatenate it at the decoder as well, since we want to be able then to generate new instances associating a specific label, but I'm missing more the mathematical motivations for which we concatenate it two times.</p>
  <p>To be more precise, let's consider how the  objective function of <em>CVAE</em> is defined:</p>
  <p><span class="math-container">$$\mathcal{L}_{CVAE} = \mathbb{E}_\textbf{z}[log \, p_\theta(\textbf{x}|\textbf{z},\textbf{c})] - D_{KL}[q_\phi(\textbf{z}|\textbf{x},\textbf{c})||p(\textbf{z}|\textbf{c})] $$</span></p>
  <p><strong>My question:</strong> <em>Why do we condition on <span class="math-container">$\textbf{c}$</span> in both terms?</em></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T00:04:35.010Z
Comments.Text: To make sure I understand your question, in the objective function, $c$ is what you call _label_? [This paper](https://papers.nips.cc/paper/2015/file/8d55a249e6baa5c06772297520da2051-Paper.pdf) seems to use a different notation for the objective function. It might be a good idea to provide the reference you're using.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T00:16:56.297Z
Comments.Text: Exactly, $\textbf{c}$ is the label in the objective function I have specified
Comments.UserDisplayName: James Arten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T08:59:27.303Z
Comments.Text: So, your question is: "In CVAE, why are both the encoder and decoder defined as a function or probability distribution that is conditioned on the label"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T09:52:18.560Z
Comments.Text: Yes that's pretty much what I wanted to ask
Comments.UserDisplayName: James Arten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-17T10:04:32.373Z
Comments.Text: Ok, can you please provide the source of this equation? Because I'm looking at the original paper, as I already said, and it uses a different notation, and it's harder for me to associate the symbols there with yours. It seems that you're using $c$ to denote what the authors of the paper denote by $x$, and you use $x$ to denote what they denote by $y$, the label or output variable. $z$ is the latent vector both in your case and in the original paper. $c$ seems to be the input to the auto-encoder, not a label.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-17T10:05:16.850Z
Comments.Text: So, they are not conditioning on a "label" (which would be $y$ in the paper and $x$ in your notation), but conditioning on the inputs. So, please, edit your post to reflect this comment and to either make your notation equal to the original paper's one or to provide a reference that uses this notation. Moreover, your question "Why do we condition both on encoder and decoder?" is not specific and clear enough. Because, even in the original ELBO, you condition on something. You want to ask why we're conditioning on $c$ in both terms.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-17T14:14:50.773Z
Comments.Text: That's exactly what I'm trying to better frame. Why do we condition on $\textbf{c}$ in both terms of the loss? I've modified the question clearly expressing this doubt.
Comments.UserDisplayName: James Arten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-17T14:15:59.733Z
Comments.Text: I had understood that this was your question, but I recommend that you follow my other suggestions in the comments above.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-16T14:39:51.070Z
Comments.Text: I suspect that z needs to be conditioned on c to get the KL maffs to work out. Crudely it also makes sense that the encoder can do a better job with more information, in which case the decoder can benefit. But I also expect it sometimes won't matter in practice. Not been able to find work on this specifically. So am going to test.
Comments.UserDisplayName: Fred Shone
Tags: <objective-functions><variational-autoencoder><conditional-vae>
Title: In CVAE's objective function, why do both terms condition on $\textbf{c}$?
UserDisplayName: James Arten
ThreadText: # In CVAE's objective function, why do both terms condition on $\textbf{c}$?
  
  (Asked by: username_0 on 2022-03-05)
  
  I don't quite understand why, in *Conditional Variational Autoencoder (CVAE)*, we concatenate a conditioning vector two times, at encoder and decoder respectively.
  After we concatenate it once at the beginning, isn't the latent distribution going to already incorporate knowledge about the label?
  
  I know that, from a practical perspective we need to concatenate it at the decoder as well, since we want to be able then to generate new instances associating a specific label, but I'm missing more the mathematical motivations for which we concatenate it two times.
  
  To be more precise, let's consider how the objective function of *CVAE* is defined:
  
  $$\mathcal{L}\_{CVAE} = \mathbb{E}\_\textbf{z}[log \, p\_\theta(\textbf{x}|\textbf{z},\textbf{c})] - D\_{KL}[q\_\phi(\textbf{z}|\textbf{x},\textbf{c})||p(\textbf{z}|\textbf{c})] $$
  
  **My question:** *Why do we condition on $\textbf{c}$ in both terms?*
  
  **Tags:** objective-functions, variational-autoencoder, conditional-vae
  > **Comments:**
  > **username_1**: To make sure I understand your question, in the objective function, $c$ is what you call _label_? [This paper](https://papers.nips.cc/paper/2015/file/8d55a249e6baa5c06772297520da2051-Paper.pdf) seems to use a different notation for the objective function. It might be a good idea to provide the reference you're using.  
  > **username_0**: Exactly, $\textbf{c}$ is the label in the objective function I have specified  
  > **username_1**: So, your question is: "In CVAE, why are both the encoder and decoder defined as a function or probability distribution that is conditioned on the label"?  
  > **username_0**: Yes that's pretty much what I wanted to ask  
  > **username_1**: Ok, can you please provide the source of this equation? Because I'm looking at the original paper, as I already said, and it uses a different notation, and it's harder for me to associate the symbols there with yours. It seems that you're using $c$ to denote what the authors of the paper denote by $x$, and you use $x$ to denote what they denote by $y$, the label or output variable. $z$ is the latent vector both in your case and in the original paper. $c$ seems to be the input to the auto-encoder, not a label.  
  > **username_1**: So, they are not conditioning on a "label" (which would be $y$ in the paper and $x$ in your notation), but conditioning on the inputs. So, please, edit your post to reflect this comment and to either make your notation equal to the original paper's one or to provide a reference that uses this notation. Moreover, your question "Why do we condition both on encoder and decoder?" is not specific and clear enough. Because, even in the original ELBO, you condition on something. You want to ask why we're conditioning on $c$ in both terms.  
  > **username_0**: That's exactly what I'm trying to better frame. Why do we condition on $\textbf{c}$ in both terms of the loss? I've modified the question clearly expressing this doubt.  
  > **username_1**: I had understood that this was your question, but I recommend that you follow my other suggestions in the comments above.  
  > **username_2**: I suspect that z needs to be conditioned on c to get the KL maffs to work out. Crudely it also makes sense that the encoder can do a better job with more information, in which case the decoder can benefit. But I also expect it sometimes won't matter in practice. Not been able to find work on this specifically. So am going to test.  
  
  ---
  
  ## 0 answers
  
  