Answers.Body: <p>Maybe a bit too trivial to work out of the shelf, but I would try to add a component to the adversarial loss based precisely on the given set of coefficients.</p>
  <p>Something like:</p>
  <p><span class="math-container">$L_{linear}= \frac{\gamma}{k} \prod_{k}A_{k}\hat{x}$</span></p>
  <p>which combined with the adversarial loss (assuming minimax, but any other choice is fine as well) would become:</p>
  <p><span class="math-container">$L(G, D)=E_{x}[log(D(x))] + E_{z}[(log(1 - D(G(z)))] + \frac{\gamma}{k} \prod_{k}A_{k}\hat{x} $</span></p>
  <p>where <span class="math-container">$A_{k}$</span> is a set of fixed coefficients of an hyperplane, and <span class="math-container">$\hat{x}$</span> the vector sampled from the generator. I would use a product operator since the loss should drop to zero when <span class="math-container">$\hat{x}$</span> lies in one of the hyperplanes.
  Instead <span class="math-container">$\gamma$</span> can be used to scale the loss to a value in the range of the generator and discriminator losses.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T16:23:05.397Z
Answers.Text: many thanks, that looks interesting. When you say *adversarial loss* , are you considering the mean loss between generator and discriminator? Are you considering the standard GAN approach where binary classification loss is considered?
Answers.UserDisplayName: James Arten
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T16:30:29.760Z
Answers.Text: precisely, I updated the answer to include the total loss, but I stress out again here that the choice of the adversarial loss doesn't matter, vanilla gan, non saturating gan, wasserstein gan are all good options. And you can add as many extra custom loss to it, as long as you formulate them to be converging to 0. (possibly smoothly, which is what my answer lack, it might be the case that as formulate naively now that extra component will assume large values now and then, fact that doesn't not prevent convergence per se).
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T17:11:24.790Z
Answers.Text: many thanks Edoardo. Just as a quick clarification, is the constraining term in the loss function intended to be $\frac{\gamma}{k} \mathbb{E}_z[\Pi_k A_k G(z)]$ , so we take the mean over the batch?
Answers.UserDisplayName: James Arten
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T18:44:50.817Z
Answers.Text: good point! Yes it's definitely the case that for batches larger than 1 the loss score should be averaged
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T16:19:23.107Z
Answers.LastActivityDate: 2022-03-08T16:27:28.227Z
Answers.LastEditDate: 2022-03-08T16:27:28.227Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Given that I'm training a generative model, (say a <em>generative adversarial network</em>), and I know that my (real) inputs (let's say vectors <span class="math-container">$\textbf{x} \in \mathbb{R}^n$</span>) satisfy linear constraints of the form e.g. <span class="math-container">$a_1\textbf{x}_1 + \dots a_n\textbf{x}_n =0$</span>, where the coefficients are <em>fixed</em>, is there a way to inject this knowledge during training?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-08T15:10:38.953Z
Comments.Text: are the $a$ coefficient specific values, or generic values? Cause if the latter I suspect the model will collapse and generate only zero values, since that's equivalent to ask the model to solve $XX^{T}=0$ which is true only for $X=0$
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-08T15:12:25.487Z
Comments.Text: also, just as a side note, GANs are not models but rather training approaches, the generative model is only the generator architecture
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-08T15:42:14.053Z
Comments.Text: no, actually the coefficients are thought to be specific, we would assume that the original data lie in the union of hyperplanes for some fixed coefficients.
Comments.UserDisplayName: James Arten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-08T16:29:27.610Z
Comments.Text: @JamesArten Can you please include these details in the post itself and rewrite the title in the form of a question, i.e. your specific question? Thanks.
Comments.UserDisplayName: nbro
Tags: <training><generative-adversarial-networks><generative-model>
Title: Is there a way to inject linear constrains during GAN training?
UserDisplayName: James Arten
ThreadText: # Is there a way to inject linear constrains during GAN training?
  
  (Asked by: username_0 on 2022-03-08)
  
  Given that I'm training a generative model, (say a *generative adversarial network*), and I know that my (real) inputs (let's say vectors $\textbf{x} \in \mathbb{R}^n$) satisfy linear constraints of the form e.g. $a\_1\textbf{x}\_1 + \dots a\_n\textbf{x}\_n =0$, where the coefficients are *fixed*, is there a way to inject this knowledge during training?
  
  **Tags:** training, generative-adversarial-networks, generative-model
  
  > **Comments:**
  > **username_1**: are the $a$ coefficient specific values, or generic values? Cause if the latter I suspect the model will collapse and generate only zero values, since that's equivalent to ask the model to solve $XX^{T}=0$ which is true only for $X=0$  
  > **username_1**: also, just as a side note, GANs are not models but rather training approaches, the generative model is only the generator architecture  
  > **username_0**: no, actually the coefficients are thought to be specific, we would assume that the original data lie in the union of hyperplanes for some fixed coefficients.  
  > **username_2**: @username_0 Can you please include these details in the post itself and rewrite the title in the form of a question, i.e. your specific question? Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-08. Score: 1 | Accepted answer
  
  Maybe a bit too trivial to work out of the shelf, but I would try to add a component to the adversarial loss based precisely on the given set of coefficients.
  
  Something like:
  
  $L\_{linear}= \frac{\gamma}{k} \prod\_{k}A\_{k}\hat{x}$
  
  which combined with the adversarial loss (assuming minimax, but any other choice is fine as well) would become:
  
  $L(G, D)=E\_{x}[log(D(x))] + E\_{z}[(log(1 - D(G(z)))] + \frac{\gamma}{k} \prod\_{k}A\_{k}\hat{x} $
  
  where $A\_{k}$ is a set of fixed coefficients of an hyperplane, and $\hat{x}$ the vector sampled from the generator. I would use a product operator since the loss should drop to zero when $\hat{x}$ lies in one of the hyperplanes.
  Instead $\gamma$ can be used to scale the loss to a value in the range of the generator and discriminator losses.
  
  > **Comments:**
  > **username_0**: many thanks, that looks interesting. When you say *adversarial loss* , are you considering the mean loss between generator and discriminator? Are you considering the standard GAN approach where binary classification loss is considered?  
  > **username_1**: precisely, I updated the answer to include the total loss, but I stress out again here that the choice of the adversarial loss doesn't matter, vanilla gan, non saturating gan, wasserstein gan are all good options. And you can add as many extra custom loss to it, as long as you formulate them to be converging to 0. (possibly smoothly, which is what my answer lack, it might be the case that as formulate naively now that extra component will assume large values now and then, fact that doesn't not prevent convergence per se).  
  > **username_0**: many thanks username_1. Just as a quick clarification, is the constraining term in the loss function intended to be $\frac{\gamma}{k} \mathbb{E}_z[\Pi_k A_k G(z)]$ , so we take the mean over the batch?  
  > **username_1**: good point! Yes it's definitely the case that for batches larger than 1 the loss score should be averaged  
  
  ---
  
  