Answers.Body: <p>It's probably important that both steps be able to tune the learned representation Ψ. Also, that you are always minimizing the combined loss.</p>
  <p>You should do:</p>
  <ol>
  <li>Freeze Γ and tune Ψ,Ω to minimize ℓ(Ψ,Γ)−ℓ(Ψ,Ω),</li>
  <li>Freeze Ω and tune Ψ,Γ to minimize ℓ(Ψ,Γ)−ℓ(Ψ,Ω).</li>
  </ol>
  <p>Or, better yet, if Γ and Ω accept the same examples, to just train both in each training step.</p>
  <p>Your setup is fine, and sensible. For example, in learning voice conversion, some architectures will have a module adds the following to the objective: This representation of the speech should make it difficult to predict who the speaker is (and thus only model the semantic content of the speech, not speaker style).</p>
  <p>These Greek caps seem like a strange notation to me, btw.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-29T04:04:52.913Z
Answers.Text: $\Gamma$ and $\Omega$ does accept the same examples. However, if I train all the layers with $\ell_s - \alpha \ell_a$ as the objective to be minimized, I'm not sure it'll achieve \eqref{*}. I need the $\Psi$ layer to be such that $\Omega$ cannot be tuned further to lower $\ell_a$. This is what ensures that $\Psi$ has been optimized to prevent prediction of $a$.
Answers.UserDisplayName: ChargeShivers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-30T01:46:58.423Z
Answers.Text: Again, your goal is to induce a representation Ψ that is maximally effective at predicting s using Γ and maximally confusing at predicting  using Ω, weighted by . It doesn't really matter if you train back and forth using freezing, or jointly without freezing. Training jointly without freezing is more elegant. I can try to find examples from the VC literature showing this is fine, but you haven't presented an argument why it shouldn't work. Rather than hypothesizing endlessly, why not just try both approaches and see how they work? If you can't evaluate these approaches, let's discuss that
Answers.UserDisplayName: Joseph Turian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-30T04:46:58.140Z
Answers.Text: Here is the simple reason it won't work:
  $$\min_{\Psi,\Gamma, \Omega} \left[ \ell_s(\Psi,\Gamma) - \alpha  \ell_a(\Psi,\Omega) \right] \neq \min_{\Psi, \Gamma} \left[ \ell_s(\Psi,\Gamma) - \alpha \min_\Omega \ell_a(\Psi,\Omega) \right]$$
  The LHS is a minima while the RHS \eqref{*} is a saddle point. If you try gradient descent with this objective, the $\Omega$ layer gets updated as
  $ \Omega \leftarrow \Omega \color{red}{+} \eta \; \alpha \nabla_\Omega\ell_a$. Intuitively, you want $\Omega$ to be the best it can possibly be (for the chosen $\Psi$) at predicting $s$ and still fail miserably.
Answers.UserDisplayName: ChargeShivers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-30T04:48:35.043Z
Answers.Text: Training all layers jointly is my preference too. In fact, my first attempt  was just that with $\ell_s -\alpha \ell_a$ as the objective. It's when I couldn't get it to converge that I realized it's not even what I need.
  Now, there is this [DANN paper][1] someone just directed me to with a neat gradient reversal layer trick that I think should work for my case too. I haven't quite got it working yet though.
  
  
    [1]: https://arxiv.org/abs/1505.07818
Answers.UserDisplayName: ChargeShivers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-29T00:57:11.683Z
Answers.LastActivityDate: 2022-08-29T00:57:11.683Z
Answers.UserDisplayName: Joseph Turian
Body: <p>I need my model to predict <span class="math-container">$s$</span> from my data <span class="math-container">$x$</span>. Additionally, I need the model to <strong>not</strong> use signals in <span class="math-container">$x$</span> that are predictive of a separate target <span class="math-container">$a$</span>. My approach is to transform <span class="math-container">$x$</span> into a representation <span class="math-container">$\Psi(x)$</span> such that it's good at predicting <span class="math-container">$s$</span> but bad at predicting <span class="math-container">$a$</span>.</p>
  <p>Concretely, let the prediction for <span class="math-container">$s$</span> be
  <span class="math-container">\begin{equation} \hat{s} = (\Gamma \circ \Psi)(x), \end{equation}</span>
  and that for <span class="math-container">$a$</span> be
  <span class="math-container">$$\hat{a} = (\Omega \circ \Psi)(x),$$</span>
  where <span class="math-container">$\Psi, \Gamma, \Omega$</span> are implemented as MLP layers. Further, let <span class="math-container">$\ell_s(\Psi,\Gamma)$</span> and <span class="math-container">$\ell_a(\Psi,\Omega)$</span> be the loss functions corresponding to <span class="math-container">$s$</span> and <span class="math-container">$a$</span>, respectively.</p>
  <p>The objective thus is to simultaneously</p>
  <ul>
  <li>minimize <span class="math-container">$$\ell_s(\Psi,\Gamma)$$</span> (good at predicting <span class="math-container">$s$</span>), and</li>
  <li>maximize <span class="math-container">$$\min_\Omega \ell_a(\Psi,\Omega) := -f(\Psi)$$</span> (bad at predicting <span class="math-container">$a$</span>)</li>
  </ul>
  <p>There may not exist a <span class="math-container">$(\Psi, \Gamma)$</span> combination that simultaneously achieves both. So the hope is to find something on the Pareto frontier by linearly combining the objectives:
  <span class="math-container">\begin{equation}
  \tag{*}\label{*}
  \min_{\Psi, \Gamma} \left[ \ell_s(\Psi,\Gamma) + \alpha f(\Psi) \right] 
  = \min_{\Psi, \Gamma} \left[ \ell_s(\Psi,\Gamma) - \alpha \min_\Omega \ell_a(\Psi,\Omega) \right] 
  \end{equation}</span>
  where <span class="math-container">$\alpha &gt; 0 $</span> is a hyperparameter that trades off the two original objectives.
  <a href="https://i.sstatic.net/D5XXP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/D5XXP.png" alt="enter image description here" /></a>
  However, I'm not sure how to train the layers <span class="math-container">$\Psi, \Gamma, \Omega$</span> to achieve \eqref{*}.
  One procedure I could think of is to alternate between</p>
  <ol>
  <li>Freeze <span class="math-container">$\Psi, \Gamma$</span> and tune <span class="math-container">$\Omega$</span> to minimize <span class="math-container">$\ell_a(\Psi,\Omega)$</span>,</li>
  <li>Freeze <span class="math-container">$\Omega$</span> and tune <span class="math-container">$\Psi, \Gamma$</span> to minimize <span class="math-container">$\ell_s(\Psi,\Gamma) - \alpha \ell_a(\Psi,\Omega)$</span></li>
  </ol>
  <p>until convergence. Would that work?</p>
  
Tags: <optimization><multilayer-perceptrons><representation-learning><multi-task-learning>
Title: Multi-objective training involving maximization of one loss function and minimization of another
UserDisplayName: ChargeShivers
ThreadText: # Multi-objective training involving maximization of one loss function and minimization of another
  
  (Asked by: username_0 on 2022-08-25)
  
  I need my model to predict $s$ from my data $x$. Additionally, I need the model to **not** use signals in $x$ that are predictive of a separate target $a$. My approach is to transform $x$ into a representation $\Psi(x)$ such that it's good at predicting $s$ but bad at predicting $a$.
  
  Concretely, let the prediction for $s$ be
  \begin{equation} \hat{s} = (\Gamma \circ \Psi)(x), \end{equation}
  and that for $a$ be
  $$\hat{a} = (\Omega \circ \Psi)(x),$$
  where $\Psi, \Gamma, \Omega$ are implemented as MLP layers. Further, let $\ell\_s(\Psi,\Gamma)$ and $\ell\_a(\Psi,\Omega)$ be the loss functions corresponding to $s$ and $a$, respectively.
  
  The objective thus is to simultaneously
  
  * minimize $$\ell\_s(\Psi,\Gamma)$$ (good at predicting $s$), and
  * maximize $$\min\_\Omega \ell\_a(\Psi,\Omega) := -f(\Psi)$$ (bad at predicting $a$)
  
  There may not exist a $(\Psi, \Gamma)$ combination that simultaneously achieves both. So the hope is to find something on the Pareto frontier by linearly combining the objectives:
  \begin{equation}
  \tag{\*}\label{\*}
  \min\_{\Psi, \Gamma} \left[ \ell\_s(\Psi,\Gamma) + \alpha f(\Psi) \right]
  = \min\_{\Psi, \Gamma} \left[ \ell\_s(\Psi,\Gamma) - \alpha \min\_\Omega \ell\_a(\Psi,\Omega) \right]
  \end{equation}
  where $\alpha > 0 $ is a hyperparameter that trades off the two original objectives.
  [![enter image description here](https://i.sstatic.net/D5XXP.png)](https://i.sstatic.net/D5XXP.png)
  However, I'm not sure how to train the layers $\Psi, \Gamma, \Omega$ to achieve \eqref{\*}.
  One procedure I could think of is to alternate between
  
  1. Freeze $\Psi, \Gamma$ and tune $\Omega$ to minimize $\ell\_a(\Psi,\Omega)$,
  2. Freeze $\Omega$ and tune $\Psi, \Gamma$ to minimize $\ell\_s(\Psi,\Gamma) - \alpha \ell\_a(\Psi,\Omega)$
  
  until convergence. Would that work?
  
  **Tags:** optimization, multilayer-perceptrons, representation-learning, multi-task-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-29. Score: 0
  
  It's probably important that both steps be able to tune the learned representation Ψ. Also, that you are always minimizing the combined loss.
  
  You should do:
  
  1. Freeze Γ and tune Ψ,Ω to minimize ℓ(Ψ,Γ)−ℓ(Ψ,Ω),
  2. Freeze Ω and tune Ψ,Γ to minimize ℓ(Ψ,Γ)−ℓ(Ψ,Ω).
  
  Or, better yet, if Γ and Ω accept the same examples, to just train both in each training step.
  
  Your setup is fine, and sensible. For example, in learning voice conversion, some architectures will have a module adds the following to the objective: This representation of the speech should make it difficult to predict who the speaker is (and thus only model the semantic content of the speech, not speaker style).
  
  These Greek caps seem like a strange notation to me, btw.
  
  > **Comments:**
  > **username_0**: $\Gamma$ and $\Omega$ does accept the same examples. However, if I train all the layers with $\ell_s - \alpha \ell_a$ as the objective to be minimized, I'm not sure it'll achieve \eqref{*}. I need the $\Psi$ layer to be such that $\Omega$ cannot be tuned further to lower $\ell_a$. This is what ensures that $\Psi$ has been optimized to prevent prediction of $a$.  
  > **username_1**: Again, your goal is to induce a representation Ψ that is maximally effective at predicting s using Γ and maximally confusing at predicting  using Ω, weighted by . It doesn't really matter if you train back and forth using freezing, or jointly without freezing. Training jointly without freezing is more elegant. I can try to find examples from the VC literature showing this is fine, but you haven't presented an argument why it shouldn't work. Rather than hypothesizing endlessly, why not just try both approaches and see how they work? If you can't evaluate these approaches, let's discuss that  
  > **username_0**: Here is the simple reason it won't work: $$\min_{\Psi,\Gamma, \Omega} \left[ \ell_s(\Psi,\Gamma) - \alpha  \ell_a(\Psi,\Omega) \right] \neq \min_{\Psi, \Gamma} \left[ \ell_s(\Psi,\Gamma) - \alpha \min_\Omega \ell_a(\Psi,\Omega) \right]$$ The LHS is a minima while the RHS \eqref{*} is a saddle point. If you try gradient descent with this objective, the $\Omega$ layer gets updated as $ \Omega \leftarrow \Omega \color{red}{+} \eta \; \alpha \nabla_\Omega\ell_a$. Intuitively, you want $\Omega$ to be the best it can possibly be (for the chosen $\Psi$) at predicting $s$ and still fail miserably.  
  > **username_0**: Training all layers jointly is my preference too. In fact, my first attempt  was just that with $\ell_s -\alpha \ell_a$ as the objective. It's when I couldn't get it to converge that I realized it's not even what I need. Now, there is this [DANN paper][1] someone just directed me to with a neat gradient reversal layer trick that I think should work for my case too. I haven't quite got it working yet though.     [1]: https://arxiv.org/abs/1505.07818  
  
  ---
  
  