Answers.Body: <p>The tendency in literature in the last years (at least for computer vision problems) seems to point towards the single model option (I'll try to remember to come back and add some links to papers mentioning this when I find them), although this IMO is really data- and problem-dependent. </p>
  
  <p>In your case, I would set up a network for the mapping <span class="math-container">$x$</span> to <span class="math-container">$g(x)$</span>, with a training-only auxiliary loss calculated on the mapping <span class="math-container">$x$</span> to <span class="math-container">$f1(x)$</span> and compare this with a model trained only on "<span class="math-container">$x$</span> to <span class="math-container">$g(x)$</span>".</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-29T11:24:49.843Z
Answers.Text: I didn't understand your second paragraph. Did you mean to say that you'd try both options and see which one is better?
Answers.UserDisplayName: Gilad Deutsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-29T11:33:00.313Z
Answers.Text: No. I'd try the second option and compare it with a model that predicts **both** F1(X) and g(X). The model has two regression heads, one "earlier" in the network for F1(X) and one deeper that regresses g(X). You train as a multi-task problem using a combined weighed loss on both regression targets, but then for inference you only use the output from the head that predicts g(X). The idea is to use the first regression head to steer the training in the "easier" direction by incorporating the knowledge that predicting F1(X) helps predicting G(X).
Answers.UserDisplayName: GPhilo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T09:16:08.597Z
Answers.LastActivityDate: 2020-04-28T09:16:08.597Z
Answers.UserDisplayName: GPhilo
Body: <p>Let's say one wants to use a neural net to learn some function <span class="math-container">$g(x)$</span>. Let's say that we know that <span class="math-container">$g$</span> is a combination of two functions (or two sub-problems), <span class="math-container">$g(x)=f_2(f_1(x))$</span>, and that we have two datasets </p>
  
  <ol>
  <li>composed of <span class="math-container">$x$</span> samples and their corresponding <span class="math-container">$g(x)$</span> labels, and </li>
  <li>composed of <span class="math-container">$x$</span> samples and their corresponding <span class="math-container">$f_1(x)$</span> labels. </li>
  </ol>
  
  <p>Should we use two nets, one to learn the mapping from <span class="math-container">$x$</span> samples to <span class="math-container">$f_1(x)$</span> using dataset 1 and another net to learn the mapping from <span class="math-container">$f_1(x)$</span> to <span class="math-container">$g(x)$</span> (note that we can build a dataset composed of <span class="math-container">$f_1(x)$</span> samples and <span class="math-container">$g(x)$</span> labels with the trained net), or just one net to learn mappings from <span class="math-container">$x$</span> to <span class="math-container">$g(x)$</span> using dataset 1? </p>
  
  <p>Intuitively, the first option seems to be better since we take advantage of our knowledge that <span class="math-container">$f_1$</span> is a "sub-problem" of <span class="math-container">$g$</span>. </p>
  
Tags: <machine-learning><deep-learning><ai-design><computational-learning-theory>
Title: A model for each sub-problem vs one model for the whole problem
UserDisplayName: Gilad Deutsch
ThreadText: # A model for each sub-problem vs one model for the whole problem
  
  (Asked by: username_0 on 2020-04-27)
  
  Let's say one wants to use a neural net to learn some function $g(x)$. Let's say that we know that $g$ is a combination of two functions (or two sub-problems), $g(x)=f\_2(f\_1(x))$, and that we have two datasets
  
  1. composed of $x$ samples and their corresponding $g(x)$ labels, and
  2. composed of $x$ samples and their corresponding $f\_1(x)$ labels.
  
  Should we use two nets, one to learn the mapping from $x$ samples to $f\_1(x)$ using dataset 1 and another net to learn the mapping from $f\_1(x)$ to $g(x)$ (note that we can build a dataset composed of $f\_1(x)$ samples and $g(x)$ labels with the trained net), or just one net to learn mappings from $x$ to $g(x)$ using dataset 1?
  
  Intuitively, the first option seems to be better since we take advantage of our knowledge that $f\_1$ is a "sub-problem" of $g$.
  
  **Tags:** machine-learning, deep-learning, ai-design, computational-learning-theory
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-28. Score: 1
  
  The tendency in literature in the last years (at least for computer vision problems) seems to point towards the single model option (I'll try to remember to come back and add some links to papers mentioning this when I find them), although this IMO is really data- and problem-dependent.
  
  In your case, I would set up a network for the mapping $x$ to $g(x)$, with a training-only auxiliary loss calculated on the mapping $x$ to $f1(x)$ and compare this with a model trained only on "$x$ to $g(x)$".
  
  > **Comments:**
  > **username_0**: I didn't understand your second paragraph. Did you mean to say that you'd try both options and see which one is better?  
  > **username_1**: No. I'd try the second option and compare it with a model that predicts **both** F1(X) and g(X). The model has two regression heads, one "earlier" in the network for F1(X) and one deeper that regresses g(X). You train as a multi-task problem using a combined weighed loss on both regression targets, but then for inference you only use the output from the head that predicts g(X). The idea is to use the first regression head to steer the training in the "easier" direction by incorporating the knowledge that predicting F1(X) helps predicting G(X).  
  
  ---
  
  