Answers.Body: <p>It is possible to have an output for every combination of independent classes.  As another poster has noted this scales poorly.  It is also possible to have a network which diverges into two sets of outputs, each softmax-ed among themselves.  This reduces some overhead compared to two separate networks.</p>
  <p>The question about whether either will generalize well is born out by the data.  The first few hidden layers may perform some transformation which makes  indoor/outdoor and daytime/nighttime separable and only decision boundaries need to be drawn by the final weights/activations.  This is the best case scenario.  The worst case scenario is that there are no common transformations which separate all of these classes and the divergent network begins at the input layer at which case one may as well have two independent networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-24T21:32:24.280Z
Answers.LastActivityDate: 2024-04-24T21:32:24.280Z
Answers.UserDisplayName: foreverska
Answers.Body: <p>Softmax gives you a probability distribution, thus its output are positive and sum to 1... if you want &quot;outdoor&quot; AND &quot;daytime&quot;, you want a <span class="math-container">$1$</span> on the &quot;outdoor&quot; neuron and a <span class="math-container">$1$</span> on the &quot;daytime&quot; neuron... clearly, that's not possible if you use a single softmax, as the sum should add up to 2</p>
  <p>What you can do, is to create a cartesian product of your output, and thus having an output like:</p>
  <ul>
  <li>neuron 1: outdoor and daytime</li>
  <li>neuron 2: outdoor and night time</li>
  <li>neuron 3: indoor and daytime</li>
  <li>neuron 4: indoor and night time</li>
  </ul>
  <p>However, this solution works only because you are in a very small example, if instead you have 10 options for the first classification, and 10 for the second one, in the case of 2 softmax, you would have 20 neurons, instead with this &quot;joint&quot; representation, you would need 100...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-26T14:31:45.493Z
Answers.Text: Ok, I got it. So, one logit to classify as indoor OR outdoor and another one to classify as daytime OR nightime. At first, I thought that would be more effective to look at the whole picture and classify with the x AND y, but I got it now. Thanks.
Answers.UserDisplayName: Dimitri
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-24T19:56:41.367Z
Answers.LastActivityDate: 2024-04-24T19:56:41.367Z
Answers.UserDisplayName: Alberto
Body: <p>Simply put, the question 11 in chapter 4 of Aurélien Géron's book <a href="https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/" rel="nofollow noreferrer">&quot;Hands-on Machine Learning&quot;</a> asks:</p>
  <blockquote>
  <p>Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime. Should you implement two logistic regression classifiers or one softmax regression classifier?</p>
  </blockquote>
  <p>which he gives the answer in the jupyter notebook that accompanies the book (the given answer quoted below is exactly how he gave in the book and is complete):</p>
  <blockquote>
  <p>If you want to classify pictures as outdoor/indoor and daytime/nighttime, since these are not exclusive classes (i.e., all four combinations are possible) you should train two Logistic Regression classifiers.</p>
  </blockquote>
  <p>And I am not sure if I got it. My answer was direct: one softmax!
  I was surprised the the answer was different from that and I don't know if I understand it.</p>
  <p>What I thought:</p>
  <p>Isn't the idea of softmax regression: multiclass classification? When he says &quot;all four combinations are possible&quot; doesn't he means <code>outdoor daytime</code>, <code>outdoor nighttime</code>, <code>indoor daytime</code>, and <code>indoor nighttime</code>? If those four classes are possible and I want to classify pictures as one between those four, this is the task I expect to be more appropriate for a Softmax regression than two separate logistic regressions.</p>
  <p>More than that, I think that Softmax regression allows for joint modeling instead of treating each classification task independently, which would be the case of using two logistic regressions. Considering all classes together and learning the relationships between the features can capture dependencies between then, right?</p>
  <p>About the parameter estimation: isn't estimating a single set of parameters that map inputs to probabilities of each class more efficient than training two separate models?</p>
  <p>Generalization: isn't sofmatx possibly better to generalize to unseen combinations of <code>outdoor/indoor</code> and <code>daytime/nighttime</code> since it learns a unified decision boundary?</p>
  <p>Am I wrong? There are more things to consider or two logistic regressions separate will be always better than one softmax model?</p>
  
Tags: <machine-learning><softmax><logistic-regression>
Title: One Softmax or two separate logistic regressions for the task of classifying pictures as a/b and c/d
UserDisplayName: Dimitri
ThreadText: # One Softmax or two separate logistic regressions for the task of classifying pictures as a/b and c/d
  
  (Asked by: username_0 on 2024-04-24)
  
  Simply put, the question 11 in chapter 4 of Aurélien Géron's book ["Hands-on Machine Learning"](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/) asks:
  
  > Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime. Should you implement two logistic regression classifiers or one softmax regression classifier?
  
  which he gives the answer in the jupyter notebook that accompanies the book (the given answer quoted below is exactly how he gave in the book and is complete):
  
  > If you want to classify pictures as outdoor/indoor and daytime/nighttime, since these are not exclusive classes (i.e., all four combinations are possible) you should train two Logistic Regression classifiers.
  
  And I am not sure if I got it. My answer was direct: one softmax!
  I was surprised the the answer was different from that and I don't know if I understand it.
  
  What I thought:
  
  Isn't the idea of softmax regression: multiclass classification? When he says "all four combinations are possible" doesn't he means `outdoor daytime`, `outdoor nighttime`, `indoor daytime`, and `indoor nighttime`? If those four classes are possible and I want to classify pictures as one between those four, this is the task I expect to be more appropriate for a Softmax regression than two separate logistic regressions.
  
  More than that, I think that Softmax regression allows for joint modeling instead of treating each classification task independently, which would be the case of using two logistic regressions. Considering all classes together and learning the relationships between the features can capture dependencies between then, right?
  
  About the parameter estimation: isn't estimating a single set of parameters that map inputs to probabilities of each class more efficient than training two separate models?
  
  Generalization: isn't sofmatx possibly better to generalize to unseen combinations of `outdoor/indoor` and `daytime/nighttime` since it learns a unified decision boundary?
  
  Am I wrong? There are more things to consider or two logistic regressions separate will be always better than one softmax model?
  
  **Tags:** machine-learning, softmax, logistic-regression
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-04-24. Score: 2 | Accepted answer
  
  Softmax gives you a probability distribution, thus its output are positive and sum to 1... if you want "outdoor" AND "daytime", you want a $1$ on the "outdoor" neuron and a $1$ on the "daytime" neuron... clearly, that's not possible if you use a single softmax, as the sum should add up to 2
  
  What you can do, is to create a cartesian product of your output, and thus having an output like:
  
  * neuron 1: outdoor and daytime
  * neuron 2: outdoor and night time
  * neuron 3: indoor and daytime
  * neuron 4: indoor and night time
  
  However, this solution works only because you are in a very small example, if instead you have 10 options for the first classification, and 10 for the second one, in the case of 2 softmax, you would have 20 neurons, instead with this "joint" representation, you would need 100...
  
  > **Comments:**
  > **username_0**: Ok, I got it. So, one logit to classify as indoor OR outdoor and another one to classify as daytime OR nightime. At first, I thought that would be more effective to look at the whole picture and classify with the x AND y, but I got it now. Thanks.  
  
  ---
  
  ### Answer by username_1 on 2024-04-24. Score: 2
  
  It is possible to have an output for every combination of independent classes. As another poster has noted this scales poorly. It is also possible to have a network which diverges into two sets of outputs, each softmax-ed among themselves. This reduces some overhead compared to two separate networks.
  
  The question about whether either will generalize well is born out by the data. The first few hidden layers may perform some transformation which makes indoor/outdoor and daytime/nighttime separable and only decision boundaries need to be drawn by the final weights/activations. This is the best case scenario. The worst case scenario is that there are no common transformations which separate all of these classes and the divergent network begins at the input layer at which case one may as well have two independent networks.
  
  
  ---
  
  