Answers.Body: <p>I think there's a crucial point missed in the question, touched by jros answer but without further elaboration.</p>
  <p>If you <strong>train a model on domain A</strong>: single lightning condition and <strong>test it on domain B</strong>: two lightning condition then <strong>you're not evaluating generalization but transfer learning capabilities</strong>. Or to phrase it differently you're evaluating how close domain A and B are for the model you trained.</p>
  <p>The test set as you said is truly made of instances never seen by the model during training, but it should nevertheless be representative, i.e. correctly sampled, from the training domain, or <em>from the same distribution</em> as jros wrote. So the generalization of your model, trained on single lightning condition, should be evaluated on single lighting condition as well.</p>
  <p>A final remark about the rest that have been said:</p>
  <ul>
  <li>everything holds only under the assumptions that the initial training dataset is not only unbiased but also balanced. In a real case scenario changing the training distribution from something specific (single light condition) to another distribution (multiple light conditions) might well be lead to a worse model, simply cause the problem is now inherently harder to solve.</li>
  </ul>
  <p>So the answer to your question (regarding both, true generalization on same distribution and what you describe, transfer learning) is actually just empirical.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-29T01:10:45.140Z
Answers.Text: Although you say this in other parts of your answer, when you write "So the generalization of your model, trained on single lightning condition, should be evaluated on single lighting condition as well.", I think you should also emphasize that this is true only if you're trying to learn a target function where the inputs are _assumed_ to have the same lighting condition and you _assume_ that your training data is representative of your target function or probability distribution (you say that in other parts of the answer, but that sentence read alone may be misleading).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-02T04:32:23.130Z
Answers.Text: I dont exactly need my model trained on domain A to test on domain B. My situation is that my model performs not as expected on domain A only images, I am waiting to get new data from other domains, and i wonder whether training model again on set with samples from additional domains would also help the performance on set A, or do I have to fix something within the model first.
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-02T04:36:45.547Z
Answers.Text: Amd this has led me to a bit more general theoretical question that I asked, cus I was hoping that since most of the features from domain B and C and D would  be same as in A (cus general shapes of gestures are always the same), maybe it will also help domain A.
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-28T08:37:00.000Z
Answers.LastActivityDate: 2021-09-28T08:37:00.000Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <h5>Generalization</h5>
  <p>In machine learning, generalization describes a model's ability to properly correct its algorithms to predict new data from the same distribution as the data used to train the model.</p>
  <p>By providing additional training for your model (on data with varying lighting conditions), you are correct that you would be increasing the capabilities of your model.</p>
  <h5>Does better generalization equal better performance?</h5>
  <p>Consider two models and assume they have the same number of coefficients/ layers &amp; nodes:</p>
  <ul>
  <li><em>Model 1</em>, which was trained solely on a single lighting condition</li>
  <li><em>Model 2</em>, which was trained on multiple lighting conditions</li>
  </ul>
  <p>Let's say we have two test sets as well:</p>
  <ul>
  <li><em>Test Set 1</em> : data with the same lighting condition as the data used to train <em>Model 1</em></li>
  <li><em>Test Set 2</em> : data with the multiple lighting conditions</li>
  </ul>
  <p><em>Model 1</em> would be expected to have an equal or better performance than <em>Model 2</em> on <em>Test Set 1</em>, due to &quot;over&quot; fitting on that lighting condition, but as you noted, <em>Model 1</em> would not perform as well on <em>Test Set 2</em> as on <em>Test Set 1</em>. We also would not expect <em>Model 2</em> to perform better on <em>Test Set 1</em> than <em>Model 1</em> due to the generalizability achieved during training.</p>
  <p>Simply put, you're probably sacrificing some lighting condition-specific accuracy for better accuracy across multiple lighting conditions.</p>
  <h4>However</h4>
  <p>By allowing your <em>Model 2</em> to increase the layers &amp; nodes, or coefficients (including interactions), <em>Model 2</em> may well be capable of performing just as well. All this depends on the size of training sets as well. For instance, if <em>Model 1</em> is trained on 1,000 data points from the single lighting condition, and <em>Model 2</em> is trained on 500 data points, <em>Model 1</em> is generally expected to perform better on <em>Test Set 1</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-27T17:25:59.440Z
Answers.LastActivityDate: 2021-09-27T17:25:59.440Z
Answers.UserDisplayName: jros
Body: <p>I am sorry but I have to explain my question using an example, I do not know how to ask it in proper scientific terms.</p>
  <p>Let's assume, I have trained a deep learning model on classifying hand gestures, but training and testing datasets' images are shot only in one lighting conditions and I achieved certain accuracy, let's assume 85%. As far as I understand, adding more data of the same hand gestures images but shot with different lightning should increase my model's &quot;generalization&quot; capabilities, right?</p>
  <p>So the question is, if I take this model, trained in two lightning conditions, and test it only on the dataset of the first lightning conditions, would that increase it's accuracy (the 85%) or maybe this &quot;generalization&quot; would only mean that it can now also classify correctly images with different lightning, but not increase the accuracy on the first set?</p>
  
Tags: <neural-networks><deep-learning><convolutional-neural-networks><computer-vision><generalization>
Title: How general is generalization?
UserDisplayName: GKozinski
ThreadText: # How general is generalization?
  
  (Asked by: username_0 on 2021-09-25)
  
  I am sorry but I have to explain my question using an example, I do not know how to ask it in proper scientific terms.
  
  Let's assume, I have trained a deep learning model on classifying hand gestures, but training and testing datasets' images are shot only in one lighting conditions and I achieved certain accuracy, let's assume 85%. As far as I understand, adding more data of the same hand gestures images but shot with different lightning should increase my model's "generalization" capabilities, right?
  
  So the question is, if I take this model, trained in two lightning conditions, and test it only on the dataset of the first lightning conditions, would that increase it's accuracy (the 85%) or maybe this "generalization" would only mean that it can now also classify correctly images with different lightning, but not increase the accuracy on the first set?
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, computer-vision, generalization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-09-28. Score: 1
  
  I think there's a crucial point missed in the question, touched by username_3 answer but without further elaboration.
  
  If you **train a model on domain A**: single lightning condition and **test it on domain B**: two lightning condition then **you're not evaluating generalization but transfer learning capabilities**. Or to phrase it differently you're evaluating how close domain A and B are for the model you trained.
  
  The test set as you said is truly made of instances never seen by the model during training, but it should nevertheless be representative, i.e. correctly sampled, from the training domain, or *from the same distribution* as username_3 wrote. So the generalization of your model, trained on single lightning condition, should be evaluated on single lighting condition as well.
  
  A final remark about the rest that have been said:
  
  * everything holds only under the assumptions that the initial training dataset is not only unbiased but also balanced. In a real case scenario changing the training distribution from something specific (single light condition) to another distribution (multiple light conditions) might well be lead to a worse model, simply cause the problem is now inherently harder to solve.
  
  So the answer to your question (regarding both, true generalization on same distribution and what you describe, transfer learning) is actually just empirical.
  
  > **Comments:**
  > **username_2**: Although you say this in other parts of your answer, when you write "So the generalization of your model, trained on single lightning condition, should be evaluated on single lighting condition as well.", I think you should also emphasize that this is true only if you're trying to learn a target function where the inputs are _assumed_ to have the same lighting condition and you _assume_ that your training data is representative of your target function or probability distribution (you say that in other parts of the answer, but that sentence read alone may be misleading).  
  > **username_0**: I dont exactly need my model trained on domain A to test on domain B. My situation is that my model performs not as expected on domain A only images, I am waiting to get new data from other domains, and i wonder whether training model again on set with samples from additional domains would also help the performance on set A, or do I have to fix something within the model first.  
  > **username_0**: Amd this has led me to a bit more general theoretical question that I asked, cus I was hoping that since most of the features from domain B and C and D would  be same as in A (cus general shapes of gestures are always the same), maybe it will also help domain A.  
  
  ---
  
  ### Answer by username_3 on 2021-09-27. Score: 0
  
  ##### Generalization
  
  In machine learning, generalization describes a model's ability to properly correct its algorithms to predict new data from the same distribution as the data used to train the model.
  
  By providing additional training for your model (on data with varying lighting conditions), you are correct that you would be increasing the capabilities of your model.
  
  ##### Does better generalization equal better performance?
  
  Consider two models and assume they have the same number of coefficients/ layers & nodes:
  
  * *Model 1*, which was trained solely on a single lighting condition
  * *Model 2*, which was trained on multiple lighting conditions
  
  Let's say we have two test sets as well:
  
  * *Test Set 1* : data with the same lighting condition as the data used to train *Model 1*
  * *Test Set 2* : data with the multiple lighting conditions
  
  *Model 1* would be expected to have an equal or better performance than *Model 2* on *Test Set 1*, due to "over" fitting on that lighting condition, but as you noted, *Model 1* would not perform as well on *Test Set 2* as on *Test Set 1*. We also would not expect *Model 2* to perform better on *Test Set 1* than *Model 1* due to the generalizability achieved during training.
  
  Simply put, you're probably sacrificing some lighting condition-specific accuracy for better accuracy across multiple lighting conditions.
  
  #### However
  
  By allowing your *Model 2* to increase the layers & nodes, or coefficients (including interactions), *Model 2* may well be capable of performing just as well. All this depends on the size of training sets as well. For instance, if *Model 1* is trained on 1,000 data points from the single lighting condition, and *Model 2* is trained on 500 data points, *Model 1* is generally expected to perform better on *Test Set 1*.
  
  
  ---
  
  