Answers.Body: <p>Don't care classes are not advisable since don't care doesn't have any inherent pattern which would hurt your model inadvertently.</p>
  <p>You can assign class weights for your classes. CrossEntropyLoss has an argument 'weights'.</p>
  <p>BatchNormalization can be left alone since the normalization should happen on the basis of all data and not just some of the data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T00:55:24.510Z
Answers.Text: So a 0 weight shouldn't be used? How would I solve the problem then?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T05:52:31.177Z
Answers.Text: Adding a dummy class I said would be bad. Never said that bout the weight, set 0 or very low, try both.
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T11:10:34.263Z
Answers.Text: Please explain why a 0 weight is different than a dummy
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T11:50:35.263Z
Answers.Text: A dummy variable will add an extra class while class weights operate inside the already existing labels. By giving less weights to them they will be factored less in loss calculation. But, if you add dummy class then you will have a new class then the loss will  be calculated against (num_classes + 1) classes. Since, your dummy class will have no frequentist patterns in them, it will just confuse your model while minimizing its loss. Also, using class weights is easy. Or you can try out both and see the results.
Answers.UserDisplayName: Abhishek Verma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T15:50:31.487Z
Answers.Text: Sorry, I am not following. Please explain how using weights is mathematically different than what I propose - predicting just 3 classes, and not backpropagating for other classes. If I allow predicting for 5 classes, how should I handle those at inference?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T01:09:04.127Z
Answers.Text: @Gulzar I think the best suggestion is to just try both. I personally don't believe adding a fourth "dont care" class will hurt the model, as there is a pattern to it - low confidence in the other 3 classes. But there's no way to know without trying, so try with the CE  weights, then try with a don't care class and compare performance
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-11T19:02:03.083Z
Answers.LastActivityDate: 2021-03-11T19:02:03.083Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>This is a theoretical question.</p>
  <h2>Setup</h2>
  <p>I have a time series classification task in which I should output a classification of 3 classes for every time stamp <code>t</code>.</p>
  <p>All data is labeled per frame.</p>
  <h2>The problem:</h2>
  <p>In the data set are more than 3 classes [which are also imbalanced].</p>
  <p>My net should see all samples sequentially, because it uses that for historical information.<br />
  Thus, I can't just eliminate all irrelevant class samples at preprocessing time.</p>
  <p>In case of a prediction on a frame which is labeled differently than those 3 classes, I don't care about the result.</p>
  <hr />
  <h2>My thoughts:</h2>
  <ol>
  <li>The net will predict for 3 classes</li>
  <li>The net will only learn (pass backward gradient) for valid classes, and just not calculate loss for other classes.</li>
  </ol>
  <h2>Questions</h2>
  <ol>
  <li>Is this the way to go for &quot;don't care&quot; classes in classification?</li>
  <li>How to calculate loss only for relevant classes in Pytorch?</li>
  <li>Should I apply some normalization per batch, or change batch norm layers if dropping variable samples per batch?</li>
  </ol>
  <p>I am using <code>nn.CrossEntropyLoss()</code> as my criterion, which has only <code>mean</code> or <code>sum</code> as reductions.<br />
  I need to mask the batch so that the reduction will only apply for samples whose label is valid.</p>
  <p>I could use <code>reduction='none'</code> and do that manually, or I could do that before the loss and keep using <code>reduction='mean'</code>.<br />
  Is there some method to do this using built in Pytorth tools?</p>
  <p>Maybe this can be done in the data-fetching phase somehow?</p>
  <hr />
  <p>I am looking some standard, vanilla, thumb rule implementation to tackle this. The least fancy the better.</p>
  <hr />
  <p>I am aware this is more than a single question.
  They are still not separable, as the solution will be unified most likely.</p>
  
Tags: <deep-learning><classification><long-short-term-memory><time-series><normalisation>
Title: How to define a "don't care" class in time series classification in Pytorch?
UserDisplayName: Gulzar
ThreadText: # How to define a "don't care" class in time series classification in Pytorch?
  
  (Asked by: username_0 on 2021-03-11)
  
  This is a theoretical question.
  
  ## Setup
  
  I have a time series classification task in which I should output a classification of 3 classes for every time stamp `t`.
  
  All data is labeled per frame.
  
  ## The problem:
  
  In the data set are more than 3 classes [which are also imbalanced].
  
  My net should see all samples sequentially, because it uses that for historical information.  
  Thus, I can't just eliminate all irrelevant class samples at preprocessing time.
  
  In case of a prediction on a frame which is labeled differently than those 3 classes, I don't care about the result.
  
  ---
  
  ## My thoughts:
  
  1. The net will predict for 3 classes
  2. The net will only learn (pass backward gradient) for valid classes, and just not calculate loss for other classes.
  
  ## Questions
  
  1. Is this the way to go for "don't care" classes in classification?
  2. How to calculate loss only for relevant classes in Pytorch?
  3. Should I apply some normalization per batch, or change batch norm layers if dropping variable samples per batch?
  
  I am using `nn.CrossEntropyLoss()` as my criterion, which has only `mean` or `sum` as reductions.  
  I need to mask the batch so that the reduction will only apply for samples whose label is valid.
  
  I could use `reduction='none'` and do that manually, or I could do that before the loss and keep using `reduction='mean'`.  
  Is there some method to do this using built in Pytorth tools?
  
  Maybe this can be done in the data-fetching phase somehow?
  
  ---
  
  I am looking some standard, vanilla, thumb rule implementation to tackle this. The least fancy the better.
  
  ---
  
  I am aware this is more than a single question.
  They are still not separable, as the solution will be unified most likely.
  
  **Tags:** deep-learning, classification, long-short-term-memory, time-series, normalisation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-11. Score: 0
  
  Don't care classes are not advisable since don't care doesn't have any inherent pattern which would hurt your model inadvertently.
  
  You can assign class weights for your classes. CrossEntropyLoss has an argument 'weights'.
  
  BatchNormalization can be left alone since the normalization should happen on the basis of all data and not just some of the data.
  
  > **Comments:**
  > **username_0**: So a 0 weight shouldn't be used? How would I solve the problem then?  
  > **username_1**: Adding a dummy class I said would be bad. Never said that bout the weight, set 0 or very low, try both.  
  > **username_0**: Please explain why a 0 weight is different than a dummy  
  > **username_1**: A dummy variable will add an extra class while class weights operate inside the already existing labels. By giving less weights to them they will be factored less in loss calculation. But, if you add dummy class then you will have a new class then the loss will  be calculated against (num_classes + 1) classes. Since, your dummy class will have no frequentist patterns in them, it will just confuse your model while minimizing its loss. Also, using class weights is easy. Or you can try out both and see the results.  
  > **username_0**: Sorry, I am not following. Please explain how using weights is mathematically different than what I propose - predicting just 3 classes, and not backpropagating for other classes. If I allow predicting for 5 classes, how should I handle those at inference?  
  > **username_2**: @Gulzar I think the best suggestion is to just try both. I personally don't believe adding a fourth "dont care" class will hurt the model, as there is a pattern to it - low confidence in the other 3 classes. But there's no way to know without trying, so try with the CE  weights, then try with a don't care class and compare performance  
  
  ---
  
  