Answers.Body: <p>I feel like from the information your giving (<em>some</em> sort of biased data) you can't get an answer as robust as you'd like (what <em>algorithmic</em> changes can be made).</p>
  <p>In general, the reason these methods like DNN's work is that they learn from the data. What you train it to do is what it is capable of, and there's little one can do to 'balance' it to classes of data it just never sees. It's like training someone to do algebra then giving them a trigonometry test. It's all math, sure, but you just never can expect much without the proper learning.</p>
  <p>That being said, you should perhaps look at other methods to work with this data, or to approach the problem. Given that you cannot collect unbiased data and that you can't explain more due to confidentiality, I really doubt anyone here can help you that much.</p>
  <p>I can at the most point you to this article : <a href="https://ai2-s2-pdfs.s3.amazonaws.com/277c/3795f7a66fda3fd70607d1fb45b66730c7ba.pdf" rel="nofollow noreferrer">&quot;Classification on Data with Biased Class Distribution&quot;</a>.</p>
  <p>And suggest that perhaps your current approach may not be the most appropriate given the unfortunate circumstances.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-01-12T12:22:50.350Z
Answers.Text: @Tejal Could you post this paper as an answer with some details why it was relevant to your problem and how did you solve it.
Answers.UserDisplayName: kenorb
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-22T23:45:18.837Z
Answers.Text: Thanks! I found this paper that is very relevant to my problem - https://arxiv.org/pdf/1412.6596v3.pdf. I should have mentioned that my labels are very noisy, so stratified sampling is not a solution.
Answers.UserDisplayName: Tejal
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-23T13:47:56.020Z
Answers.Text: I see. Good to know you found a relevant paper. Good luck in your research!
Answers.UserDisplayName: Avik Mohan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-08-22T21:58:39.513Z
Answers.LastActivityDate: 2021-04-02T19:55:36.483Z
Answers.LastEditDate: 2021-04-02T19:55:36.483Z
Answers.UserDisplayName: Avik Mohan
Body: <p>The obvious solution is to ensure that the training data is balanced - but in my particular case that is impossible. What corrections can one perform in such a scenario?</p>
  
  <p>I know that my training data is heavily biased towards a particular class, say, and I cannot change that. Moreover, the labels are very noisy. Conditioned on this piece of information, is there anything I can do by tweaking the training process itself/ something else, to correct for the bias in the training data?</p>
  
  <p>The data comes from an experiment (from an electron microscope), and I cannot collect more data. It's always going to be biased in this way, so alternatively-biased is also not an option. I'm sorry that I'm unable to provide any more details due to confidentiality.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T19:53:34.690Z
Comments.Text: Is this a programming question?
Comments.UserDisplayName: Mithical
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T20:08:56.393Z
Comments.Text: I'm VTCing as 'Unclear what you're asking'.
Comments.UserDisplayName: Mithical
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T20:10:11.067Z
Comments.Text: No, I'm talking about algorithmic changes
Comments.UserDisplayName: Tejal
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T20:12:17.040Z
Comments.Text: You'll have to add a lot more info into your question. Like, why the solution is impossible in your case. I'm still not sure what your problem is...
Comments.UserDisplayName: Mithical
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T20:16:31.573Z
Comments.Text: Let me rephrase - I know that my training data is heavily biased towards a particular class, say, and I cannot change that. Conditioned on this piece of information, is there anything I can do by tweaking the training process itself/ something else, to correct for the bias in the training data? Does this make more sense?
Comments.UserDisplayName: Tejal
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T20:19:07.687Z
Comments.Text: Please edit it into your post.
Comments.UserDisplayName: Mithical
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T20:20:04.177Z
Comments.Text: What prevents you from adding additional unbiased (or alternatively-biased) training data?
Comments.UserDisplayName: NietzscheanAI
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-22T20:22:06.703Z
Comments.Text: The data comes from an experiment (from an electron microscope), and I cannot collect more data. It's always going to be biased in this way, so alternatively-biased is also not an option. I'm sorry that I'm unable to provide any more details due to confidentiality.
Comments.UserDisplayName: Tejal
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-23T07:10:36.687Z
Comments.Text: @Tejal Concrete details aren't really necessary - it should be entirely possible to discuss this in the abstract. Why not fit a parameteric statistical model (e.g. Mixture of Gaussians) to your data, play around with the model parameters to adjust the bias, then use the model as a generator for the training set?
Comments.UserDisplayName: NietzscheanAI
Tags: <neural-networks><research><deep-neural-networks><training>
Title: What can be done to correct for sampling bias introduced from (noisy) training data while training a DNN?
UserDisplayName: Tejal
ThreadText: # What can be done to correct for sampling bias introduced from (noisy) training data while training a DNN?
  
  (Asked by: username_0 on 2016-08-22)
  
  The obvious solution is to ensure that the training data is balanced - but in my particular case that is impossible. What corrections can one perform in such a scenario?
  
  I know that my training data is heavily biased towards a particular class, say, and I cannot change that. Moreover, the labels are very noisy. Conditioned on this piece of information, is there anything I can do by tweaking the training process itself/ something else, to correct for the bias in the training data?
  
  The data comes from an experiment (from an electron microscope), and I cannot collect more data. It's always going to be biased in this way, so alternatively-biased is also not an option. I'm sorry that I'm unable to provide any more details due to confidentiality.
  
  **Tags:** neural-networks, research, deep-neural-networks, training
  
  > **Comments:**
  > **username_1**: Is this a programming question?  
  > **username_1**: I'm VTCing as 'Unclear what you're asking'.  
  > **username_0**: No, I'm talking about algorithmic changes  
  > **username_1**: You'll have to add a lot more info into your question. Like, why the solution is impossible in your case. I'm still not sure what your problem is...  
  > **username_0**: Let me rephrase - I know that my training data is heavily biased towards a particular class, say, and I cannot change that. Conditioned on this piece of information, is there anything I can do by tweaking the training process itself/ something else, to correct for the bias in the training data? Does this make more sense?  
  > **username_1**: Please edit it into your post.  
  > **username_2**: What prevents you from adding additional unbiased (or alternatively-biased) training data?  
  > **username_0**: The data comes from an experiment (from an electron microscope), and I cannot collect more data. It's always going to be biased in this way, so alternatively-biased is also not an option. I'm sorry that I'm unable to provide any more details due to confidentiality.  
  > **username_2**: @username_0 Concrete details aren't really necessary - it should be entirely possible to discuss this in the abstract. Why not fit a parameteric statistical model (e.g. Mixture of Gaussians) to your data, play around with the model parameters to adjust the bias, then use the model as a generator for the training set?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2016-08-22. Score: 2
  
  I feel like from the information your giving (*some* sort of biased data) you can't get an answer as robust as you'd like (what *algorithmic* changes can be made).
  
  In general, the reason these methods like DNN's work is that they learn from the data. What you train it to do is what it is capable of, and there's little one can do to 'balance' it to classes of data it just never sees. It's like training someone to do algebra then giving them a trigonometry test. It's all math, sure, but you just never can expect much without the proper learning.
  
  That being said, you should perhaps look at other methods to work with this data, or to approach the problem. Given that you cannot collect unbiased data and that you can't explain more due to confidentiality, I really doubt anyone here can help you that much.
  
  I can at the most point you to this article : ["Classification on Data with Biased Class Distribution"](https://ai2-s2-pdfs.s3.amazonaws.com/277c/3795f7a66fda3fd70607d1fb45b66730c7ba.pdf).
  
  And suggest that perhaps your current approach may not be the most appropriate given the unfortunate circumstances.
  
  > **Comments:**
  > **username_0**: Thanks! I found this paper that is very relevant to my problem - https://arxiv.org/pdf/1412.6596v3.pdf. I should have mentioned that my labels are very noisy, so stratified sampling is not a solution.  
  > **username_3**: I see. Good to know you found a relevant paper. Good luck in your research!  
  > **username_4**: @username_0 Could you post this paper as an answer with some details why it was relevant to your problem and how did you solve it.  
  
  ---
  
  