Answers.Body: <p>You should look into &quot;missing values&quot;. This is an entire research field in itself.</p>
  <p>First, you need to identify the type of missing values:</p>
  <ol>
  <li>They can be missing purely at random.</li>
  <li>Whether they are missing or not is itself a useful feature, and should be treated as a class of its own.</li>
  </ol>
  <p>(Those two are the best case scenarios.)</p>
  <ol start="3">
  <li>Whether they are missing or not depends on the underlying (unknown) value. For example, a thermometer might fail occasionally if the temperatures get too high. In your case, certain types of companies might be more likely to not share their information.</li>
  <li>Information might be missing specifically to mislead you, the data analyst. This is the worst possible scenario, and there is not much you can do.</li>
  </ol>
  <p>So, what do you do about it? A few typical options:</p>
  <ol>
  <li>Throw out all the rows with missing data: we do not have enough information about these companies.</li>
  <li>Throw out all the columns with missing data: this field is not reliably measurable and we shouldn't use it.</li>
  <li>Try to guess the missing values. This can be done if the amount of missing data is small. Either you train a predictive model based on the non-missing data, or you fill in the median for that type of row, or you fill in the value of the &quot;closest&quot; matching row. This can be dangerous.</li>
  <li>Some algorithms are OK with missing data. Check the documentation for your models and algorithms to see how they deal with missing values.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-12T23:20:27.730Z
Answers.Text: Thanks for an answer! I've got this data in some competition, so it is possible that missing values were selected randomly (but on purpose) by orgs. It looks like a real situation: some companies - are already clients, and some are new so we don't know much. I think that in my case throw all columns is bad - we'll miss too much data. And throw out all this rows also not an option, because test data has the same structure - half of record without financial info.
Answers.UserDisplayName: Denis Ka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-12T23:20:34.477Z
Answers.Text: I think it will be good to find out whether the fact of missing finance is a feature. But can i make (and how) more than 2 classes on presented data? How can i categorise several features into a new feature ? %)
Answers.UserDisplayName: Denis Ka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T01:15:44.010Z
Answers.Text: I'd introduce a new boolean feature: "is_missing". Then you can see if it correlates with any of the known features. To see if it can be predicted, take the part of the data set where it is available, and train a predictor on that particular feature. Depending on how well it does (validation score) you can try to predict the missing ones.
Answers.UserDisplayName: Robby Goetschalckx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T01:16:07.200Z
Answers.Text: Essentially you're treating the possibly missing value as the target of a prediction problem.
Answers.UserDisplayName: Robby Goetschalckx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-12T21:58:43.070Z
Answers.LastActivityDate: 2020-09-12T21:58:43.070Z
Answers.UserDisplayName: Robby Goetschalckx
Body: <p>Sorry if this is too noob question, I'm just a beginner.</p>
  <p>I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of employees and date of registration)</p>
  <p>I have to predict the probability of default. And the data has gaps: about the half of the companies have no financial data at all. But general features are 100% filled.</p>
  <p>What is the best practice for such a situation?</p>
  <p>Will be great if you can give some example links to read.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-13T02:31:34.700Z
Comments.Text: May help https://stats.stackexchange.com/questions/98953/why-doesnt-random-forest-handle-missing-values-in-predictors
Comments.UserDisplayName: caveman
Tags: <machine-learning><prediction><data-preprocessing><feature-engineering>
Title: How to perform prediction when some features have missing values?
UserDisplayName: Denis Ka
ThreadText: # How to perform prediction when some features have missing values?
  
  (Asked by: username_0 on 2020-09-12)
  
  Sorry if this is too noob question, I'm just a beginner.
  
  I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of employees and date of registration)
  
  I have to predict the probability of default. And the data has gaps: about the half of the companies have no financial data at all. But general features are 100% filled.
  
  What is the best practice for such a situation?
  
  Will be great if you can give some example links to read.
  
  **Tags:** machine-learning, prediction, data-preprocessing, feature-engineering
  > **Comments:**
  > **username_1**: May help https://stats.stackexchange.com/questions/98953/why-doesnt-random-forest-handle-missing-values-in-predictors  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-09-12. Score: 3
  
  You should look into "missing values". This is an entire research field in itself.
  
  First, you need to identify the type of missing values:
  
  1. They can be missing purely at random.
  2. Whether they are missing or not is itself a useful feature, and should be treated as a class of its own.
  
  (Those two are the best case scenarios.)
  
  3. Whether they are missing or not depends on the underlying (unknown) value. For example, a thermometer might fail occasionally if the temperatures get too high. In your case, certain types of companies might be more likely to not share their information.
  4. Information might be missing specifically to mislead you, the data analyst. This is the worst possible scenario, and there is not much you can do.
  
  So, what do you do about it? A few typical options:
  
  1. Throw out all the rows with missing data: we do not have enough information about these companies.
  2. Throw out all the columns with missing data: this field is not reliably measurable and we shouldn't use it.
  3. Try to guess the missing values. This can be done if the amount of missing data is small. Either you train a predictive model based on the non-missing data, or you fill in the median for that type of row, or you fill in the value of the "closest" matching row. This can be dangerous.
  4. Some algorithms are OK with missing data. Check the documentation for your models and algorithms to see how they deal with missing values.
  
  > **Comments:**
  > **username_0**: Thanks for an answer! I've got this data in some competition, so it is possible that missing values were selected randomly (but on purpose) by orgs. It looks like a real situation: some companies - are already clients, and some are new so we don't know much. I think that in my case throw all columns is bad - we'll miss too much data. And throw out all this rows also not an option, because test data has the same structure - half of record without financial info.  
  > **username_0**: I think it will be good to find out whether the fact of missing finance is a feature. But can i make (and how) more than 2 classes on presented data? How can i categorise several features into a new feature ? %)  
  > **username_2**: I'd introduce a new boolean feature: "is_missing". Then you can see if it correlates with any of the known features. To see if it can be predicted, take the part of the data set where it is available, and train a predictor on that particular feature. Depending on how well it does (validation score) you can try to predict the missing ones.  
  > **username_2**: Essentially you're treating the possibly missing value as the target of a prediction problem.  
  
  ---
  
  