Answers.Body: <p>Happy to give a general approach, we would need more context to pinpoint the actual problem.</p>
  <p>In general you should try to understand the data before applying any model. If you are using Neural Networks the distribution of the data should not be an issue. You should first check for:</p>
  <ul>
  <li>Outliers</li>
  <li>Correlation between variables</li>
  <li>Scale of the features</li>
  <li>Size of the different populations (you mentioned you are analysing multiple vendors, do you have the same amount of data for each?)</li>
  <li>Missing values</li>
  <li>Data corruption</li>
  </ul>
  <p>The different distribution shapes might indicate an issue with the data, e.g. some vendors might not record the data accurately or as often as others. If you think that there are no underlying issues, then you could try:</p>
  <ul>
  <li>Use the average to differentiate the vendors (i.e. median if the distribution is heavily skewed, mean if its a normal distribution)</li>
  <li>Use ANOVA to classify each vendor</li>
  <li>Scale your features so that each fall within the same range, or try to centre their distributions</li>
  </ul>
  <p>A good model must also be tested for its generalization performance (i.e. how well it responds to new data points), this has more to do with not overfitting the model during the training process.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-23T12:40:01.413Z
Answers.LastActivityDate: 2022-07-23T12:40:01.413Z
Answers.UserDisplayName: DCrown
Body: <p>I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my hypothesis, as I see my target variable shifts and changes value ranges with time, as different vendors' data is achieved.</p>
  <p>Is there a way, in a real application system, to implement a mechanism to detect such data drift/change (whether in target variable or features distribution)?</p>
  <p>If so, how can I deal with it? Should I be constantly re-training a model to deal with the new data? What is the common practices to deal with new observed data?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-23T08:24:47.467Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <machine-learning><data-preprocessing><data-drift>
Title: How to detect and deal with data distribution drift/change?
UserDisplayName: John Coles
ThreadText: # How to detect and deal with data distribution drift/change?
  
  (Asked by: username_0 on 2022-07-23)
  
  I'm working on a problem in ML to assess the performance of multiple vendors. I have a set of features in my dataset, and it appears each vendor is characterized by its own distribution. This is my hypothesis, as I see my target variable shifts and changes value ranges with time, as different vendors' data is achieved.
  
  Is there a way, in a real application system, to implement a mechanism to detect such data drift/change (whether in target variable or features distribution)?
  
  If so, how can I deal with it? Should I be constantly re-training a model to deal with the new data? What is the common practices to deal with new observed data?
  
  **Tags:** machine-learning, data-preprocessing, data-drift
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-07-23. Score: 1
  
  Happy to give a general approach, we would need more context to pinpoint the actual problem.
  
  In general you should try to understand the data before applying any model. If you are using Neural Networks the distribution of the data should not be an issue. You should first check for:
  
  * Outliers
  * Correlation between variables
  * Scale of the features
  * Size of the different populations (you mentioned you are analysing multiple vendors, do you have the same amount of data for each?)
  * Missing values
  * Data corruption
  
  The different distribution shapes might indicate an issue with the data, e.g. some vendors might not record the data accurately or as often as others. If you think that there are no underlying issues, then you could try:
  
  * Use the average to differentiate the vendors (i.e. median if the distribution is heavily skewed, mean if its a normal distribution)
  * Use ANOVA to classify each vendor
  * Scale your features so that each fall within the same range, or try to centre their distributions
  
  A good model must also be tested for its generalization performance (i.e. how well it responds to new data points), this has more to do with not overfitting the model during the training process.
  
  
  ---
  
  