Answers.Body: <p>While I can see that there are some heuristics that can tell you whether an entry is 'weird', I don't see any way that you can correct this. Where would you get a correct value from?</p>
  
  <p>I would perhaps start with a statistical analysis, looking at the distribution of values to get an idea of the state the data is in. From this you can then already see some values that wouldn't make sense (this depends very much on what data it is: census data will include toddlers, but credit card applications wouldn't).</p>
  
  <p>Your best bet is to encode some constraints, such as a minimum/maximum range for each value, and perhaps some correspondences, such as a maximum weight for a certain age range. This you can do with simple conditional statements in Python.</p>
  
  <p>You can then flag up those entries which don't look quite right. What you do with them depends on your application. You can filter them out, or manually change the values to something that is consistent with possible values.</p>
  
  <p>Machine Learning techniques would not be able to do anything better than that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-10T18:22:10.723Z
Answers.LastActivityDate: 2020-02-10T18:22:10.723Z
Answers.UserDisplayName: Oliver Mason
Body: <p>I'm working on data cleaning and I'm stuck. I have a data set with 3 columns: <em>id</em>, <em>age</em>, and <em>weight</em>.</p>
  
  <p>Supposing I have an entry:</p>
  
  <pre><code>id:1 | age:3 (years)  | weight: 150 (kg)
  </code></pre>
  
  <p>How can we detect that the information is wrong, assuming I have a thousand lines? </p>
  
  <p>And how can I correct it (using Python)?</p>
  
  <p>Is there any function in Python that I can use or should I use machine learning techniques?</p>
  
Tags: <data-preprocessing>
Title: How to automatically detect and correct false information in columnar data?
UserDisplayName: souum
ThreadText: # How to automatically detect and correct false information in columnar data?
  
  (Asked by: username_0 on 2020-02-10)
  
  I'm working on data cleaning and I'm stuck. I have a data set with 3 columns: id, age, and weight.
  Supposing I have an entry:
  id:1 | age:3 (years)  | weight: 150 (kg)
  
  How can we detect that the information is wrong, assuming I have a thousand lines? 
  And how can I correct it (using Python)?
  Is there any function in Python that I can use or should I use machine learning techniques?
  
  **Tags:** data-preprocessing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-10. Score: 2
  
  While I can see that there are some heuristics that can tell you whether an entry is 'weird', I don't see any way that you can correct this. Where would you get a correct value from?
  
  I would perhaps start with a statistical analysis, looking at the distribution of values to get an idea of the state the data is in. From this you can then already see some values that wouldn't make sense (this depends very much on what data it is: census data will include toddlers, but credit card applications wouldn't).
  
  Your best bet is to encode some constraints, such as a minimum/maximum range for each value, and perhaps some correspondences, such as a maximum weight for a certain age range. This you can do with simple conditional statements in Python.
  
  You can then flag up those entries which don't look quite right. What you do with them depends on your application. You can filter them out, or manually change the values to something that is consistent with possible values.
  
  Machine Learning techniques would not be able to do anything better than that.
  
  
  ---
  
  