Answers.Body: <p>In my experience, it is more common to use all <em>clinical</em> data as-is for clinical studies. And if data is missing, either omit the sample or omit the variable with missing data.</p>
  
  <p>If your classifier can't handle the wide variation commonly seen in clinical studies then you may want to use a classifier which is less impacted by outliers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-01T12:50:33.130Z
Answers.Text: Thanks for the response. Upvoted.
Answers.UserDisplayName: The Great
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-31T03:36:12.570Z
Answers.LastActivityDate: 2019-12-31T03:36:12.570Z
Answers.UserDisplayName: Reilstein
Answers.Body: <p>Removing outliers is common practice in statistical modeling and perfectly acceptable. However, with regards 1.5 IQR I am far from certain about this approach. Normally, if you want to be conservative then 3 standard deviations (SD) denote an outlier, which is more stringent than IQR. Some use 2 SD. If the value is lower than 2 SD from the group mean it isn't an outlier, that's because 1.96 SD is the normal distribution.</p>
  
  <p>Normally you would delete the outlier and it takes no further part in the calculation, assigning it to 'na' should work. The subsequent calculation will recognise an 'na' value, I think in python it is np('na') ... Np = numpy.</p>
  
  <p>In this case you appear to be deleting the value and treating it as missing data. Treatment of missing data is complicated I agree 100%. Mean, median are possibilities for missing data I can't see this working for binary data. One approach is regression and in python there is a command specifically to do this (which I forget), R will have the same command and it will assign a value  for the missing data. However, you can't do this for outliers.</p>
  
  <p>The other routes to remove outliers are via transformation, log transformation is the most common. Transformation is a big subject.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-01T00:51:29.920Z
Answers.Text: thanks upvoted for the response
Answers.UserDisplayName: The Great
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-31T14:35:46.613Z
Answers.LastActivityDate: 2019-12-31T14:35:46.613Z
Answers.UserDisplayName: M__
Body: <p>I am working on clinical study where I am focusing on binary classification problem.</p>
  
  <p>There are few variables which has outliers.</p>
  
  <p>I am trying to know which would be the right value to replace outliers in a clinical study?</p>
  
  <p>Is it <code>mean</code>, <code>median</code> <code>min</code>,<code>max</code> or <code>mode</code> of a column of interest?</p>
  
  <p>Currently I use <code>min</code> and <code>max</code> for values below and above 1.5*IQR threshold</p>
  
  <p>I use <code>IQR</code> approach to identify outliers but would like to know how it is done usually in clinical studies?</p>
  
  <p>Can anyone help me with this? </p>
  
Tags: <public-databases><literature-search><data-management><data-preprocessing><public-dataset>
Title: Appropriate value to replace outliers in clinical studies
ThreadText: # Appropriate value to replace outliers in clinical studies
  
  (Asked by: username_0 on 2019-12-30)
  
  I am working on clinical study where I am focusing on binary classification problem.
  
  There are few variables which has outliers.
  
  I am trying to know which would be the right value to replace outliers in a clinical study?
  
  Is it `mean`, `median` `min`,`max` or `mode` of a column of interest?
  
  Currently I use `min` and `max` for values below and above 1.5\*IQR threshold
  
  I use `IQR` approach to identify outliers but would like to know how it is done usually in clinical studies?
  
  Can anyone help me with this?
  
  **Tags:** public-databases, literature-search, data-management, data-preprocessing, public-dataset
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-12-31. Score: 3
  
  In my experience, it is more common to use all *clinical* data as-is for clinical studies. And if data is missing, either omit the sample or omit the variable with missing data.
  
  If your classifier can't handle the wide variation commonly seen in clinical studies then you may want to use a classifier which is less impacted by outliers.
  
  > **Comments:**
  > **username_0**: Thanks for the response. Upvoted.  
  
  ---
  
  ### Answer by username_2 on 2019-12-31. Score: 1
  
  Removing outliers is common practice in statistical modeling and perfectly acceptable. However, with regards 1.5 IQR I am far from certain about this approach. Normally, if you want to be conservative then 3 standard deviations (SD) denote an outlier, which is more stringent than IQR. Some use 2 SD. If the value is lower than 2 SD from the group mean it isn't an outlier, that's because 1.96 SD is the normal distribution.
  
  Normally you would delete the outlier and it takes no further part in the calculation, assigning it to 'na' should work. The subsequent calculation will recognise an 'na' value, I think in python it is np('na') ... Np = numpy.
  
  In this case you appear to be deleting the value and treating it as missing data. Treatment of missing data is complicated I agree 100%. Mean, median are possibilities for missing data I can't see this working for binary data. One approach is regression and in python there is a command specifically to do this (which I forget), R will have the same command and it will assign a value for the missing data. However, you can't do this for outliers.
  
  The other routes to remove outliers are via transformation, log transformation is the most common. Transformation is a big subject.
  
  > **Comments:**
  > **username_0**: thanks upvoted for the response  
  
  ---
  
  