Answers.Body: <p>Here is what I'd do:</p>
  <h2>Whats the right way to ordinal encode data?</h2>
  <p>If there's a natural ordering to the data, which is your case, then it's fine to map the data to a non-decreasing sequence of numbers. It's not very important where you start the sequence, because you should always normalize your input data.</p>
  <p>If you don't normalize the data, then it's going to make learning much more difficult: the weights will have to have wildly different magnitudes to compensate for the difference in the magnitude of your inputs, and the variance of the gradient updates is also going to be very large, which is detrimental to learning.</p>
  <h2>What's the threshold for dropping outliers?</h2>
  <p>It's actually quite important to understand whether these outliers are erroneous data or not. If not, you want to keep them and might actually need to re-balance your dataset, or change your loss function to overweight them, so that your model correctly classifies them.</p>
  <p>Imagine a situation where you want to know whether someone is a basketball player or not: your dataset will be mostly made of average heights, with a few extreme outliers which you actually want to classify as basketball players!</p>
  <h2>Always Normalize? Always Standardize?</h2>
  <p>As I explained above, always standardize</p>
  <h2>How to fill missing categorical data with similar probabilities (SEX for instance)</h2>
  <p>How to deal with missing data also depends on why the data is missing. In your case, I could imagine that whether or not the &quot;sex&quot; variable is reported by the user is highly correlated with the outcome you're trying to predict.</p>
  <p>One could imagine that people not reporting their &quot;sex&quot; are more privacy conscious and therefore have higher standards when judging a service, which would entail a higher probability of being dissatisfied.</p>
  <p>In that case, I would have a special value for missing data, and the neural network will use that information for its predictions.</p>
  <h2>Is mean the most effective way to fill null numerical data?</h2>
  <p>Same answer as above, you should consider whether or not the data is reported  is important to the classification task.</p>
  <h2>Is there a threshold for dropping highly correlated features to improve performance?</h2>
  <p>In general, if you suspect you have highly correlated features, you can do something like <a href="https://en.wikipedia.org/wiki/Principal_component_analysis" rel="nofollow noreferrer">PCA</a> first on the input data and keep only the K top components. However, K is now a hyper parameter of your algorithm, which you need to tune.</p>
  <p>In general though, neural networks are quite good at discovering correlation in the input data, so I would not try to decorrelate the input features as a preprocessing step.</p>
  <p>Hope this helps!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-06T15:38:35.230Z
Answers.Text: Thanks for the detailed reply! I felt the same way about the UNK vs Male/Female give the model a new variable to learn off of. I generally kept outliers if they were reasonable (a few age 80 folks for instance), but was really just guessing.
Answers.UserDisplayName: FIRE KhaN
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-04T20:01:40.473Z
Answers.LastActivityDate: 2022-05-04T20:01:40.473Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>I recently participated in a competitive &quot;hackathon&quot; with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and answers: ex. &quot;Seat_Comfort&quot; : Extremely Unacceptable, Unacceptable, Needs improvement, acceptable, Good, extremely good</p>
  <p>Other numerical info such as travel distance and sex/age was also included. Some Null data was throughout.</p>
  <p>My question is what is the &quot;right&quot; way to preprocess this data?</p>
  <p>The ordinal &quot;survey&quot; data I encoded as -3, -2, -1, 1, 2 ,3. Other categorical variables I hot-encoded. I standardized but DID NOT Normalize the data (trial and error did not show significant accuracy gains after normalization). For the most part I filled Nulls with averages, but did not really remove outliers first.</p>
  <p>I spent so much time on trial and error in preprocessing the data and trying new approaches that I barely had time to hypertune the actual ANN parameters. I decided to come here and ask the &quot;right&quot; way to do it, if there is such a thing or standard.</p>
  <p>For starters was ordinal encoding with negative numbers the way to do it? or should I have stuck to 1,2,3,4,5,6.</p>
  <p>travel distance mean was like 2000, should that have been Normalized considering the relatively high values? Should I have removed outliers as a matter of course?</p>
  <p>Some of the features were highly correlated with each other and I removed them (0.6 - 0.9) correlation. Is there a &quot;best case&quot; threshold for removing correlated features?</p>
  <p>At one point I filled missing &quot;SEX&quot; values with UNK, as opposed to Male/Female in the data</p>
  <p>If it matters at all, my ANN was a 3 hidden layer with dropouts and 1 TANH and 2 SELU activation. Sigmoid activation for the output obviously. I implemented a bagging, ensemble method to good results in the competition, but was still left with these burning questions.</p>
  <p>To summarize:</p>
  <p>Whats the right way to ordinal encode data?</p>
  <p>Whats the threshold for dropping outliers?</p>
  <p>Always Normalize? Always Standardize?</p>
  <p>How to fill missing categorical data with similar probabilities (SEX for instance)</p>
  <p>Is mean the most effective way to fill null numerical data?</p>
  <p>Is there a threshold for dropping highly correlated features to improve performance?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-05T06:58:33.730Z
Comments.Text: You're asking too many questions in a single post. Only one main specific question per post.
Comments.UserDisplayName: nbro
Tags: <neural-networks><data-preprocessing>
Title: Is this the right approach to preprocessing data for artificial neural-networks?
UserDisplayName: FIRE KhaN
ThreadText: # Is this the right approach to preprocessing data for artificial neural-networks?
  
  (Asked by: username_0 on 2022-05-04)
  
  I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and answers: ex. "Seat\_Comfort" : Extremely Unacceptable, Unacceptable, Needs improvement, acceptable, Good, extremely good
  
  Other numerical info such as travel distance and sex/age was also included. Some Null data was throughout.
  
  My question is what is the "right" way to preprocess this data?
  
  The ordinal "survey" data I encoded as -3, -2, -1, 1, 2 ,3. Other categorical variables I hot-encoded. I standardized but DID NOT Normalize the data (trial and error did not show significant accuracy gains after normalization). For the most part I filled Nulls with averages, but did not really remove outliers first.
  
  I spent so much time on trial and error in preprocessing the data and trying new approaches that I barely had time to hypertune the actual ANN parameters. I decided to come here and ask the "right" way to do it, if there is such a thing or standard.
  
  For starters was ordinal encoding with negative numbers the way to do it? or should I have stuck to 1,2,3,4,5,6.
  
  travel distance mean was like 2000, should that have been Normalized considering the relatively high values? Should I have removed outliers as a matter of course?
  
  Some of the features were highly correlated with each other and I removed them (0.6 - 0.9) correlation. Is there a "best case" threshold for removing correlated features?
  
  At one point I filled missing "SEX" values with UNK, as opposed to Male/Female in the data
  
  If it matters at all, my ANN was a 3 hidden layer with dropouts and 1 TANH and 2 SELU activation. Sigmoid activation for the output obviously. I implemented a bagging, ensemble method to good results in the competition, but was still left with these burning questions.
  
  To summarize:
  
  Whats the right way to ordinal encode data?
  
  Whats the threshold for dropping outliers?
  
  Always Normalize? Always Standardize?
  
  How to fill missing categorical data with similar probabilities (SEX for instance)
  
  Is mean the most effective way to fill null numerical data?
  
  Is there a threshold for dropping highly correlated features to improve performance?
  
  **Tags:** neural-networks, data-preprocessing
  > **Comments:**
  > **username_1**: You're asking too many questions in a single post. Only one main specific question per post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-04. Score: 0 | Accepted answer
  
  Here is what I'd do:
  
  ## Whats the right way to ordinal encode data?
  
  If there's a natural ordering to the data, which is your case, then it's fine to map the data to a non-decreasing sequence of numbers. It's not very important where you start the sequence, because you should always normalize your input data.
  
  If you don't normalize the data, then it's going to make learning much more difficult: the weights will have to have wildly different magnitudes to compensate for the difference in the magnitude of your inputs, and the variance of the gradient updates is also going to be very large, which is detrimental to learning.
  
  ## What's the threshold for dropping outliers?
  
  It's actually quite important to understand whether these outliers are erroneous data or not. If not, you want to keep them and might actually need to re-balance your dataset, or change your loss function to overweight them, so that your model correctly classifies them.
  
  Imagine a situation where you want to know whether someone is a basketball player or not: your dataset will be mostly made of average heights, with a few extreme outliers which you actually want to classify as basketball players!
  
  ## Always Normalize? Always Standardize?
  
  As I explained above, always standardize
  
  ## How to fill missing categorical data with similar probabilities (SEX for instance)
  
  How to deal with missing data also depends on why the data is missing. In your case, I could imagine that whether or not the "sex" variable is reported by the user is highly correlated with the outcome you're trying to predict.
  
  One could imagine that people not reporting their "sex" are more privacy conscious and therefore have higher standards when judging a service, which would entail a higher probability of being dissatisfied.
  
  In that case, I would have a special value for missing data, and the neural network will use that information for its predictions.
  
  ## Is mean the most effective way to fill null numerical data?
  
  Same answer as above, you should consider whether or not the data is reported is important to the classification task.
  
  ## Is there a threshold for dropping highly correlated features to improve performance?
  
  In general, if you suspect you have highly correlated features, you can do something like [PCA](https://en.wikipedia.org/wiki/Principal_component_analysis) first on the input data and keep only the K top components. However, K is now a hyper parameter of your algorithm, which you need to tune.
  
  In general though, neural networks are quite good at discovering correlation in the input data, so I would not try to decorrelate the input features as a preprocessing step.
  
  Hope this helps!
  
  > **Comments:**
  > **username_0**: Thanks for the detailed reply! I felt the same way about the UNK vs Male/Female give the model a new variable to learn off of. I generally kept outliers if they were reasonable (a few age 80 folks for instance), but was really just guessing.  
  
  ---
  
  