Should I treat these data points as outliers? Currently, I am building my analytics portfolio as part of the Google Data Analytics course. I chose the option to analyze Divvy Bike Sharing data for the year 2021. But now I'm currently stuck in the part where I need to identify outliers in the dataset. I'm focusing on the 'ride_length' column which shows the duration of each ride and I'm using two methods which are:

*

*IQR (data points that fall below 25th or above 75th percentile are outliers)

*1% and 99% rule (data points that fall below 1% percentile or above 99% percentile are outliers)

Note: the ride_length column is counted in minutes
A) IQR METHOD
The first method that I use to detect outliers is the IQR proximity rule (The data points which fall below the 25th percentile or above the 75th percentile are outliers). Here's the code:
lower_bound_iqr <- quantile(df_2021_test$ride_length, 0.25)
upper_bound_iqr <- quantile(df_2021_test$ride_length, 0.75)

lower_bound_iqr
25% 
6.98 

upper_bound_iqr
75% 
20.98 

Key takeaways:

*

*ride_length that falls below 6.98 minutes is considered an outlier

*ride_length that falls above 20.98 minutes is considered an outlier

Then I count the percentages of outliers in the data:
outliers_iqr <- which(df_2021_test$ride_length < lower_bound_iqr | df_2021_test$ride_length > upper_bound_iqr)
(count(df_2021_test[outliers_iqr, ]) / count(df_2021_test)) * 100 

         n
1 49.89316

The result is that 49.89 % of data are considered outliers. I think this is too much data to exclude for the analysis to begin as it will reduce the accuracy of the analysis. Or am I wrong? Therefore I move to the second method
B) 1% and 99% Percentile Rule
This method state that data points that are far from the 99% percentile and less than 1% percentile are considered an outlier. Here's the code:
lower_bound <- quantile(df_2021_test$ride_length, 0.01)
upper_bound <- quantile(df_2021_test$ride_length, 0.99)

lower_bound
1% 
1.82 
upper_bound
99% 
115.63 

Key takeaways:

*

*ride_length that falls below 1.82 minutes are considered outliers

*ride_length that falls above 115.63 minutes (approx. 2 hours) are considered outliers

Again, I count the percentages of outliers in the data:
outliers <- which(df_2021_test$ride_length < lower_bound | df_2021_test$ride_length > upper_bound)
(count(df_2021_test[outliers, ]) / count(df_2021_test)) * 100

         n
1 1.982182

The result is that 1.98 % of data are considered outliers. I think this is fine to exclude for the analysis to begin as it will not reduce the accuracy of the analysis that much. Or am I wrong?
Here are my questions:

*

*When identifying outliers in the data, what should you choose between the two of the method above? Or is there another better method?

*Is my way of identifying outliers in the dataset correct? Or am I missing something?

I have detailed all of my steps to identify outliers above and again, it's not an error in the code it's just that I'm confused as to whether my method of identifying them is correct or if is there any better way or something that I miss.

It should be obvious that 50% of your data will fall above the 75th or below the 25th centile, and that 2% is either above the 99th or below the 1st.
There is no data-driven answer to 'what is an outlier'.  Data points might be far away from the rest of a distribution for many different reasons, and how you identify these and what you do with them should depend on why you think they occurred and what downstream analysis you are planning (ultimately what question you are asking of the data).
If you can specify these things you'll have a clearer idea of how to identify and handle outliers.  Do not rely on simple data-driven rules.
