Random forest vs. XGBoost vs. MLP Regressor for estimating claims costs
Context
I'm building a (toy) machine learning model estimate the cost of an insurance claim (injury related). Aim is to teach myself machine learning by doing. I have settled on three algorithms to test: Random forest, XGBoost and a multi-layer perceptron.
Data set
The data set has the following columns:
python cols = [ 'AGE_RANGE', 'GENDER', 'TOTAL_PAID', 'INDUSTRY_DESCRIPTION', 'WORKER_AGE', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']
'TOTAL_PAID' is the label ($s). 392
The rest are features. The majority of features are categorical:
python categoricals = [ 'AGE_RANGE', 'GENDER', 'INDUSTRY_DESCRIPTION', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']
Code:
Import:
```python import pandas as pd import numpy as np
cols = [ 'AGE_RANGE', 'GENDER', 'TOTAL_PAID', 'INDUSTRY_DESCRIPTION', 'WORKER_AGE', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE'] features = pd.read_csv('gs://longtailclaims2/filename.csv', usecols = cols, header=0, encoding='ISO-8859-1') categoricals = [ 'AGE_RANGE', 'GENDER', 'INDUSTRY_DESCRIPTION', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE'] ```
First I turn categorical values into 0s and 1s:
python features2 = pd.get_dummies(features, columns = categoricals)
Then I isolate features from labels (TOTAL_PAID):
python labels = np.array(features['TOTAL_PAID']) features = features2.drop('TOTAL_PAID', axis = 1) feature_list = list(features.columns) feature_list_no_facts = list(features.columns)
Spit into SK Learn training and test set:
```python
Using Skicit-learn to split data into training and testing sets
from sklearn.model_selection import train_test_split
Split the data into training and testing sets
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size = 0.25, random_state = 42) test_features.head(5) test_features.head(5) ```
Then I explore the data:
print('Training Features Shape:', train_features.shape) print('Training Labels Shape:', train_labels.shape) print('Testing Features Shape:', test_features.shape) print('Testing Labels Shape:', test_labels.shape) Output: Training Features Shape: (128304, 337) Training Labels Shape: (128304,) Testing Features Shape: (42768, 337) Testing Labels Shape: (42768,)
1. XGBRegressor
First we try training XGBoost model. I needed so push # estimates above 10,000 to get a decent accuracy (R2 > 0.94)
```python
Import the model we are using
from xgboost import XGBRegressor from sklearn.ensemble import RandomForestRegressor from sklearn import ensemble from sklearn.metrics import mean_squared_error
x = XGBRegressor(random_state = 44, n_jobs = 8, n_estimators = 10000, max_depth=10, verbosity = 3) x.fit(train_features, train_labels) print('xgboost train score: ', x.score(train_features, train_labels)) predictions = x.predict(test_features) print('xgboost test score: ', x.score(test_features, test_labels)) ```
Here, train and test score: R2 ~0.94.
2. Random Forest Regressor
Then we try Random Forest model. After some fiddling it appears 100 estimators is enough to get a pretty good accuracy (R2 > 0. 422