Big difference in randomTree accuracy with train and test sets
I've created a model using randomForest for the following dataset: https://archive.ics.uci.edu/ml/datasets/Contraceptive+Method+Choice
The thing i'm questioning is that the results of the model when used on my training and testing sets are vastly different.
r library(randomForest) library(caret) df <- read.csv('cmc.csv')
Changing values to factors ```r df$Wife.s.education <- as.factor(df$Wife.s.education) df$Husband.s.education <- as.factor(df$Husband.s.education) df$Wife.s.religion <- as.factor(df$Wife.s.religion) df$Wife.s.now.working. <- as.factor(df$Wife.s.now.working.) df$Husband.s.occupation <- as.factor(df$Husband.s.occupation) df$Standard.of.living.index <- as.factor(df$Standard.of.living.index) df$Media.exposure <- as.factor(df$Media.exposure)
add string representation for readiblilty
df[df$Contraceptive.method.used == 1,]$Contraceptive.method.used <- "No-use" df[df$Contraceptive.method.used == 2,]$Contraceptive.method.used <- "Long-term" df[df$Contraceptive.method.used == 3,]$Contraceptive.method.used <- "Short-term" df$Contraceptive.method.used <- as.factor(df$Contraceptive.method.used) ```
Splitting data: r set.seed(47) ind <- sample(2, nrow(df), replace = TRUE, prob = c(0.7,0.3)) inTrain <- createDataPartition(y=df$Contraceptive.method.used, p=.7, list = FALSE) training = df[ind==1,] testing = df[ind==2,]
Creating model: model <- randomForest(Contraceptive.method.used ~., data = training, proximity=TRUE)
```r
Prediction & Confusion Matrix - training data
p1 <- predict(model, training) confusionMatrix(p1, training$Contraceptive.method.used) Confusion Matrix Results (training):r Reference Prediction Long-term No-use Short-term Long-term 208 9 18 No-use 4 408 5 Short-term 16 14 338
Overall Statistics
Accuracy : 0.9353
95% CI : (0.9184, 0.9496)
No Information Rate : 0.4225
P-Value [Acc > NIR] : < 2e-16
Kappa : 0.9002
Mcnemar's Test P-Value : 0.09773
```
```r
Prediction & Confusion Matrix - testing data
p2 <- predict(model, testing) confusionMatrix(p2, testing$Contraceptive.method.used) ```
Confusion Matrix Results (testing) ```r Reference Prediction Long-term No-use Short-term Long-term 42 11 23 No-use 27 122 47 Short-term <PHONE_NUMBER>
Overall Statistics
Accuracy : 0.5386
95% CI : (0.4915, 0.5853)
No Information Rate : 0.4371
P-Value [Acc > NIR] : 8.972e-06
Kappa : 0.2788
Mcnemar's Test P-Value : 0.005869
```
As we can see theres is a massive change in the two results, if i print my model I get the following results:
```r Type of random forest: classification Number of trees: 500 No. of variables tried at each split: 3
OOB estimate of error rate: 46.96%
Confusion matrix: Long-term No-use Short-term class.error Long-term 76 63 89 <PHONE_NUMBER> No-use 45 282 104 <PHONE_NUMBER> Short-term 72 106 183 0.<PHONE_NUMBER> ``` This makes me belief that the test results are correct however I'm not sure why there is such a big difference when used on the training set, is this due to overfitting? If so how can this be handled?
Any guidance would be awesome. 613