Hold-Out VS Cross-Validation - R caret I have a question regarding hold-out vs. cross-validation. I have a dataset with ~650 cases which I am analyzing in R using the caret package. There I have a regression problem and a classification problem.
First I created a training and test dataset with an 80:20 split and then I used cross-validation (10-fold, 3 repeats) on the training dataset to fit the final model. Then I tested the model on the test data set.
The question now is,

*

*what do I specify as RMSE and AUC ROC?

*The results of the fitted model on the training data set or on the test data set?

I have the feeling that the results in the test data set are very strongly dependent on the coincidence of the split in the training and test data set.

*

*Also, would it be okay to have only one training dataset?

*If so, how would one plot an AUC ROC and a bland-altman?

I'm a little confused that even though you have cross-validation, you should still have an additional independent test dataset. In principle one has then so to speak a training, validation and test dataset.

*

*Wouldn't a pure training and validation dataset be possible?


Thanks for the responses so far! That has helped me a lot. If I take a nested CV for the training and then evaluate it on a hold-out dataset, the AUC ROC still fluctuates massively!
I have shown this once here:
           [,1]
 [1,] 0.8489011
 [2,] 0.8401598
 [3,] 0.7405095
 [4,] 0.8031968
 [5,] 0.7604895
 [6,] 0.8653846
 [7,] 0.8231768
 [8,] 0.8551449
 [9,] 0.8146853
[10,] 0.8381618

Thanks for your further help!
Code:
ctrl <- trainControl(method="cv", number=10, repeats=5, 
                                 classProbs = TRUE, summaryFunction = prSummary, search="random")
v <- c()
for (i in seq(1,10)) {
cfPartition <- createDataPartition(
  y = dsCf$VALUE,
  p = .80,
  list = FALSE
)
dsTrainCf <- dsCf[ cfPartition,]
dsTestCf  <- dsCf[-cfPartition,]
mGLMCf <- train($VALUE,~., data=dsTrainCf, method="glm", family = "binomial", 
                trControl=ctrl,  
                metric="AUC", preProcess = c("center","scale"))
rs <- data.frame(obs = dsTestCf$$VALUE,
                 pred = predict(mGLMCf,newdata=dsTestCf,type="raw"),
                 prob = predict(mGLMCf,newdata=dsTestCf,type="prob"))
roc.GLMCf <- roc(rs$obs,rs$prob.prcbYES)
v  <- rbind(v,as.numeric(roc.GLMCf$auc))
}
print(v)
mean(v)
sd(v)

