title: How to calculate 95% CI for accuracy and kappa in caret
body: <p>I am running k-fold repeated training with the caret package and would like to calculate the confidence interval for my accuracy metrics. This tutorial prints a caret training object that shows accuracy/kappa metrics and associated SD: <a href="https://machinelearningmastery.com/tune-machine-learning-algorithms-in-r/" rel="nofollow noreferrer">https://machinelearningmastery.com/tune-machine-learning-algorithms-in-r/</a>. However, when I do this, all that is listed are the metric average values.</p>
  <pre><code>control &lt;- trainControl(method=&quot;repeatedcv&quot;, number=10, repeats=3, search=&quot;grid&quot;)
  set.seed(12345)
  tunegrid &lt;- expand.grid(.mtry=4)
  rf_gridsearch &lt;- train(as.factor(gear)~., data=mtcars, method=&quot;rf&quot;, 
                         metric=&quot;Accuracy&quot;, 
                         tuneGrid=tunegrid, 
                         trControl=control)
  print(rf_gridsearch)
  </code></pre>
  <pre><code>&gt; print(rf_gridsearch)
  Random Forest 
  
  32 samples
  10 predictors
   3 classes: '3', '4', '5' 
  
  No pre-processing
  Resampling: Cross-Validated (10 fold, repeated 3 times) 
  Summary of sample sizes: 29, 28, 30, 29, 27, 28, ... 
  Resampling results:
  
    Accuracy   Kappa    
    0.8311111  0.7021759
  
  Tuning parameter 'mtry' was held constant at a value of 4
  </code></pre>