Answers.Body: <p>Concerning <em><span class="math-container">$k$</span>-fold Cross Validation</em>, I like to think of it by considering two extremes you can do: <em>Leave-One-Out Cross-Validation</em> where you leave one sample each time and train your model on the remaining <span class="math-container">$n-1$</span>, and <em>2-fold Cross Validation</em> at which you split your dataset in half and train (and validate) two models on two different halves.</p>
  
  <p>The important aspect when choosing <span class="math-container">$k$</span> is a <em>bias-variance tradeoff</em>. Note that in <em>LOOCV</em> you train each model using almost as many samples as there are available (<span class="math-container">$n-1$</span>), so the validation step should give you an <em>unbiased estimate</em> of the real test error. However, each model in <em>LOOCV</em> is trained each time on <strong>almost exactly same dataset!</strong>. This has important consequences, since the output of each model is highly correlated with each other. Since the mean of highly correlated variables has big variance, <em>LOOCV</em> will suffer from huge variance.</p>
  
  <p>On the other hand, in <em>2-fold CV</em> the models share no common samples, so they are not correlated and therefore their outputs have low variance. But since we train each model using only a half of available samples, the procedure will have a high bias (the estimated test error will be off from the true test error).</p>
  
  <p>What to do in this scenario? Choose something in the middle. Usually <span class="math-container">$k=5$</span> and <span class="math-container">$k=10$</span> should be a good choice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-20T07:21:33.710Z
Answers.Text: I would love to do 5 or 10 fold cross validation, but since the dataset only has 20 samples this is simply not feasible. The set will have only 4 or 2 samples respectively, which will yield horrible ROC AUC.
Answers.UserDisplayName: Diogo Bastos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-18T10:34:12.223Z
Answers.LastActivityDate: 2019-05-18T10:34:12.223Z
Answers.UserDisplayName: Tomasz Bartkowiak
Body: <p>I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.</p>
  <p>Here's what I was thinking:</p>
  <ol>
  <li><p>train/test split the data, with only one data point in the test set.</p>
  </li>
  <li><p>train the model on training data, potentially with grid-search/cross-validation</p>
  </li>
  <li><p>use the best model from step 2 to make a prediction on the one data point and save the prediction in an array</p>
  </li>
  <li><p>repeat the previous steps until all the data points have been in the test set</p>
  </li>
  <li><p>calculate your preferred metric of choice (accuracy, f1-score, auc, etc) using these predictions</p>
  </li>
  </ol>
  <p>The pros of this approach would be to:</p>
  <ul>
  <li>You don't have to split the data into train/test so you can train
  with more data points.</li>
  </ul>
  <p>The cons would be:</p>
  <ul>
  <li><p>This approach suffers from potential(?) data leakage.</p>
  </li>
  <li><p>You are calculating an accuracy metric from a bunch of predictions that potentially came from different models, due to the grid searches, so I'm not sure how accurate it is going to be.</p>
  </li>
  </ul>
  <p>I have tried the standard approaches of train/test splitting, but since I need to take out at least 5 points for testing, then I don't have enough points for training and the ROC AUC becomes very bad.</p>
  <p>I would really appreciate some feedback about whether this approach is actually feasible or not and why.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-11T10:49:10.703Z
Comments.Text: This is an old question, but it's not clear to me what your doubt really is. LOOCV is typically used for testing and/or validation, so what is your real question here? By the way, I edited the post to put in the title what I think is your question, but, to be honest, I don't really know what it is.
Comments.UserDisplayName: nbro
Tags: <training><cross-validation><testing><loocv>
Title: Should I use leave-one-out cross-validation for testing?
UserDisplayName: Diogo Bastos
ThreadText: # Should I use leave-one-out cross-validation for testing?
  
  (Asked by: username_0 on 2019-05-17)
  
  I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.
  
  Here's what I was thinking:
  
  1. train/test split the data, with only one data point in the test set.
  2. train the model on training data, potentially with grid-search/cross-validation
  3. use the best model from step 2 to make a prediction on the one data point and save the prediction in an array
  4. repeat the previous steps until all the data points have been in the test set
  5. calculate your preferred metric of choice (accuracy, f1-score, auc, etc) using these predictions
  
  The pros of this approach would be to:
  
  * You don't have to split the data into train/test so you can train
    with more data points.
  
  The cons would be:
  
  * This approach suffers from potential(?) data leakage.
  * You are calculating an accuracy metric from a bunch of predictions that potentially came from different models, due to the grid searches, so I'm not sure how accurate it is going to be.
  
  I have tried the standard approaches of train/test splitting, but since I need to take out at least 5 points for testing, then I don't have enough points for training and the ROC AUC becomes very bad.
  
  I would really appreciate some feedback about whether this approach is actually feasible or not and why.
  
  **Tags:** training, cross-validation, testing, loocv
  > **Comments:**
  > **username_1**: This is an old question, but it's not clear to me what your doubt really is. LOOCV is typically used for testing and/or validation, so what is your real question here? By the way, I edited the post to put in the title what I think is your question, but, to be honest, I don't really know what it is.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-05-18. Score: 2
  
  Concerning *$k$-fold Cross Validation*, I like to think of it by considering two extremes you can do: *Leave-One-Out Cross-Validation* where you leave one sample each time and train your model on the remaining $n-1$, and *2-fold Cross Validation* at which you split your dataset in half and train (and validate) two models on two different halves.
  
  The important aspect when choosing $k$ is a *bias-variance tradeoff*. Note that in *LOOCV* you train each model using almost as many samples as there are available ($n-1$), so the validation step should give you an *unbiased estimate* of the real test error. However, each model in *LOOCV* is trained each time on **almost exactly same dataset!**. This has important consequences, since the output of each model is highly correlated with each other. Since the mean of highly correlated variables has big variance, *LOOCV* will suffer from huge variance.
  
  On the other hand, in *2-fold CV* the models share no common samples, so they are not correlated and therefore their outputs have low variance. But since we train each model using only a half of available samples, the procedure will have a high bias (the estimated test error will be off from the true test error).
  
  What to do in this scenario? Choose something in the middle. Usually $k=5$ and $k=10$ should be a good choice.
  
  > **Comments:**
  > **username_0**: I would love to do 5 or 10 fold cross validation, but since the dataset only has 20 samples this is simply not feasible. The set will have only 4 or 2 samples respectively, which will yield horrible ROC AUC.  
  
  ---
  
  