Answers.Body: <blockquote>
  <p>It says here that we can use <span class="math-container">$\hat{x} \pm 1.96 \frac{\hat{\sigma}}{\sqrt{n}}$</span>, but I don't quite understand the theory behind.</p>
  </blockquote>
  <p>Following the Gaussian distribution, <span class="math-container">$1.96$</span> is an approximate value by which we multiply the sample standard deviation <span class="math-container">$\hat{\sigma}$</span> to get the <span class="math-container">$95\%$</span> confidence interval for unknown <span class="math-container">$x$$-$</span>i.e., <span class="math-container">$95\%$</span> of multiple intervals <span class="math-container">$[\hat{x} - 1.96\frac{\hat{\sigma}}{\sqrt{n}}, \hat{x}+1.96\frac{\hat{\sigma}}{\sqrt{n}}]$</span> constructed on the basis of different experiments and their corresponding test-score lists will contain the true value of test score <span class="math-container">$x$</span>.</p>
  <p>I guess this makes sense for <span class="math-container">$k \geq 10$</span> cross-validation, although this issue baffles me too, and from my experience, practitioners either report <span class="math-container">$\text{mean}(x) \pm \text{std}(x)$</span> or just leave the details out.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T19:34:17.053Z
Answers.Text: You don't need k-fold cv. If you are using a test set, then report the mean error and standard error based on the size of the test set.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-20T17:57:20.817Z
Answers.Text: @NeilSlater What if we calculate accuracy, AUROC, etc.?
Answers.UserDisplayName: Sanjar Adilov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-20T20:05:47.000Z
Answers.Text: Accuracy is just fine, you can treat it as a binary variable per example and calculate mean/sd from a single test set (in fact the accuracy measure is already the mean of this variable). With AUROC yes you would need to do differently, but I am not sure it is a done thing to report AUROC with bounds. The problem of quoting cv-based results is most often k-fold cv is used for model selection, resulting in biased measurements.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-21T08:13:03.153Z
Answers.Text: @NeilSlater Sorry, I'm not sure I get your point. There are benchmarks with no explicit train-test splits, so researches usually propose their own cv strategies. Of course, they should not search for best models based on test errors; they should split train sets into train-valid sets, explore test sets only after best models acc. to val. sets are obtained, and immediately finalize the results. Pseudocode will be `cv_errors = cross_val_scores(GridSearchCV(MyModel()), X, y)`. In this case, how should we present the distribution of `cv_errors`? In my answer, I assume that 95% CI makes sense.
Answers.UserDisplayName: Sanjar Adilov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-21T10:21:42.963Z
Answers.Text: My point is that it would be more usual to to use a hold-out test set for reporting mean and confidence interval on results, and that this works. Your last paragraph declares that the confidence bounds on metrics may only make sense for k-fold cv. But for metrics such as MSE and accuracy that is not the case, they will have confidence bounds based on the size of a single test set.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-21T10:55:33.493Z
Answers.Text: @NeilSlater I do not say CIs make sense *only* for k-fold cv. OP asked about summarizing repeated experiments - i.e., $k > 1$-times validation/testing (not necessarily $k$-*fold*, btw), and I wanted to emphasize that there might be a rule of thumb for minimum $k$. And I think your solution with one hold-out test set also makes sense.
Answers.UserDisplayName: Sanjar Adilov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T18:41:21.730Z
Answers.LastActivityDate: 2022-05-19T19:08:03.910Z
Answers.LastEditDate: 2022-05-19T19:08:03.910Z
Answers.UserDisplayName: Sanjar Adilov
Body: <p>It has been asked <a href="https://ai.stackexchange.com/q/31737/2444">here</a> if we should repeat lengthy experiments.</p>
  <p>Let's say I can repeat them, how should I present them? For instance, if I am measuring the accuracy of a model on test data during some training epochs, and I repeat various times this training, I will have different values of test accuracy. I can average them to take into account all the experiments. Can I then calculate a sort of confidence interval to say that the accuracy will most likely be within an interval? Does this make sense? If it does, what formula should I use?</p>
  <p>It says <a href="https://machinelearningmastery.com/estimate-number-experiment-repeats-stochastic-machine-learning-algorithms/" rel="nofollow noreferrer">here</a> that we can use
  <span class="math-container">$\hat{x} \pm 1.96 \frac{\hat{\sigma}}{\sqrt{n}}$</span>, but I don't quite understand the theory behind.</p>
  
Tags: <machine-learning><statistics>
Title: If I can repeat ML experiments, how can I bound my results?
UserDisplayName: biofa.esp
ThreadText: # If I can repeat ML experiments, how can I bound my results?
  
  (Asked by: username_0 on 2022-05-15)
  
  It has been asked [here](https://ai.stackexchange.com/q/31737/2444) if we should repeat lengthy experiments.
  
  Let's say I can repeat them, how should I present them? For instance, if I am measuring the accuracy of a model on test data during some training epochs, and I repeat various times this training, I will have different values of test accuracy. I can average them to take into account all the experiments. Can I then calculate a sort of confidence interval to say that the accuracy will most likely be within an interval? Does this make sense? If it does, what formula should I use?
  
  It says [here](https://machinelearningmastery.com/estimate-number-experiment-repeats-stochastic-machine-learning-algorithms/) that we can use
  $\hat{x} \pm 1.96 \frac{\hat{\sigma}}{\sqrt{n}}$, but I don't quite understand the theory behind.
  
  **Tags:** machine-learning, statistics
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-19. Score: 0
  
  > It says here that we can use $\hat{x} \pm 1.96 \frac{\hat{\sigma}}{\sqrt{n}}$, but I don't quite understand the theory behind.
  
  Following the Gaussian distribution, $1.96$ is an approximate value by which we multiply the sample standard deviation $\hat{\sigma}$ to get the $95\%$ confidence interval for unknown $x$$-$i.e., $95\%$ of multiple intervals $[\hat{x} - 1.96\frac{\hat{\sigma}}{\sqrt{n}}, \hat{x}+1.96\frac{\hat{\sigma}}{\sqrt{n}}]$ constructed on the basis of different experiments and their corresponding test-score lists will contain the true value of test score $x$.
  
  I guess this makes sense for $k \geq 10$ cross-validation, although this issue baffles me too, and from my experience, practitioners either report $\text{mean}(x) \pm \text{std}(x)$ or just leave the details out.
  
  > **Comments:**
  > **username_2**: You don't need k-fold cv. If you are using a test set, then report the mean error and standard error based on the size of the test set.  
  > **username_1**: @NeilSlater What if we calculate accuracy, AUROC, etc.?  
  > **username_2**: Accuracy is just fine, you can treat it as a binary variable per example and calculate mean/sd from a single test set (in fact the accuracy measure is already the mean of this variable). With AUROC yes you would need to do differently, but I am not sure it is a done thing to report AUROC with bounds. The problem of quoting cv-based results is most often k-fold cv is used for model selection, resulting in biased measurements.  
  > **username_1**: @NeilSlater Sorry, I'm not sure I get your point. There are benchmarks with no explicit train-test splits, so researches usually propose their own cv strategies. Of course, they should not search for best models based on test errors; they should split train sets into train-valid sets, explore test sets only after best models acc. to val. sets are obtained, and immediately finalize the results. Pseudocode will be `cv_errors = cross_val_scores(GridSearchCV(MyModel()), X, y)`. In this case, how should we present the distribution of `cv_errors`? In my answer, I assume that 95% CI makes sense.  
  > **username_2**: My point is that it would be more usual to to use a hold-out test set for reporting mean and confidence interval on results, and that this works. Your last paragraph declares that the confidence bounds on metrics may only make sense for k-fold cv. But for metrics such as MSE and accuracy that is not the case, they will have confidence bounds based on the size of a single test set.  
  > **username_1**: @NeilSlater I do not say CIs make sense *only* for k-fold cv. OP asked about summarizing repeated experiments - i.e., $k > 1$-times validation/testing (not necessarily $k$-*fold*, btw), and I wanted to emphasize that there might be a rule of thumb for minimum $k$. And I think your solution with one hold-out test set also makes sense.  
  
  ---
  
  