Answers.Body: <blockquote>
  <p>I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs?</p>
  </blockquote>
  <p>Unfortunately, the question you ask in the main body of your question here (&quot;how do expert AI researchers do things&quot;) often turns out to actually be different from the question in your title (&quot;how should I do things&quot;).</p>
  <p>The very short answer would be that, ideally, you run as many repetitions as you can, but in practice it is indeed very often not feasible to do more than one or a handful.</p>
  <hr />
  <p>For the long answer, I think it actually may be quite different also depending on what kind of machine learning you are doing. Personally, I am much more familiar with Reinforcement Learning and similar kinds of problems, i.e. problems where we're generating the &quot;training&quot; data ourselves by making agents act in environments, and similarly also evaluating by again making trained agents act in environments and measuring their performance. I'm not as familiar with the state-of-the-art research in &quot;standard&quot; machine learning tasks like classification/regression, but probably the problem is much worse in RL-style problems because:</p>
  <ol>
  <li>We actually have to generate our own data, which takes a huge amount of time (sometimes much more time than the actual training itself takes)</li>
  <li>There is often a lot of randomness in how we generate our training data, so we often have very different training datasets in different runs, which can of course also lead to wildly different levels of performance across different runs</li>
  <li>We pretty much always compute gradients from subsets (batches) of data in RL, which is again a source of randomness. In contrast, in supervised ML you could consider estimating gradients from the entire dataset at once per epoch, and then at least that part of your training process becomes deterministic (and hence replicable).</li>
  </ol>
  <p>For the case of RL, recently a paper titled <a href="https://arxiv.org/abs/2108.13264" rel="nofollow noreferrer">&quot;Deep Reinforcement Learning at the Edge of the Statistical Precipice&quot;</a> appeared on arXiv, which gets into various tools and approaches you can use to more reasonably draw principled conclusions even from a small handful of repetitions of your training process (in better ways than the common practice of just reporting a mean, or even worse, just reporting the best result).</p>
  <p>For supervised ML, some similar techniques may be applicable, but the need for this may also be less great (especially if you have little randomness in your training process).</p>
  <hr />
  <blockquote>
  <p>Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I often notice that the result differences are too small to conclude whether set of parameters A is better than B. Sometimes it happens that on a first run, set A seems to work better than B, but on a second run the opposite is suggested.</p>
  </blockquote>
  <p>For this specific situation, I would be interested to know <strong>by how much</strong> A is sometimes better than B, and B is sometimes better than A. If we're talking about like 0.1% differences in accuracy either way at a baseline accuracy of 99.5%, that probably just means they're both equally good. If you were &quot;hoping&quot; for one of them to outperform the other, well then that's probably disappointing... but for the overall training process in general, you could actually draw a more positive conclusion that, apparently, it is <strong>robust</strong>; it performs similarly across different runs even with different parameters!</p>
  <p>On the other hand, if you observe differences in performance of significant magnitudes (like, 80% vs 90% accuracy), but sometimes A better by that much and sometimes B better by that much... then this strongly suggests that you indeed are dealing with high variance and you're going to have to do many repetitions to get statistically meaningful results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-19T10:15:12.427Z
Answers.LastActivityDate: 2021-09-19T10:15:12.427Z
Answers.UserDisplayName: Dennis Soemers
Answers.Body: <p>In addition to <a href="https://ai.stackexchange.com/a/31749/2444">the other good answer</a>, if you can afford the luxury of running your experiments multiple times, you can also use <a href="https://machinelearningmastery.com/hypothesis-test-for-comparing-machine-learning-algorithms/" rel="nofollow noreferrer">hypothesis testing</a> to test whether there is any <em>significant</em> difference between the performance (e.g. accuracy) of the two models.</p>
  <p>Hypothesis testing is not widely used (or, at least, reported in research papers) in the ML/DL community (given that, to apply these tests, you often need to repeat the experiments several times, which can be expensive, as you already know), but there are some researchers (for example, see <a href="https://dl.acm.org/doi/abs/10.1145/3460319.3464825" rel="nofollow noreferrer">this paper</a>) that often use these statistical tools.</p>
  <p>I must also admit that I don't know how valuable they can really be. I've used them in the past, but not enough to say that they can really be useful in the context of machine learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-19T15:12:28.823Z
Answers.LastActivityDate: 2021-09-19T15:12:28.823Z
Answers.UserDisplayName: nbro
Body: <p>I am doing my MSc thesis on deep learning. My model takes many hours to train. Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I often notice that the result differences are too small to conclude whether the set of parameters A is better than B. Sometimes it happens that on a first run, set A seems to work better than B, but on a second run the opposite is suggested.</p>
  <p>The logical approach for that would be to repeat experiments and average out. But it seems unpractical given that each run could take so many hours.</p>
  <p>I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T18:40:41.853Z
Comments.Text: The goal has always been tune it enough to beat SOTA and then publish a paper with the model/weights.
Comments.UserDisplayName: FourierFlux
Tags: <deep-learning><ai-design><research>
Title: Should I repeat lengthy deep learning experiments to average results ? How to decide how many times to repeat?
UserDisplayName: Manveru
ThreadText: # Should I repeat lengthy deep learning experiments to average results ? How to decide how many times to repeat?
  
  (Asked by: username_0 on 2021-09-18)
  
  I am doing my MSc thesis on deep learning. My model takes many hours to train. Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I often notice that the result differences are too small to conclude whether the set of parameters A is better than B. Sometimes it happens that on a first run, set A seems to work better than B, but on a second run the opposite is suggested.
  
  The logical approach for that would be to repeat experiments and average out. But it seems unpractical given that each run could take so many hours.
  
  I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs?
  
  **Tags:** deep-learning, ai-design, research
  
  > **Comments:**
  > **username_1**: The goal has always been tune it enough to beat SOTA and then publish a paper with the model/weights.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-09-19. Score: 3
  
  > I wonder how expert AI researchers deal with that, do they perform multiple experiments, even if this takes extremely long? Do they draw conclusions from single runs?
  
  Unfortunately, the question you ask in the main body of your question here ("how do expert AI researchers do things") often turns out to actually be different from the question in your title ("how should I do things").
  
  The very short answer would be that, ideally, you run as many repetitions as you can, but in practice it is indeed very often not feasible to do more than one or a handful.
  
  ---
  
  For the long answer, I think it actually may be quite different also depending on what kind of machine learning you are doing. Personally, I am much more familiar with Reinforcement Learning and similar kinds of problems, i.e. problems where we're generating the "training" data ourselves by making agents act in environments, and similarly also evaluating by again making trained agents act in environments and measuring their performance. I'm not as familiar with the state-of-the-art research in "standard" machine learning tasks like classification/regression, but probably the problem is much worse in RL-style problems because:
  
  1. We actually have to generate our own data, which takes a huge amount of time (sometimes much more time than the actual training itself takes)
  2. There is often a lot of randomness in how we generate our training data, so we often have very different training datasets in different runs, which can of course also lead to wildly different levels of performance across different runs
  3. We pretty much always compute gradients from subsets (batches) of data in RL, which is again a source of randomness. In contrast, in supervised ML you could consider estimating gradients from the entire dataset at once per epoch, and then at least that part of your training process becomes deterministic (and hence replicable).
  
  For the case of RL, recently a paper titled ["Deep Reinforcement Learning at the Edge of the Statistical Precipice"](https://arxiv.org/abs/2108.13264) appeared on arXiv, which gets into various tools and approaches you can use to more reasonably draw principled conclusions even from a small handful of repetitions of your training process (in better ways than the common practice of just reporting a mean, or even worse, just reporting the best result).
  
  For supervised ML, some similar techniques may be applicable, but the need for this may also be less great (especially if you have little randomness in your training process).
  
  ---
  
  > Part of what I do is trying different parameters and settings hoping that they will achieve different results. But I often notice that the result differences are too small to conclude whether set of parameters A is better than B. Sometimes it happens that on a first run, set A seems to work better than B, but on a second run the opposite is suggested.
  
  For this specific situation, I would be interested to know **by how much** A is sometimes better than B, and B is sometimes better than A. If we're talking about like 0.1% differences in accuracy either way at a baseline accuracy of 99.5%, that probably just means they're both equally good. If you were "hoping" for one of them to outperform the other, well then that's probably disappointing... but for the overall training process in general, you could actually draw a more positive conclusion that, apparently, it is **robust**; it performs similarly across different runs even with different parameters!
  
  On the other hand, if you observe differences in performance of significant magnitudes (like, 80% vs 90% accuracy), but sometimes A better by that much and sometimes B better by that much... then this strongly suggests that you indeed are dealing with high variance and you're going to have to do many repetitions to get statistically meaningful results.
  
  
  ---
  
  ### Answer by username_3 on 2021-09-19. Score: 1
  
  In addition to [the other good answer](https://ai.stackexchange.com/a/31749/2444), if you can afford the luxury of running your experiments multiple times, you can also use [hypothesis testing](https://machinelearningmastery.com/hypothesis-test-for-comparing-machine-learning-algorithms/) to test whether there is any *significant* difference between the performance (e.g. accuracy) of the two models.
  
  Hypothesis testing is not widely used (or, at least, reported in research papers) in the ML/DL community (given that, to apply these tests, you often need to repeat the experiments several times, which can be expensive, as you already know), but there are some researchers (for example, see [this paper](https://dl.acm.org/doi/abs/10.1145/3460319.3464825)) that often use these statistical tools.
  
  I must also admit that I don't know how valuable they can really be. I've used them in the past, but not enough to say that they can really be useful in the context of machine learning.
  
  
  ---
  
  