Answers.Body: <p>See here for a potential way to do it:</p>
  
  <p><a href="http://infinity77.net/global_optimization/#motivation-motivation" rel="nofollow noreferrer">http://infinity77.net/global_optimization/#motivation-motivation</a></p>
  
  <p><a href="http://infinity77.net/global_optimization/#rules-the-rules" rel="nofollow noreferrer">http://infinity77.net/global_optimization/#rules-the-rules</a></p>
  
  <p>You basically test the two (or more) optimization algorithms against known objective functions, with several random (but repeatable) starting points and then analyze the outcome.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-23T01:32:40.653Z
Answers.Text: Thanks!So, do you mean we choose several random seed(initial value) and run once for each of them? And then do we choose the best performance among of these runs or do we choose just the average performance, and then use it to compare different optimization algorithms?
Answers.UserDisplayName: abc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-23T06:17:06.207Z
Answers.Text: Normally you would use the average of the results. In my tests I have run multiple optimization algorithms with 100 random starting points (but the same for all algorithms) and then created a few graphics to compare performances: http://infinity77.net/global_optimization/multidimensional.html . Please don’t be caught into believing papers or website using CPU time or elapsed time as a performance measure, those are useless metrics.
Answers.UserDisplayName: Infinity77
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-22T19:53:55.717Z
Answers.LastActivityDate: 2019-09-22T19:53:55.717Z
Answers.UserDisplayName: Infinity77
Body: <p>When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 score .etc, and do the same for an old optimization algorithm, or do we need to compute the average performance,i.e.,the average value of accuracy or f1 scores for these runs, to show that it is better than the old optimization algorithm? Because when I read the <strong>papers on a new optimization algorithm</strong>, I don't know how they calculate the performance and draw the train-loss vs iters curves, because it has random effects, and for different runs we may get different performance and different curves. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-05T16:21:50.317Z
Comments.Text: Hi and welcome to AI SE. Could you please link any paper that you talk about in your question?
Comments.UserDisplayName: naive
Tags: <optimization><convergence><performance><hyperparameter-optimization>
Title: How can we conclude that an optimization algorithm is better than another one
UserDisplayName: user29902
ThreadText: # How can we conclude that an optimization algorithm is better than another one
  
  (Asked by: username_0 on 2019-09-22)
  
  When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 score .etc, and do the same for an old optimization algorithm, or do we need to compute the average performance,i.e.,the average value of accuracy or f1 scores for these runs, to show that it is better than the old optimization algorithm? Because when I read the **papers on a new optimization algorithm**, I don't know how they calculate the performance and draw the train-loss vs iters curves, because it has random effects, and for different runs we may get different performance and different curves.
  
  **Tags:** optimization, convergence, performance, hyperparameter-optimization
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE. Could you please link any paper that you talk about in your question?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-22. Score: 1
  
  See here for a potential way to do it:
  
  <http://infinity77.net/global_optimization/#motivation-motivation>
  
  <http://infinity77.net/global_optimization/#rules-the-rules>
  
  You basically test the two (or more) optimization algorithms against known objective functions, with several random (but repeatable) starting points and then analyze the outcome.
  
  > **Comments:**
  > **username_3**: Thanks!So, do you mean we choose several random seed(initial value) and run once for each of them? And then do we choose the best performance among of these runs or do we choose just the average performance, and then use it to compare different optimization algorithms?  
  > **username_2**: Normally you would use the average of the results. In my tests I have run multiple optimization algorithms with 100 random starting points (but the same for all algorithms) and then created a few graphics to compare performances: http://infinity77.net/global_optimization/multidimensional.html . Please don’t be caught into believing papers or website using CPU time or elapsed time as a performance measure, those are useless metrics.  
  
  ---
  
  