Answers.Body: <p>How to find the best configuration for an algorithm is an open research question in AI. The topic in general is known as `hyper-parameter optimization' and there are a range of possible methods:</p>
  
  <p>One of the most popular is <a href="http://iridia.ulb.ac.be/irace/" rel="nofollow noreferrer">IRace</a>, but other possibilities include:</p>
  
  <ul>
  <li><p><a href="https://github.com/JasperSnoek/spearmint" rel="nofollow noreferrer">Spearmint</a>: uses wrappers in Matlab or Python. It uses MongoDb,
  and Bayesian optimisation algorithms.</p></li>
  <li><p><a href="http://www.cs.ubc.ca/labs/beta/Projects/SMAC/" rel="nofollow noreferrer">SMAC</a> requires a python wrapper for the algorithm to be optimized
  and has a command line interface.</p></li>
  <li><p><a href="https://github.com/hyperopt/hyperopt" rel="nofollow noreferrer">Hyperopt</a>: a Python library which uses Random Search and Tree of
  Parzen Estimators.</p></li>
  </ul>
  
  <p><a href="http://www.automl.org/papers/13-BayesOpt%5C_EmpiricalFoundation.pdf" rel="nofollow noreferrer">This paper</a> argues that Spearmint performs the best, compared with SMAC and Hyperopt, but with significantly longer running times in some cases.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-10T22:24:46.453Z
Answers.LastActivityDate: 2016-11-10T22:24:46.453Z
Answers.UserDisplayName: NietzscheanAI
Body: <p>Is there any methodology to find proper parameter settings for a given meta-heuristic algorithm, e.g. the <a href="https://en.wikipedia.org/wiki/Firefly_algorithm" rel="nofollow noreferrer">firefly algorithm</a> or the <a href="https://en.wikipedia.org/wiki/Cuckoo_search" rel="nofollow noreferrer">cuckoo search</a>? Is this an open issue in optimization? Is extensive experimentation, measurements and intuition the only way to figure out which are the best settings? </p>
  
Tags: <search><heuristics><hyperparameter-optimization><meta-heuristics>
Title: How to find proper parameter settings for a given optimization algorithm?
UserDisplayName: Jairo
ThreadText: # How to find proper parameter settings for a given optimization algorithm?
  
  (Asked by: username_0 on 2016-11-10)
  
  Is there any methodology to find proper parameter settings for a given meta-heuristic algorithm, e.g. the [firefly algorithm](https://en.wikipedia.org/wiki/Firefly_algorithm) or the [cuckoo search](https://en.wikipedia.org/wiki/Cuckoo_search)? Is this an open issue in optimization? Is extensive experimentation, measurements and intuition the only way to figure out which are the best settings?
  
  **Tags:** search, heuristics, hyperparameter-optimization, meta-heuristics
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2016-11-10. Score: 2
  
  How to find the best configuration for an algorithm is an open research question in AI. The topic in general is known as `hyper-parameter optimization' and there are a range of possible methods:
  
  One of the most popular is [IRace](http://iridia.ulb.ac.be/irace/), but other possibilities include:
  
  * [Spearmint](https://github.com/JasperSnoek/spearmint): uses wrappers in Matlab or Python. It uses MongoDb,
    and Bayesian optimisation algorithms.
  * [SMAC](http://www.cs.ubc.ca/labs/beta/Projects/SMAC/) requires a python wrapper for the algorithm to be optimized
    and has a command line interface.
  * [Hyperopt](https://github.com/hyperopt/hyperopt): a Python library which uses Random Search and Tree of
    Parzen Estimators.
  
  [This paper](http://www.automl.org/papers/13-BayesOpt%5C_EmpiricalFoundation.pdf) argues that Spearmint performs the best, compared with SMAC and Hyperopt, but with significantly longer running times in some cases.
  
  
  ---
  
  