Answers.Body: <p>A surrogate model is a simplified model. It is a mapping <span class="math-container">$y_S=f_S(x)$</span> that approximates the original model <span class="math-container">$y=f(x)$</span>, in a given domain, reasonably well. Source: <a href="https://onlinelibrary.wiley.com/doi/book/10.1002/9780470770801" rel="nofollow noreferrer">Engineering Design via Surrogate Modelling: A Practical Guide</a></p>
  
  <p>In the context of Bayesian optimization, one wants to optimize a function <span class="math-container">$y=f(x)$</span> which is expensive (very time consuming) to evaluate, therefore one optimizes the surrogate model <span class="math-container">$y_S=f_S(x)$</span> which is cheaper (faster) to evaluate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-08T20:07:38.483Z
Answers.Text: Forgive my ignorance, but why exactly is yS=fS(x) faster to evaluate?
Answers.UserDisplayName: Goose
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T21:10:39.043Z
Answers.Text: Imagine that the original model is computed from Finite Element simulations (x would be some geometric parameter or material constant for instance and f(x) some quantity of interest) and f_S is a polynomial approximation like a0 + a1x + a2x^2. f(x) can take some hours to evaluate whereas f_S(x) can be calculated pretty fast
Answers.UserDisplayName: Javier-Acuna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-16T16:53:21.590Z
Answers.LastActivityDate: 2019-11-19T16:49:58.793Z
Answers.LastEditDate: 2019-11-19T16:49:58.793Z
Answers.UserDisplayName: Javier-Acuna
Answers.Body: <p>Recently, I've been thinking this question as well. After reading several papers, finally came up with some thoughts about the surrogate model.
  In FEM(finite element method), we try to find a weak form to approximate the strong form so that we can solve the weak form analytically. (weak form: approximation equation; strong form: PDE in real world) In my opinion, the surrogate model can be regarded as 'weak form'. There are many methods can form a surrogate model. And if we use a NN model as the surrogate model, the training process is equivalent to 'solving analytically'.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T03:08:16.977Z
Answers.LastActivityDate: 2021-04-15T03:08:16.977Z
Answers.UserDisplayName: T.C. Liu
Answers.Body: <h2>What is <a href="http://krasserm.github.io/2018/03/21/bayesian-optimization/" rel="nofollow noreferrer">Bayesian optimization</a>?</h2>
  
  <h3>Introduction</h3>
  
  <p><a href="https://en.wikipedia.org/wiki/Bayesian_optimization" rel="nofollow noreferrer">Bayesian optimization (BO)</a> is an optimization technique used to model an <strong>unknown</strong> (<a href="https://arxiv.org/pdf/1807.02811.pdf" rel="nofollow noreferrer">usually continuous</a>) function <span class="math-container">$f: \mathbb{R}^d \rightarrow Y$</span>, where <a href="https://arxiv.org/pdf/1807.02811.pdf" rel="nofollow noreferrer">typically <span class="math-container">$d \leq 20$</span></a>, so it can be used to solve regression and classification problems, where you want to find an approximation of <span class="math-container">$f$</span>. In this sense, BO is similar to the usual approach of training a neural network with gradient descent combined with the back-propagation algorithm, so that to optimize an objective function. However, BO is particularly suited for regression or classification problems where <strong>the unknown function <span class="math-container">$f$</span> is expensive to evaluate</strong> (that is, given the input <span class="math-container">$\mathbf{x} \in \mathbb{R}^d$</span>, the computation of <span class="math-container">$f(x) \in Y$</span> takes a lot of time or, in general, resources). For example, when doing hyper-parameter tuning, we usually need to first train the model with the new hyper-parameters before evaluating the specific configuration of hyper-parameters, but this usually takes a lot of time (hours, days or even months), especially when you are training deep neural networks with big datasets. Moreover, BO does not involve the computation of gradients and <a href="https://arxiv.org/pdf/1807.02811.pdf" rel="nofollow noreferrer">it usually assumes that <span class="math-container">$f$</span> lacks properties such as concavity or linearity</a>.</p>
  
  <h3>How does Bayesian optimization work?</h3>
  
  <p>There are three main concepts in BO</p>
  
  <ul>
  <li>the <strong>surrogate model</strong>, which models an unknown function, </li>
  <li>a method for <strong>statistical inference</strong>, which is used to update the surrogate model, and</li>
  <li>the <strong>acquisition function</strong>, which is used to guide the statistical inference and thus it is used to update the surrogate model</li>
  </ul>
  
  <p>The surrogate model is usually a <a href="https://distill.pub/2019/visual-exploration-gaussian-processes" rel="nofollow noreferrer">Gaussian process</a>, which is just a fancy name to denote a <a href="https://en.wikipedia.org/wiki/Gaussian_process" rel="nofollow noreferrer">collection of random variables such that the joint distribution of those random variables is a multivariate <em>Gaussian probability distribution</em></a> (hence the name <em>Gaussian</em> process). Therefore, in BO, we often use a Gaussian probability distribution (the surrogate model) to model the possible functions that are consistent with the data. In other words, given that we do not know <span class="math-container">$f$</span>, rather than finding the usual <em>point estimate</em> (or maximum likelihood estimate), like in the usual case of supervised learning mentioned above, we maintain a Gaussian probability distribution that describes our uncertainty about the unknown <span class="math-container">$f$</span>.</p>
  
  <p>The method of statistical inference is often just an iterative application of the Bayes rule (hence the name <em>Bayesian</em> optimization), where you want to find the posterior, given a prior, a likelihood and the evidence. In BO, you usually place a prior on <span class="math-container">$f$</span>, which is a multivariate Gaussian distribution, then you use the Bayes rule to find the posterior distribution of <span class="math-container">$f$</span> given the data.</p>
  
  <p>What is the data in this case? In BO, the data are the outputs of <span class="math-container">$f$</span> evaluated at certain points of the domain of <span class="math-container">$f$</span>. The acquisition function is used to choose these points of the domain of <span class="math-container">$f$</span>, based on the computed posterior distribution. In other words, based on the current uncertainty about <span class="math-container">$f$</span> (the posterior), the acquisition function attempts to cleverly choose points of the domain of <span class="math-container">$f$</span>, <span class="math-container">$\mathbf{x} \in \mathbb{R}^d$</span>, which will be used to find an updated posterior.  Why do we need the acquisition function? Why can't we simply evaluate <span class="math-container">$f$</span> at random domain points? Given that <span class="math-container">$f$</span> is expensive to evaluate, we need a clever way to choose the points where we want to evaluate <span class="math-container">$f$</span>. More specifically, we want to evaluate <span class="math-container">$f$</span> where we are more uncertain about it.</p>
  
  <p>There are several acquisition functions, such as <a href="https://arxiv.org/pdf/1807.02811.pdf" rel="nofollow noreferrer"><em>expected improvement, knowledge-gradient, entropy search</em>, and <em>predictive entropy search</em></a>, so there are different ways of choosing the points of the domain of <span class="math-container">$f$</span> where we want to evaluate it to update the posterior, each of which deals with the exploration-exploitation dilemma differently.</p>
  
  <h3>What can Bayesian optimization be used for?</h3>
  
  <p>BO can be used for tuning hyper-parameters (also called hyper-parameter optimisation) of machine learning models, such as neural networks, but <a href="https://arxiv.org/pdf/1807.02811.pdf" rel="nofollow noreferrer">it has also been used to solve other problems</a>. </p>
  
  <h2>What is an alternative surrogate model?</h2>
  
  <p>In the book <a href="https://link.springer.com/content/pdf/10.1007%2F978-3-030-05318-5.pdf" rel="nofollow noreferrer">Automated Machine Learning: Methods, Systems, Challenges</a> (by Frank Hutter et al.) that you are quoting, the authors say that the commonly used surrogate model <em>Gaussian process</em> scales cubically in the number of data points, so sparse Gaussian processes are often used. Moreover, Gaussian processes also scale badly with the number of dimensions. In <a href="https://link.springer.com/content/pdf/10.1007%2F978-3-030-05318-5.pdf" rel="nofollow noreferrer">section 1.3.2.2.</a>, the authors describe some <a href="https://www.merriam-webster.com/dictionary/alternative" rel="nofollow noreferrer"><strong>alternative</strong></a> surrogate models to the Gaussian processes, for example, alternatives that use neural networks or random forests.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-17T18:40:01.317Z
Answers.LastActivityDate: 2019-11-17T19:56:47.560Z
Answers.LastEditDate: 2019-11-17T19:56:47.560Z
Answers.UserDisplayName: nbro
Body: <p>In the following paragraph from the book <a href="https://link.springer.com/book/10.1007%2F978-3-030-05318-5" rel="nofollow noreferrer">Automated Machine Learning: Methods, Systems, Challenges</a> (by Frank Hutter et al.)</p>
  
  <blockquote>
    <p>In this section we first give a brief introduction to Bayesian optimization, present alternative surrogate models used in it, describe extensions to conditional and constrained configuration spaces, and then discuss several important applications to hyperparameter optimization.</p>
  </blockquote>
  
  <p>What is an "alternative surrogate model"? What exactly does "alternative" mean?</p>
  
Tags: <terminology><definitions><hyperparameter-optimization><bayesian-optimization>
Title: What is a "surrogate model"?
UserDisplayName: yousef yegane
ThreadText: # What is a "surrogate model"?
  
  (Asked by: username_0 on 2019-11-16)
  
  In the following paragraph from the book [Automated Machine Learning: Methods, Systems, Challenges](https://link.springer.com/book/10.1007%2F978-3-030-05318-5) (by Frank Hutter et al.)
  
  > In this section we first give a brief introduction to Bayesian optimization, present alternative surrogate models used in it, describe extensions to conditional and constrained configuration spaces, and then discuss several important applications to hyperparameter optimization.
  
  What is an "alternative surrogate model"? What exactly does "alternative" mean?
  
  **Tags:** terminology, definitions, hyperparameter-optimization, bayesian-optimization
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2019-11-17. Score: 5 | Accepted answer
  
  ## What is [Bayesian optimization](http://krasserm.github.io/2018/03/21/bayesian-optimization/)?
  
  ### Introduction
  
  [Bayesian optimization (BO)](https://en.wikipedia.org/wiki/Bayesian_optimization) is an optimization technique used to model an **unknown** ([usually continuous](https://arxiv.org/pdf/1807.02811.pdf)) function $f: \mathbb{R}^d \rightarrow Y$, where [typically $d \leq 20$](https://arxiv.org/pdf/1807.02811.pdf), so it can be used to solve regression and classification problems, where you want to find an approximation of $f$. In this sense, BO is similar to the usual approach of training a neural network with gradient descent combined with the back-propagation algorithm, so that to optimize an objective function. However, BO is particularly suited for regression or classification problems where **the unknown function $f$ is expensive to evaluate** (that is, given the input $\mathbf{x} \in \mathbb{R}^d$, the computation of $f(x) \in Y$ takes a lot of time or, in general, resources). For example, when doing hyper-parameter tuning, we usually need to first train the model with the new hyper-parameters before evaluating the specific configuration of hyper-parameters, but this usually takes a lot of time (hours, days or even months), especially when you are training deep neural networks with big datasets. Moreover, BO does not involve the computation of gradients and [it usually assumes that $f$ lacks properties such as concavity or linearity](https://arxiv.org/pdf/1807.02811.pdf).
  
  ### How does Bayesian optimization work?
  
  There are three main concepts in BO
  
  * the **surrogate model**, which models an unknown function,
  * a method for **statistical inference**, which is used to update the surrogate model, and
  * the **acquisition function**, which is used to guide the statistical inference and thus it is used to update the surrogate model
  
  The surrogate model is usually a [Gaussian process](https://distill.pub/2019/visual-exploration-gaussian-processes), which is just a fancy name to denote a [collection of random variables such that the joint distribution of those random variables is a multivariate *Gaussian probability distribution*](https://en.wikipedia.org/wiki/Gaussian_process) (hence the name *Gaussian* process). Therefore, in BO, we often use a Gaussian probability distribution (the surrogate model) to model the possible functions that are consistent with the data. In other words, given that we do not know $f$, rather than finding the usual *point estimate* (or maximum likelihood estimate), like in the usual case of supervised learning mentioned above, we maintain a Gaussian probability distribution that describes our uncertainty about the unknown $f$.
  
  The method of statistical inference is often just an iterative application of the Bayes rule (hence the name *Bayesian* optimization), where you want to find the posterior, given a prior, a likelihood and the evidence. In BO, you usually place a prior on $f$, which is a multivariate Gaussian distribution, then you use the Bayes rule to find the posterior distribution of $f$ given the data.
  
  What is the data in this case? In BO, the data are the outputs of $f$ evaluated at certain points of the domain of $f$. The acquisition function is used to choose these points of the domain of $f$, based on the computed posterior distribution. In other words, based on the current uncertainty about $f$ (the posterior), the acquisition function attempts to cleverly choose points of the domain of $f$, $\mathbf{x} \in \mathbb{R}^d$, which will be used to find an updated posterior. Why do we need the acquisition function? Why can't we simply evaluate $f$ at random domain points? Given that $f$ is expensive to evaluate, we need a clever way to choose the points where we want to evaluate $f$. More specifically, we want to evaluate $f$ where we are more uncertain about it.
  
  There are several acquisition functions, such as [*expected improvement, knowledge-gradient, entropy search*, and *predictive entropy search*](https://arxiv.org/pdf/1807.02811.pdf), so there are different ways of choosing the points of the domain of $f$ where we want to evaluate it to update the posterior, each of which deals with the exploration-exploitation dilemma differently.
  
  ### What can Bayesian optimization be used for?
  
  BO can be used for tuning hyper-parameters (also called hyper-parameter optimisation) of machine learning models, such as neural networks, but [it has also been used to solve other problems](https://arxiv.org/pdf/1807.02811.pdf).
  
  ## What is an alternative surrogate model?
  
  In the book [Automated Machine Learning: Methods, Systems, Challenges](https://link.springer.com/content/pdf/10.1007%2F978-3-030-05318-5.pdf) (by Frank Hutter et al.) that you are quoting, the authors say that the commonly used surrogate model *Gaussian process* scales cubically in the number of data points, so sparse Gaussian processes are often used. Moreover, Gaussian processes also scale badly with the number of dimensions. In [section 1.3.2.2.](https://link.springer.com/content/pdf/10.1007%2F978-3-030-05318-5.pdf), the authors describe some [**alternative**](https://www.merriam-webster.com/dictionary/alternative) surrogate models to the Gaussian processes, for example, alternatives that use neural networks or random forests.
  
  
  ---
  
  ### Answer by username_1 on 2019-11-16. Score: 5
  
  A surrogate model is a simplified model. It is a mapping $y\_S=f\_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: [Engineering Design via Surrogate Modelling: A Practical Guide](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470770801)
  
  In the context of Bayesian optimization, one wants to optimize a function $y=f(x)$ which is expensive (very time consuming) to evaluate, therefore one optimizes the surrogate model $y\_S=f\_S(x)$ which is cheaper (faster) to evaluate.
  
  > **Comments:**
  > **username_2**: Forgive my ignorance, but why exactly is yS=fS(x) faster to evaluate?  
  > **username_1**: Imagine that the original model is computed from Finite Element simulations (x would be some geometric parameter or material constant for instance and f(x) some quantity of interest) and f_S is a polynomial approximation like a0 + a1x + a2x^2. f(x) can take some hours to evaluate whereas f_S(x) can be calculated pretty fast  
  
  ---
  
  ### Answer by username_3 on 2021-04-15. Score: 0
  
  Recently, I've been thinking this question as well. After reading several papers, finally came up with some thoughts about the surrogate model.
  In FEM(finite element method), we try to find a weak form to approximate the strong form so that we can solve the weak form analytically. (weak form: approximation equation; strong form: PDE in real world) In my opinion, the surrogate model can be regarded as 'weak form'. There are many methods can form a surrogate model. And if we use a NN model as the surrogate model, the training process is equivalent to 'solving analytically'.
  
  
  ---
  
  