Answers.Body: <p>I would distinguish at least 2 cases when it comes to a generic expression like <em>prior knowledge</em>:</p>
  <ul>
  <li>generic extra information provide to a model, really close if not the same as feature engineering.</li>
  <li>literal prior probability distributions used to initialize or guide a model during training.</li>
  </ul>
  <p>For the first case there's plenty of examples that we can provide. The most intuitive maybe is use of masks in computer vision. Let's say we want to clean an image (i.e. haze removal) as a pre processing step for a self driving car system. Then we could train a model and feed to it not only the image captured by the camera, but also a depth mask estimated using another model. In this case the <em>other model</em> works as a prior distribution, since the model is not learning it, it just leverage that extra information that comes with the image. <a href="https://i.sstatic.net/SN4gi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/SN4gi.png" alt="enter image description here" /></a></p>
  <p>For the second case, there are specific class of models that learn and sometimes require prior distributions as an input, the most known to me are <a href="https://jorisbaan.nl/2021/03/02/introduction-to-bayesian-deep-learning.html#DeepLearning" rel="nofollow noreferrer">Bayesian Neural Networks</a>.
  <a href="https://i.sstatic.net/rQdvn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rQdvn.png" alt="enter image description here" /></a></p>
  <p>Why bothering providing a prior for these models? Well, there are at least 2 reasons: sometimes we have information about a system we're trying to describe, so we can make the training more efficient, for example we're might trying to fit a coin toss model, but we know the coin is not fair and that the resulting probabilities returned by the model should not be 50/50. The second reason is that sometimes we also want to train models that do not return only raw probabilities, but also estimate the uncertainty level of those probabilities. To do that, the model learns a posterior probability over the data, and it does that by updating an initial prior. Note that the prior for this class of models can also be randomly initialize.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-08T12:29:11.310Z
Answers.Text: This answer provides good information, but it mixes concepts, so it may be confusing. For example, you say that the second way to interpret "prior knowledge" is "literal prior probability distributions used to **initialize or guide a model during training**", but then you give the example of "Bayesian neural networks". It's true that BNNs are based on having distributions over the weights, but in your second point you mention probability distributions to **initialize or guide a model during training**, which seemed to suggest that you were referring to ways to initialise the weights.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-08T12:30:53.773Z
Answers.Text: Note that I am not saying that BNNs are not a good example where we can introduce prior knowledge. In fact, in BNNs, we can really specify priors for the weights (as priors in the Bayes' rule, which are used to introduce "prior knowledge"), but it just seems that you were referring to the initialization of the weights and other techniques to guide the training of non-Bayesian neural networks. Another thing you may want to point out is the relationship between "prior knowledge" and "inductive bias" (and/or model architecture).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-08T09:05:25.003Z
Answers.LastActivityDate: 2021-10-08T10:25:07.237Z
Answers.LastEditDate: 2021-10-08T10:25:07.237Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>It is known that machine learning algorithms expect feature engineering as an initial step. Now, consider the following paragraph, taken from <a href="https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=34" rel="nofollow noreferrer">1.1 The deep learning revolution</a> of the textbook named <strong>Deep learning with PyTorch</strong> by <em>Eli Stevens, Luca Antiga, Thomas Viehmann</em>, regarding the role of feature engineering in deep learning</p>
  <blockquote>
  <p>Deep learning, on the other hand, deals with finding such
  representations automatically, from raw data, in order to successfully
  perform a task. In the ones versus zeros example, filters would be
  refined during training by iteratively looking at pairs of examples
  and target labels. <strong>This is not to say that feature engineering has
  no place with deep learning; we often need to inject some form of
  prior knowledge in a learning system. However, the ability of a neural
  network to ingest data and extract useful representations on the basis
  of examples is what makes deep learning so powerful.</strong> The focus of
  deep learning practitioners is not so much on handcrafting those
  representations, but on operating on a mathematical entity so that
  it discovers representations from the training data autonomously.
  Often, these automatically created features are better than those that
  are handcrafted! As with many disruptive technologies, this fact has
  led to a change in perspective.</p>
  </blockquote>
  <p>The paragraph clearly saying that we need to inject some form of prior knowledge into the learning system. What can be a concrete example for such prior knowledge we are used in deep learning systems?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-08T12:23:36.243Z
Comments.Text: [Here](https://ai.stackexchange.com/q/5322/2444) is a related (if not duplicate) question.
Comments.UserDisplayName: nbro
Tags: <deep-learning><feature-engineering><inductive-bias>
Title: What can be an example for the prior knowledge used in Deep Learning systems?
UserDisplayName: hanugm
ThreadText: # What can be an example for the prior knowledge used in Deep Learning systems?
  
  (Asked by: username_0 on 2021-10-08)
  
  It is known that machine learning algorithms expect feature engineering as an initial step. Now, consider the following paragraph, taken from [1.1 The deep learning revolution](https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf#page=34) of the textbook named **Deep learning with PyTorch** by *Eli Stevens, Luca Antiga, Thomas Viehmann*, regarding the role of feature engineering in deep learning
  
  > Deep learning, on the other hand, deals with finding such
  > representations automatically, from raw data, in order to successfully
  > perform a task. In the ones versus zeros example, filters would be
  > refined during training by iteratively looking at pairs of examples
  > and target labels. **This is not to say that feature engineering has
  > no place with deep learning; we often need to inject some form of
  > prior knowledge in a learning system. However, the ability of a neural
  > network to ingest data and extract useful representations on the basis
  > of examples is what makes deep learning so powerful.** The focus of
  > deep learning practitioners is not so much on handcrafting those
  > representations, but on operating on a mathematical entity so that
  > it discovers representations from the training data autonomously.
  > Often, these automatically created features are better than those that
  > are handcrafted! As with many disruptive technologies, this fact has
  > led to a change in perspective.
  
  The paragraph clearly saying that we need to inject some form of prior knowledge into the learning system. What can be a concrete example for such prior knowledge we are used in deep learning systems?
  
  **Tags:** deep-learning, feature-engineering, inductive-bias
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/5322/2444) is a related (if not duplicate) question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-10-08. Score: 1
  
  I would distinguish at least 2 cases when it comes to a generic expression like *prior knowledge*:
  
  * generic extra information provide to a model, really close if not the same as feature engineering.
  * literal prior probability distributions used to initialize or guide a model during training.
  
  For the first case there's plenty of examples that we can provide. The most intuitive maybe is use of masks in computer vision. Let's say we want to clean an image (i.e. haze removal) as a pre processing step for a self driving car system. Then we could train a model and feed to it not only the image captured by the camera, but also a depth mask estimated using another model. In this case the *other model* works as a prior distribution, since the model is not learning it, it just leverage that extra information that comes with the image. [![enter image description here](https://i.sstatic.net/SN4gi.png)](https://i.sstatic.net/SN4gi.png)
  
  For the second case, there are specific class of models that learn and sometimes require prior distributions as an input, the most known to me are [Bayesian Neural Networks](https://jorisbaan.nl/2021/03/02/introduction-to-bayesian-deep-learning.html#DeepLearning).
  [![enter image description here](https://i.sstatic.net/rQdvn.png)](https://i.sstatic.net/rQdvn.png)
  
  Why bothering providing a prior for these models? Well, there are at least 2 reasons: sometimes we have information about a system we're trying to describe, so we can make the training more efficient, for example we're might trying to fit a coin toss model, but we know the coin is not fair and that the resulting probabilities returned by the model should not be 50/50. The second reason is that sometimes we also want to train models that do not return only raw probabilities, but also estimate the uncertainty level of those probabilities. To do that, the model learns a posterior probability over the data, and it does that by updating an initial prior. Note that the prior for this class of models can also be randomly initialize.
  
  > **Comments:**
  > **username_1**: This answer provides good information, but it mixes concepts, so it may be confusing. For example, you say that the second way to interpret "prior knowledge" is "literal prior probability distributions used to **initialize or guide a model during training**", but then you give the example of "Bayesian neural networks". It's true that BNNs are based on having distributions over the weights, but in your second point you mention probability distributions to **initialize or guide a model during training**, which seemed to suggest that you were referring to ways to initialise the weights.  
  > **username_1**: Note that I am not saying that BNNs are not a good example where we can introduce prior knowledge. In fact, in BNNs, we can really specify priors for the weights (as priors in the Bayes' rule, which are used to introduce "prior knowledge"), but it just seems that you were referring to the initialization of the weights and other techniques to guide the training of non-Bayesian neural networks. Another thing you may want to point out is the relationship between "prior knowledge" and "inductive bias" (and/or model architecture).  
  
  ---
  
  