Answers.Body: <p>Neural nets incorporate prior knowledge. This can be done in two ways: the first (most frequent and more robust) is in data augmentation. For example in convolutional networks, if we know that the "value" (whatever that is, class/regression) of the object we are looking is rotational/translational invariant (our prior knowledge), then we augment the data with random rotations/shifts. The second is in the loss function with some additional term.   </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-16T00:30:25.403Z
Answers.LastActivityDate: 2018-02-16T00:30:25.403Z
Answers.UserDisplayName: Foivos
Answers.Body: <p>Yes, we can do it in a deep learner.</p>
  <p>For example, suppose we have an input vector likes <span class="math-container">$(a, b)$</span> and from prior knowledge, we know <span class="math-container">$a^2 + b^2$</span> is important too. Hence, we can add this value to the vectors likes <span class="math-container">$(a, b, a^2 + b^2)$</span>.</p>
  <p>As another example, suppose date time is important in your data, but not encoded in the input vector. We can add this to the input vector as a third dimension.</p>
  <p>In summary, it depends on the structure of the prior knowledge, we can encode it into the input vector.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-15T19:56:29.810Z
Answers.Text: I think the questioner was looking for something like LSTM rather than ways the network designer could incorporate their own prior knowledge.
Answers.UserDisplayName: antlersoft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-02-15T17:21:56.170Z
Answers.LastActivityDate: 2020-12-03T14:41:31.547Z
Answers.LastEditDate: 2020-12-03T14:41:31.547Z
Answers.UserDisplayName: OmG
Answers.Body: <p>To add to the Foivos's answer,</p>
  
  <p>Convolutional Neural Networks are shift-invariant.
  Fukushima introduced this to his Neocognitron.
  There is a trail to introduce scale-invariance to CNN.
  <a href="https://arxiv.org/abs/1411.6369" rel="nofollow noreferrer">https://arxiv.org/abs/1411.6369</a></p>
  
  <p>Also, CNN uses structural characteristic for the prior knowledge.</p>
  
  <p>And neural networks are locally smooth.</p>
  
  <p>It is not perfect, but neural networks are incorporating a lot of prior knowledge.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-07T08:33:33.747Z
Answers.LastActivityDate: 2018-07-07T08:33:33.747Z
Answers.UserDisplayName: Yu Kobayashi
Answers.Body: <p>It kinda depends on how exactly you define knowledge, and what you believe about what the weights in a trained NN model really represent. But to answer this question in the most straightforward possible way (hopefully without sounding glib), then yes, a NN can be pre-trained, and then you can take that model and apply additional training to it, so in a sense, it is using "prior knowledge". </p>
  
  <p>If, OTOH, knowledge means something a little different to you, and you're thinking about the kind of knowledge that's encoded in a semantic network, or a conceptual graph, or something of that nature, then I don't know - offhand - of any direct way to integrate that into an ANN. What you might be able to do is combine the NN with a different kind of reasoner that reasons of the semantic network / conceptual graph, and then integrate the results.  AFAIK, the best way to do that is an unsolved research problem. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-08T07:06:44.577Z
Answers.LastActivityDate: 2018-07-08T07:06:44.577Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p>A simple example of this is token embeddings. If &quot;prior knowledge&quot; just means anything known prior to creation of the graph, then using pretrained vector embeddings meets this criteria. This is simply a way to provide a fixed method for projecting tokens into higher-dimensional space instead of training it at the same time as the rest of the model. Given that vector embeddings are somewhat interpretable and that the same embedding can be reused across tasks and models, I'd consider pretrained embeddings to be prior knowledge being incorporated.</p>
  <p>The embeddings could also technically be handcrafted, but I'm not aware of any work like that and am skeptical of its usefulness in deep models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T15:37:27.127Z
Answers.LastActivityDate: 2020-12-03T15:37:27.127Z
Answers.UserDisplayName: Andy
Body: <p>I was reading Gary Marcus's a <a href="https://arxiv.org/abs/1801.00631" rel="nofollow noreferrer">Critical Appraisal of Deep Learning</a>. One of his criticisms is that neural networks don't incorporate prior knowledge in tackling a problem. My question is: have there been any attempts at encoding prior knowledge in deep neural networks?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-15T16:20:19.157Z
Comments.Text: Learning in a NN is iterative, thus, based in prior knowledge. In particular, initial value of NN parameters is obviously a prior knowledge.
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-15T17:21:36.327Z
Comments.Text: @pasabaporaqui thanks a lot for the feedback. What my question is driving at is, in most real life problems we have additional information about the domain. Is there a way to inject this prior domain knowledge to improve the success of the model?
Comments.UserDisplayName: Seth Simba
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-15T19:50:22.197Z
Comments.Text: Strictly talking, a neural net doesn't learn. The process of learning is done outside the net (i.e. backpropagation algorithm). This is one of the reason is not correct to say that a NN simulates brain (in the brain, learning is done inside the net). If it doesn't learn, it doesn't use prior knowledge. Prior knowledge can be used to optimize the learning algorithm, including the initial net parameters, not the net behavior itself. If one rule is included in the net (as output and using it in the error function), it is not prior knowledge but a problem constrain that the net will try to fulfill
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-15T19:55:05.763Z
Comments.Text: Lookup LSTM in relation to neural networks-- it's a method to take advantage of prior knowledge
Comments.UserDisplayName: antlersoft
Tags: <neural-networks><deep-learning><deep-neural-networks><algorithmic-bias><inductive-bias>
Title: Can prior knowledge be encoded in deep neural networks?
UserDisplayName: Seth Simba
ThreadText: # Can prior knowledge be encoded in deep neural networks?
  
  (Asked by: username_0 on 2018-02-15)
  
  I was reading Gary Marcus's a [Critical Appraisal of Deep Learning](https://arxiv.org/abs/1801.00631). One of his criticisms is that neural networks don't incorporate prior knowledge in tackling a problem. My question is: have there been any attempts at encoding prior knowledge in deep neural networks?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks, algorithmic-bias, inductive-bias
  > **Comments:**
  > **username_1**: Learning in a NN is iterative, thus, based in prior knowledge. In particular, initial value of NN parameters is obviously a prior knowledge.  
  > **username_0**: @pasabaporaqui thanks a lot for the feedback. What my question is driving at is, in most real life problems we have additional information about the domain. Is there a way to inject this prior domain knowledge to improve the success of the model?  
  > **username_1**: Strictly talking, a neural net doesn't learn. The process of learning is done outside the net (i.e. backpropagation algorithm). This is one of the reason is not correct to say that a NN simulates brain (in the brain, learning is done inside the net). If it doesn't learn, it doesn't use prior knowledge. Prior knowledge can be used to optimize the learning algorithm, including the initial net parameters, not the net behavior itself. If one rule is included in the net (as output and using it in the error function), it is not prior knowledge but a problem constrain that the net will try to fulfill  
  > **username_2**: Lookup LSTM in relation to neural networks-- it's a method to take advantage of prior knowledge  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_3 on 2018-02-16. Score: 5
  
  Neural nets incorporate prior knowledge. This can be done in two ways: the first (most frequent and more robust) is in data augmentation. For example in convolutional networks, if we know that the "value" (whatever that is, class/regression) of the object we are looking is rotational/translational invariant (our prior knowledge), then we augment the data with random rotations/shifts. The second is in the loss function with some additional term.
  
  
  ---
  
  ### Answer by username_4 on 2018-02-15. Score: 4
  
  Yes, we can do it in a deep learner.
  
  For example, suppose we have an input vector likes $(a, b)$ and from prior knowledge, we know $a^2 + b^2$ is important too. Hence, we can add this value to the vectors likes $(a, b, a^2 + b^2)$.
  
  As another example, suppose date time is important in your data, but not encoded in the input vector. We can add this to the input vector as a third dimension.
  
  In summary, it depends on the structure of the prior knowledge, we can encode it into the input vector.
  
  > **Comments:**
  > **username_2**: I think the questioner was looking for something like LSTM rather than ways the network designer could incorporate their own prior knowledge.  
  
  ---
  
  ### Answer by username_5 on 2018-07-07. Score: 3
  
  To add to the Foivos's answer,
  
  Convolutional Neural Networks are shift-invariant.
  Fukushima introduced this to his Neocognitron.
  There is a trail to introduce scale-invariance to CNN.
  <https://arxiv.org/abs/1411.6369>
  
  Also, CNN uses structural characteristic for the prior knowledge.
  
  And neural networks are locally smooth.
  
  It is not perfect, but neural networks are incorporating a lot of prior knowledge.
  
  
  ---
  
  ### Answer by username_6 on 2018-07-08. Score: 1
  
  It kinda depends on how exactly you define knowledge, and what you believe about what the weights in a trained NN model really represent. But to answer this question in the most straightforward possible way (hopefully without sounding glib), then yes, a NN can be pre-trained, and then you can take that model and apply additional training to it, so in a sense, it is using "prior knowledge".
  
  If, OTOH, knowledge means something a little different to you, and you're thinking about the kind of knowledge that's encoded in a semantic network, or a conceptual graph, or something of that nature, then I don't know - offhand - of any direct way to integrate that into an ANN. What you might be able to do is combine the NN with a different kind of reasoner that reasons of the semantic network / conceptual graph, and then integrate the results. AFAIK, the best way to do that is an unsolved research problem.
  
  
  ---
  
  ### Answer by username_7 on 2020-12-03. Score: 0
  
  A simple example of this is token embeddings. If "prior knowledge" just means anything known prior to creation of the graph, then using pretrained vector embeddings meets this criteria. This is simply a way to provide a fixed method for projecting tokens into higher-dimensional space instead of training it at the same time as the rest of the model. Given that vector embeddings are somewhat interpretable and that the same embedding can be reused across tasks and models, I'd consider pretrained embeddings to be prior knowledge being incorporated.
  
  The embeddings could also technically be handcrafted, but I'm not aware of any work like that and am skeptical of its usefulness in deep models.
  
  
  ---
  
  