Answers.Body: <p>One way to look at intelligence is it's the way to compress the universe. That means we have a short mental representation of meaningful concepts.</p>
  <p>For example, if I would say &quot;there is a red swan in your building, it's dangerous and can kill you&quot;, you already have concepts of &quot;red&quot;, &quot;swan&quot;, &quot;danger&quot; and this easy allows you to add the bird to your classifier &quot;should I beware it or not&quot;. (and you probably have tried to imagine the red swan by now)</p>
  <p>There are similar properties in deep networks when deeper layers could respond to more abstract representation concepts. For example, if you have a convolutional network classifying the faces, the first layers would detect simple shapes like lines and circles, then more complex shapes and then parts of the face like eyes, nose and ears (see example <a href="https://users.informatik.haw-hamburg.de/%7Eubicomp/projekte/master2018-gsem/Zach/bericht.pdf" rel="nofollow noreferrer">here</a>)</p>
  <p>Now, let's imagine you have a classification task &quot;is the thing dangerous or not&quot;. You have a trained convolutional model and it has &quot;red&quot; and &quot;swan&quot; abstract features on some level. You could freeze all the preliminary weights and make a single backpropagation with a high learning rate only for the subnetwork which is of interest to you.</p>
  <p>That could work in theory, but the key challenge here is that most of the neural network representations are not interpretable at all and you would need to solve this problem before (there is a whole research direction with <a href="https://icml.cc/Conferences/2020/ScheduleMultitrack?event=5746" rel="nofollow noreferrer">workshops</a> and <a href="https://human-centered.ai/explainable-ai-2020/" rel="nofollow noreferrer">conferences</a> on the topic with varying success)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-31T10:34:59.987Z
Answers.Text: The human brain solves this problem easily. How does it do it?
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-31T10:45:38.627Z
Answers.Text: Does human brain solve it easily though? 
  If you told something out of your familiar context, you would probably have hard time to adopt the experience.
  And if you about the interpretability -- I don't know for sure, though I would speculate it related to multi-task of human and re-usability of concepts.
Answers.UserDisplayName: Kirill Fedyanin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-31T10:11:23.250Z
Answers.LastActivityDate: 2021-03-31T10:11:23.250Z
Answers.UserDisplayName: Kirill Fedyanin
Body: <p>Humans learn facts about the world like &quot;most A are B&quot; by own experience and by being told so (by other people or texts). The systems and mechanisms of storage and usage of such facts (by an &quot;experience system&quot; and a &quot;declarative system&quot;) are presumably quite different and may have to do with &quot;episodic memory&quot; and &quot;semantic memory&quot;. Nevertheless at least in the human brain the common currency are synaptic weights, and it would be quite interesting to know how these two systems cooperate.</p>
  <p>I assume that machine learning is mainly concerned with &quot;learning by own experience&quot; (= training data + annotations), be it supervised or unsupervised learning. I wonder which approaches there are that allow a neural network to &quot;learn by being told&quot;. One brute force approach might be to translate a declarative statement like &quot;most A are B&quot; into a set of synthetic training data, but that's definitely not how it works for humans.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T08:53:32.700Z
Comments.Text: I would say you are right, in that NNs only learn by being shown an input and an output, ie through 'experience'. "Being told" would map onto pre-defining the weights (ie bypass the training algorithm), but I guess they're too much of a black box for that to be feasible.
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T09:13:05.673Z
Comments.Text: @OliverMason. Thanks for the comment. I would not say "**predefine** the weights" or "**bypass** the training" but to complement/adjust/modulate/correct them (it) - or the other way around: let the training do the fine-tuning.
Comments.UserDisplayName: Hans-Peter Stricker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T09:15:40.683Z
Comments.Text: @OliverMason: To the rest what you say, I fully agree: The weights are too much of a black box to make anything into this direction feasible. (In human brains it works!)
Comments.UserDisplayName: Hans-Peter Stricker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T09:30:12.543Z
Comments.Text: Normally you start with random weights, and the training process adjusts them so that the mapping input to output is achieved; if you pre-define them with non-random values, you could in theory tell the NN to behave in a particular way (if you could choose the correct values).
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T10:07:53.607Z
Comments.Text: @OliverMason: Does it make a difference to start with some constant weights?
Comments.UserDisplayName: Hans-Peter Stricker
Tags: <neural-networks><machine-learning><deep-learning><declarative-programming>
Title: How to transfer declarative knowledge into neural networks
UserDisplayName: Hans-Peter Stricker
ThreadText: # How to transfer declarative knowledge into neural networks
  
  (Asked by: username_0 on 2021-03-31)
  
  Humans learn facts about the world like "most A are B" by own experience and by being told so (by other people or texts). The systems and mechanisms of storage and usage of such facts (by an "experience system" and a "declarative system") are presumably quite different and may have to do with "episodic memory" and "semantic memory". Nevertheless at least in the human brain the common currency are synaptic weights, and it would be quite interesting to know how these two systems cooperate.
  
  I assume that machine learning is mainly concerned with "learning by own experience" (= training data + annotations), be it supervised or unsupervised learning. I wonder which approaches there are that allow a neural network to "learn by being told". One brute force approach might be to translate a declarative statement like "most A are B" into a set of synthetic training data, but that's definitely not how it works for humans.
  
  **Tags:** neural-networks, machine-learning, deep-learning, declarative-programming
  > **Comments:**
  > **username_1**: I would say you are right, in that NNs only learn by being shown an input and an output, ie through 'experience'. "Being told" would map onto pre-defining the weights (ie bypass the training algorithm), but I guess they're too much of a black box for that to be feasible.  
  > **username_0**: @OliverMason. Thanks for the comment. I would not say "**predefine** the weights" or "**bypass** the training" but to complement/adjust/modulate/correct them (it) - or the other way around: let the training do the fine-tuning.  
  > **username_0**: @OliverMason: To the rest what you say, I fully agree: The weights are too much of a black box to make anything into this direction feasible. (In human brains it works!)  
  > **username_1**: Normally you start with random weights, and the training process adjusts them so that the mapping input to output is achieved; if you pre-define them with non-random values, you could in theory tell the NN to behave in a particular way (if you could choose the correct values).  
  > **username_0**: @OliverMason: Does it make a difference to start with some constant weights?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-31. Score: 1
  
  One way to look at intelligence is it's the way to compress the universe. That means we have a short mental representation of meaningful concepts.
  
  For example, if I would say "there is a red swan in your building, it's dangerous and can kill you", you already have concepts of "red", "swan", "danger" and this easy allows you to add the bird to your classifier "should I beware it or not". (and you probably have tried to imagine the red swan by now)
  
  There are similar properties in deep networks when deeper layers could respond to more abstract representation concepts. For example, if you have a convolutional network classifying the faces, the first layers would detect simple shapes like lines and circles, then more complex shapes and then parts of the face like eyes, nose and ears (see example [here](https://users.informatik.haw-hamburg.de/%7Eubicomp/projekte/master2018-gsem/Zach/bericht.pdf))
  
  Now, let's imagine you have a classification task "is the thing dangerous or not". You have a trained convolutional model and it has "red" and "swan" abstract features on some level. You could freeze all the preliminary weights and make a single backpropagation with a high learning rate only for the subnetwork which is of interest to you.
  
  That could work in theory, but the key challenge here is that most of the neural network representations are not interpretable at all and you would need to solve this problem before (there is a whole research direction with [workshops](https://icml.cc/Conferences/2020/ScheduleMultitrack?event=5746) and [conferences](https://human-centered.ai/explainable-ai-2020/) on the topic with varying success)
  
  > **Comments:**
  > **username_0**: The human brain solves this problem easily. How does it do it?  
  > **username_2**: Does human brain solve it easily though?  If you told something out of your familiar context, you would probably have hard time to adopt the experience. And if you about the interpretability -- I don't know for sure, though I would speculate it related to multi-task of human and re-usability of concepts.  
  
  ---
  
  