Answers.Body: <p>First of all, I would like to point out the main differences between knowledge base and (Deep) machine learning, specially when the main focus is on "AI" not "Data Science":</p>
  
  <ul>
  <li><p>NNs are like a black box; Even if they learn a dataset and gain the power of generalization over the problem domain, you'd never know how they are working. if you scrutinize the details of the developed model, all you see are digits, weights, poor and strong connections and transform functions. the "feature extraction" step before the training phase literally tells you: "hey human, enough with your complicated world, let's start zeros and ones". In the case of DL, it is worse! we do not even see what the selected and effective features are. I'm not a DL expert but as much as I know, DL's black box is darker! But knowledge bases are written in a human-friendly language. after a knowledge accumulation phase, you could see all the connections between the entities, and more important, you could interpret those connections. if you cut a wire in a knowledge base, your model will lose just a bit of its power, and you know what exactly it will lose; for example disconnecting the "Pluto" node from the "solar system" node, will tell your model what deGrasse Tyson told us. but in a ML model, this might turn it into a pure useless one: what happens if you manipulate the connection between the neuron number 14 and 47 in a NN model used to predict which planets belong to the solar system?!</p></li>
  <li><p>ML models are merely an inscription of the data. They do not have the power of inference, and they don't give you one. knowledge base is on the other hand capable of inference from the prior knowledge as you indicated in your question. It is shown that DL models that have been trained with say image classification data, could also be applied to voice detection problem. But this doesn't mean DL models could apply its prior knowledge in the domain of images to the domain of voices.</p></li>
  <li><p>You need kilos of data for traditional ML algorithms and tons of data for DL ones. but a single instance of a dataset will create a meaningful knowledge base for you.</p></li>
  </ul>
  
  <p>There are two main research topics in NLP: machine translation and question answering. Practically it has been shown that DL works significantly with machine translation problems but acts kind of stupid in question answering challenge, specially when the domain of topics covered in the human-machine conversation is broad. Knowledge bases are no good choice for machine translation but are probably the key to a noble question answering machine. Since what matters in machine translation is only the translated version of a text (and I don't care how on earth has the machine done that as far as it is true) but in question answering problem, I don't need a parrot who repeats the same information I gave it to him, but an intelligent creature who gives me "apple is eatable" after I tell him "apple is a fruit" and "all fruits are eatable". ML models are used to elicit underneath patterns from the dataset (translation) while knowledge bases are used to extend the domain of knowns. ML models nitpick, KBs explore!</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-06T18:26:15.133Z
Answers.Text: All fruits are edible, so long as they are not metaphoric, such the fruits "of one's labor".  (Then again, we could use "devour" in a metaphoric sense, such as when one devours a "tasty" stack answer and digests its contents;)
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-16T03:56:20.750Z
Answers.LastActivityDate: 2017-04-16T04:02:48.303Z
Answers.LastEditDate: 2017-04-16T04:02:48.303Z
Answers.UserDisplayName: Alireza
Answers.Body: <p>Although asked over 3 years ago, the question is still interesting and while I agree with the original answer, a lot can be added to it.</p>
  <p>First, I'd like to point out that the term &quot;knowledge base&quot; is very ambiguous and it means different things to different people. For example, there is no sharp distinction between knowledge base and neural network. By now NN can be so large that it essentially encodes knowledge as GPT does. So the distinction becomes a question of interface. And NN are no longer as opaque since many new techniques are available to probe the knowledge inside NN. Even more fundamental dustinction between symbolic and neural reasoning becoming less important when hybrid AI combines both in a intertwined fashion. So the historical divisions were largely about technologies and not the essence of AI.</p>
  <p>Second, when it comes to NLP there is a fundamental distinction between language as surface form of information used for communication and knowledge as deep information which cannot be accessed directly even with traditional database technologies. That fundamental divide makes the historical differences even less relevant today. NLP is where that interplay between surface and deep forms was at the forefront of AI but the same is now happening with vision and planning. The question becomes - How do we architect the interface between deep knowledge (however it is represented) and surface communication? At the moment the natural language seems to be the only viable answer. So, for example there is effort to develop a natural language interface to replace plethora of query languages use by systems.</p>
  <p>My personal prediction is that natural language will slowly evolve to include variety of technical languages and multimodal interactions. But it is not clear at all how this will happen.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-19T20:31:11.157Z
Answers.LastActivityDate: 2020-10-22T21:27:07.263Z
Answers.LastEditDate: 2020-10-22T21:27:07.263Z
Answers.UserDisplayName: Len Yabloko
Answers.Body: <p>It seems that Automated Knowledge Base Construction would be unfavorable.</p>
  <p>As <a href="https://matt-gardner.github.io/" rel="nofollow noreferrer">Matt Gardner</a> noted in <a href="https://nlphighlights.allennlp.org/083_knowledge_base_construction_with_sebastian_riedel" rel="nofollow noreferrer">NLP Highlights</a> in 2019 that:</p>
  <blockquote>
  <p>Um, but I know that Google, for instance, canceled their knowledge base construction project because there wasn’t high enough precision to actually be useful in their product.</p>
  </blockquote>
  <p>The canceled project <a href="https://research.google/pubs/pub45634/" rel="nofollow noreferrer">Knowledge Vault</a> is an Automated Knowledge Base Construction(AKBC) project launched in August 2014.</p>
  <p>There are three methods to integrate knowledge into the neural networks: 1) pre-trained models like <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">BERT</a>, <a href="https://github.com/google-research/electra" rel="nofollow noreferrer">ELECTRA</a>; 2) <a href="https://arxiv.org/abs/2005.11401" rel="nofollow noreferrer">retrieval-augmented generative model</a>; 3) flesh out the triples into natural text as that in <a href="https://ai.googleblog.com/2021/05/kelm-integrating-knowledge-graphs-with.html" rel="nofollow noreferrer">KELM</a>.</p>
  <p>In a 2020 paper <a href="https://ai.googleblog.com/2020/08/realm-integrating-retrieval-into.html" rel="nofollow noreferrer">REALM: Integrating Retrieval into Language Representation Models</a>, they utilized a retrieval rather than a knowledge base to enrich neural networks. And the best systems in the <a href="https://efficientqa.github.io/assets/report.pdf" rel="nofollow noreferrer">NeurIPS 2020 EfficientQA competition</a> all relied on retrieval.</p>
  <p>Knowledge bases that are actively being maintained receive a lot of annotation and curation, as stated in <a href="https://nlphighlights.allennlp.org/083_knowledge_base_construction_with_sebastian_riedel" rel="nofollow noreferrer">that podcast</a>. If curation and annotation are not sufficient, the knowledge base maybe cannot apply in AI.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-16T07:13:25.770Z
Answers.LastActivityDate: 2021-05-22T07:11:20.870Z
Answers.LastEditDate: 2021-05-22T07:11:20.870Z
Answers.UserDisplayName: Lerner Zhang
Body: <p>Nowadays, artificial intelligence seems almost equal to machine learning, especially deep learning. Some have said that deep learning will replace human experts, traditionally very important for feature engineering, in this field. It is said that two breakthroughs underpinned the rise of deep learning: on one hand, neuroscience, and <a href="https://en.wikipedia.org/wiki/Neuroplasticity" rel="nofollow noreferrer">neuroplasticity</a> in particular, tells us that like the human brain, which is highly plastic, artificial networks can be utilized to model almost all functions; on the other hand, the increase in computational power, in particular the introduction of GPU and FPGA, has boosted algorithmic intelligence in a magnificent way, and has been making the models created decades ago immensely powerful and versatile. I'll add that the big data (mostly labeled data) accumulated over the past years is also relevant.   </p>
  
  <p>Such developments bring computer vision (and voice recognition) into a new era, but in natural language processing and expert systems, the situation hasn't seemed to have changed very much. </p>
  
  <p>Achieving common sense for the neural networks seems a tall order, but most sentences, conversations and short texts contain inferences that should be drawn from the background world knowledge. Thus knowledge graphing is of great importance to artificial intelligence. Neural networks can be harnessed in building knowledge bases but it seems that neural network models have difficulty utilizing these constructed knowledge bases.  </p>
  
  <p>My questions are: </p>
  
  <ol>
  <li><p>Is a knowledge base (for instance, a "knowledge graph", as coined by Google) a promising branch in AI? If so, in what ways KB can empower machine learning? How can we incorporate discrete latent variables into NLU and NLG?</p></li>
  <li><p>For survival in an age dominated by DL, where is the direction for the knowledge base (or the umbrella term symbolic approach)? Is <a href="http://www.wolfram.com/" rel="nofollow noreferrer">Wolfram</a>-like z dynamic knowledge base the new direction? Or any new directions?</p></li>
  </ol>
  
  <p>Am I missing something fundamental, or some ideas that address these issues?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-03-06T17:59:03.217Z
Comments.Text: I worked for a major IT company that at one time marketed AI products. The problem with a neural network approach is that there isn't a knowledge base per sey. So without any sort of rules, it is impossible for a neural network to explain "why." Training a neural network, then formulating rules that duplicate the network would give you such answers. But there isn't any form of machine learning that is now capable of such behavior.
Comments.UserDisplayName: MaxW
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T08:25:37.257Z
Comments.Text: I learned from this podcast([Knowledge Base Construction, with Sebastian Riedel](https://nlphighlights.allennlp.org/083_knowledge_base_construction_with_sebastian_riedel)) that knowledge base is still not applicable for NLP models.
Comments.UserDisplayName: Lerner Zhang
Tags: <natural-language-processing><knowledge-representation><expert-systems><symbolic-computing><knowledge-base>
Title: What roles knowledge bases play now and will play in the future?
UserDisplayName: Lerner Zhang
ThreadText: # What roles knowledge bases play now and will play in the future?
  
  (Asked by: username_0 on 2017-03-05)
  
  Nowadays, artificial intelligence seems almost equal to machine learning, especially deep learning. Some have said that deep learning will replace human experts, traditionally very important for feature engineering, in this field. It is said that two breakthroughs underpinned the rise of deep learning: on one hand, neuroscience, and [neuroplasticity](https://en.wikipedia.org/wiki/Neuroplasticity) in particular, tells us that like the human brain, which is highly plastic, artificial networks can be utilized to model almost all functions; on the other hand, the increase in computational power, in particular the introduction of GPU and FPGA, has boosted algorithmic intelligence in a magnificent way, and has been making the models created decades ago immensely powerful and versatile. I'll add that the big data (mostly labeled data) accumulated over the past years is also relevant.
  
  Such developments bring computer vision (and voice recognition) into a new era, but in natural language processing and expert systems, the situation hasn't seemed to have changed very much.
  
  Achieving common sense for the neural networks seems a tall order, but most sentences, conversations and short texts contain inferences that should be drawn from the background world knowledge. Thus knowledge graphing is of great importance to artificial intelligence. Neural networks can be harnessed in building knowledge bases but it seems that neural network models have difficulty utilizing these constructed knowledge bases.
  
  My questions are:
  
  1. Is a knowledge base (for instance, a "knowledge graph", as coined by Google) a promising branch in AI? If so, in what ways KB can empower machine learning? How can we incorporate discrete latent variables into NLU and NLG?
  2. For survival in an age dominated by DL, where is the direction for the knowledge base (or the umbrella term symbolic approach)? Is [Wolfram](http://www.wolfram.com/)-like z dynamic knowledge base the new direction? Or any new directions?
  
  Am I missing something fundamental, or some ideas that address these issues?
  
  **Tags:** natural-language-processing, knowledge-representation, expert-systems, symbolic-computing, knowledge-base
  
  > **Comments:**
  > **username_1**: I worked for a major IT company that at one time marketed AI products. The problem with a neural network approach is that there isn't a knowledge base per sey. So without any sort of rules, it is impossible for a neural network to explain "why." Training a neural network, then formulating rules that duplicate the network would give you such answers. But there isn't any form of machine learning that is now capable of such behavior.  
  > **username_0**: I learned from this podcast([Knowledge Base Construction, with Sebastian Riedel](https://nlphighlights.allennlp.org/083_knowledge_base_construction_with_sebastian_riedel)) that knowledge base is still not applicable for NLP models.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2017-04-16. Score: 5
  
  First of all, I would like to point out the main differences between knowledge base and (Deep) machine learning, specially when the main focus is on "AI" not "Data Science":
  
  * NNs are like a black box; Even if they learn a dataset and gain the power of generalization over the problem domain, you'd never know how they are working. if you scrutinize the details of the developed model, all you see are digits, weights, poor and strong connections and transform functions. the "feature extraction" step before the training phase literally tells you: "hey human, enough with your complicated world, let's start zeros and ones". In the case of DL, it is worse! we do not even see what the selected and effective features are. I'm not a DL expert but as much as I know, DL's black box is darker! But knowledge bases are written in a human-friendly language. after a knowledge accumulation phase, you could see all the connections between the entities, and more important, you could interpret those connections. if you cut a wire in a knowledge base, your model will lose just a bit of its power, and you know what exactly it will lose; for example disconnecting the "Pluto" node from the "solar system" node, will tell your model what deGrasse Tyson told us. but in a ML model, this might turn it into a pure useless one: what happens if you manipulate the connection between the neuron number 14 and 47 in a NN model used to predict which planets belong to the solar system?!
  * ML models are merely an inscription of the data. They do not have the power of inference, and they don't give you one. knowledge base is on the other hand capable of inference from the prior knowledge as you indicated in your question. It is shown that DL models that have been trained with say image classification data, could also be applied to voice detection problem. But this doesn't mean DL models could apply its prior knowledge in the domain of images to the domain of voices.
  * You need kilos of data for traditional ML algorithms and tons of data for DL ones. but a single instance of a dataset will create a meaningful knowledge base for you.
  
  There are two main research topics in NLP: machine translation and question answering. Practically it has been shown that DL works significantly with machine translation problems but acts kind of stupid in question answering challenge, specially when the domain of topics covered in the human-machine conversation is broad. Knowledge bases are no good choice for machine translation but are probably the key to a noble question answering machine. Since what matters in machine translation is only the translated version of a text (and I don't care how on earth has the machine done that as far as it is true) but in question answering problem, I don't need a parrot who repeats the same information I gave it to him, but an intelligent creature who gives me "apple is eatable" after I tell him "apple is a fruit" and "all fruits are eatable". ML models are used to elicit underneath patterns from the dataset (translation) while knowledge bases are used to extend the domain of knowns. ML models nitpick, KBs explore!
  
  > **Comments:**
  > **username_3**: All fruits are edible, so long as they are not metaphoric, such the fruits "of one's labor".  (Then again, we could use "devour" in a metaphoric sense, such as when one devours a "tasty" stack answer and digests its contents;)  
  
  ---
  
  ### Answer by username_4 on 2020-10-19. Score: 4
  
  Although asked over 3 years ago, the question is still interesting and while I agree with the original answer, a lot can be added to it.
  
  First, I'd like to point out that the term "knowledge base" is very ambiguous and it means different things to different people. For example, there is no sharp distinction between knowledge base and neural network. By now NN can be so large that it essentially encodes knowledge as GPT does. So the distinction becomes a question of interface. And NN are no longer as opaque since many new techniques are available to probe the knowledge inside NN. Even more fundamental dustinction between symbolic and neural reasoning becoming less important when hybrid AI combines both in a intertwined fashion. So the historical divisions were largely about technologies and not the essence of AI.
  
  Second, when it comes to NLP there is a fundamental distinction between language as surface form of information used for communication and knowledge as deep information which cannot be accessed directly even with traditional database technologies. That fundamental divide makes the historical differences even less relevant today. NLP is where that interplay between surface and deep forms was at the forefront of AI but the same is now happening with vision and planning. The question becomes - How do we architect the interface between deep knowledge (however it is represented) and surface communication? At the moment the natural language seems to be the only viable answer. So, for example there is effort to develop a natural language interface to replace plethora of query languages use by systems.
  
  My personal prediction is that natural language will slowly evolve to include variety of technical languages and multimodal interactions. But it is not clear at all how this will happen.
  
  
  ---
  
  ### Answer by username_0 on 2021-01-16. Score: 1
  
  It seems that Automated Knowledge Base Construction would be unfavorable.
  
  As [Matt Gardner](https://matt-gardner.github.io/) noted in [NLP Highlights](https://nlphighlights.allennlp.org/083_knowledge_base_construction_with_sebastian_riedel) in 2019 that:
  
  > Um, but I know that Google, for instance, canceled their knowledge base construction project because there wasn’t high enough precision to actually be useful in their product.
  
  The canceled project [Knowledge Vault](https://research.google/pubs/pub45634/) is an Automated Knowledge Base Construction(AKBC) project launched in August 2014.
  
  There are three methods to integrate knowledge into the neural networks: 1) pre-trained models like [BERT](https://github.com/google-research/bert), [ELECTRA](https://github.com/google-research/electra); 2) [retrieval-augmented generative model](https://arxiv.org/abs/2005.11401); 3) flesh out the triples into natural text as that in [KELM](https://ai.googleblog.com/2021/05/kelm-integrating-knowledge-graphs-with.html).
  
  In a 2020 paper [REALM: Integrating Retrieval into Language Representation Models](https://ai.googleblog.com/2020/08/realm-integrating-retrieval-into.html), they utilized a retrieval rather than a knowledge base to enrich neural networks. And the best systems in the [NeurIPS 2020 EfficientQA competition](https://efficientqa.github.io/assets/report.pdf) all relied on retrieval.
  
  Knowledge bases that are actively being maintained receive a lot of annotation and curation, as stated in [that podcast](https://nlphighlights.allennlp.org/083_knowledge_base_construction_with_sebastian_riedel). If curation and annotation are not sufficient, the knowledge base maybe cannot apply in AI.
  
  
  ---
  
  