Answers.Body: <p>Just for the sake of completeness, I'll point out that Recurrent Neural Nets (i.e. neural nets with backwards connections) are frequently used for for Natural Language Processing (NLP). This includes variants like Bidirectional, Jordan and Elman Networks. Long Short-Term Memory (LSTM) is a more sophisticated neural net algorithm which can accomplish the same time and sequence-based tasks, but which can leverage standard learning methods like backprop since it doesn't suffer from the "vanishing gradient problem." This is because LSTMs have been brilliantly engineered as "perfect integrators," which makes it a lot easier to calculate the error gradients etc. over long periods of time. In contrast, learning with RNNs is still not theoretically well-grounded and is difficult to calculate through existing methods like Backpropagation Through Time (BPTT). In Time Delay Neural Networks (TDNNs), the idea is to add new neurons and connections with each new training example across a stretch of time or training sequence; unfortunately, this places a practical limitation on how many examples you can feed into the net before the size of the network gets out of hand or it starts forgetting, just as with RNNs. LSTMs have much longer memories (especially when augmented with Neural Turing Machines) so that'd be my first choice, assuming I wanted to use neural nets for NLP purposes. My knowledge of the subject is limited though (I'm still trying to learn the ropes) so there may be other important neural net algorithms I'm overlooking...</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-18T16:50:48.577Z
Answers.LastActivityDate: 2016-09-18T16:50:48.577Z
Answers.UserDisplayName: SQLServerSteve
Answers.Body: <p>The umbrella term for your problem is called <strong>natural language processing (NLP)</strong> -- a topic under artificial intelligence. </p>
  
  <p>There are many subtopics to this field including language semantics, grammatical analysis, parts of speech tagging, domain specific context analysis, etc. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-14T18:19:13.600Z
Answers.LastActivityDate: 2016-11-05T12:17:29.653Z
Answers.LastEditDate: 2016-11-05T12:17:29.653Z
Answers.UserDisplayName: Ébe Isaac
Answers.Body: <p>The general research area is known as <a href="https://en.wikipedia.org/wiki/Grammar_induction" rel="nofollow noreferrer">grammar induction</a>.</p>
  
  <p>It is generally framed as a supervised learning problem, with the input presented as raw text, and the desired output the corresponding <em>parse tree</em>.
  The training set often consists of both positive and negative examples.</p>
  
  <p>There is no single best method for achieving this, but some of the techniques that have been used to date include:</p>
  
  <ul>
  <li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.73.7904&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Bayesian approaches</a></li>
  <li><a href="https://pdfs.semanticscholar.org/d6ce/9f5e8dc23fe121e5f31f906b99b916027522.pdf" rel="nofollow noreferrer">Genetic Algorithms</a></li>
  <li><a href="https://www.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2011/rapporter11/svantesson_marten_11077.pdf" rel="nofollow noreferrer">Genetic Programming</a></li>
  <li><a href="http://dl.acm.org/citation.cfm?id=356816" rel="nofollow noreferrer">Blackboard Architectures</a></li>
  <li>The <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.35.1924" rel="nofollow noreferrer">UpWrite Predictor</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-14T01:40:20.553Z
Answers.Text: Well, seven years later, it turns out all you needed to do was get a ludicrous amount of english text and learn to predict the next word.
Answers.UserDisplayName: dieki
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-09-14T15:23:33.017Z
Answers.LastActivityDate: 2019-09-04T12:57:16.340Z
Answers.LastEditDate: 2019-09-04T12:57:16.340Z
Answers.UserDisplayName: NietzscheanAI
Body: <p>I was think about AIs and how they would work, when I realised that I couldn't think of a way that an AI could be taught language. A child tends to learn language through associations of language and pictures to an object (e.g., people saying the word "dog" while around a dog, and later realising that  people say "a dog" and "a car" and learn what "a" means, etc.). However, a text based AI couldn't use this method to learn, as they wouldn't have access to any sort of input device.</p>
  
  <p>The only way I could come up with is programming in every word, and rule, in the English language (or whatever language it is meant to 'speak' in), however that would, potentially, take years to do.</p>
  
  <p>Does anyone have any ideas on how this could be done? Or if it has been done already, if so how?</p>
  
  <p>By the way, in this context, I am using AI to mean an Artificial Intelligence system with near-human intelligence, and no prior knowledge of language.</p>
  
Tags: <machine-learning><natural-language-processing>
Title: How would an AI learn language?
UserDisplayName: AvahW
ThreadText: # How would an AI learn language?
  
  (Asked by: username_0 on 2016-09-14)
  
  I was think about AIs and how they would work, when I realised that I couldn't think of a way that an AI could be taught language. A child tends to learn language through associations of language and pictures to an object (e.g., people saying the word "dog" while around a dog, and later realising that people say "a dog" and "a car" and learn what "a" means, etc.). However, a text based AI couldn't use this method to learn, as they wouldn't have access to any sort of input device.
  
  The only way I could come up with is programming in every word, and rule, in the English language (or whatever language it is meant to 'speak' in), however that would, potentially, take years to do.
  
  Does anyone have any ideas on how this could be done? Or if it has been done already, if so how?
  
  By the way, in this context, I am using AI to mean an Artificial Intelligence system with near-human intelligence, and no prior knowledge of language.
  
  **Tags:** machine-learning, natural-language-processing
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2016-09-14. Score: 15 | Accepted answer
  
  The general research area is known as [grammar induction](https://en.wikipedia.org/wiki/Grammar_induction).
  
  It is generally framed as a supervised learning problem, with the input presented as raw text, and the desired output the corresponding *parse tree*.
  The training set often consists of both positive and negative examples.
  
  There is no single best method for achieving this, but some of the techniques that have been used to date include:
  
  * [Bayesian approaches](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.73.7904&rep=rep1&type=pdf)
  * [Genetic Algorithms](https://pdfs.semanticscholar.org/d6ce/9f5e8dc23fe121e5f31f906b99b916027522.pdf)
  * [Genetic Programming](https://www.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2011/rapporter11/svantesson_marten_11077.pdf)
  * [Blackboard Architectures](http://dl.acm.org/citation.cfm?id=356816)
  * The [UpWrite Predictor](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.35.1924)
  
  > **Comments:**
  > **username_4**: Well, seven years later, it turns out all you needed to do was get a ludicrous amount of english text and learn to predict the next word.  
  
  ---
  
  ### Answer by username_2 on 2016-09-14. Score: 7
  
  The umbrella term for your problem is called **natural language processing (NLP)** -- a topic under artificial intelligence.
  
  There are many subtopics to this field including language semantics, grammatical analysis, parts of speech tagging, domain specific context analysis, etc.
  
  
  ---
  
  ### Answer by username_1 on 2016-09-18. Score: 5
  
  Just for the sake of completeness, I'll point out that Recurrent Neural Nets (i.e. neural nets with backwards connections) are frequently used for for Natural Language Processing (NLP). This includes variants like Bidirectional, Jordan and Elman Networks. Long Short-Term Memory (LSTM) is a more sophisticated neural net algorithm which can accomplish the same time and sequence-based tasks, but which can leverage standard learning methods like backprop since it doesn't suffer from the "vanishing gradient problem." This is because LSTMs have been brilliantly engineered as "perfect integrators," which makes it a lot easier to calculate the error gradients etc. over long periods of time. In contrast, learning with RNNs is still not theoretically well-grounded and is difficult to calculate through existing methods like Backpropagation Through Time (BPTT). In Time Delay Neural Networks (TDNNs), the idea is to add new neurons and connections with each new training example across a stretch of time or training sequence; unfortunately, this places a practical limitation on how many examples you can feed into the net before the size of the network gets out of hand or it starts forgetting, just as with RNNs. LSTMs have much longer memories (especially when augmented with Neural Turing Machines) so that'd be my first choice, assuming I wanted to use neural nets for NLP purposes. My knowledge of the subject is limited though (I'm still trying to learn the ropes) so there may be other important neural net algorithms I'm overlooking...
  
  
  ---
  
  