Answers.Body: <p>I strongly disagree with all the former comments. Not because they are wrong, -which they are not - but because they are misleading - though unintentionally. </p>
  
  <p>For example: If one looks at these problems from an academic position, the problems will always seem insurmountable. This is because everything is coldly assessed and calculated in isolation to everything else.</p>
  
  <p>The answer predominantly lies in <strong><em>word association</em></strong>. You have to write a program that can process a vast database of digital books, to register every word and all the words in that language which are associated with it. Plus all the statistical information with each associated word and its associated punctuation. </p>
  
  <p>This will then give you the basis on which an AI can decide several things: </p>
  
  <ol>
  <li>Whether the structure of a given sentence is correct.</li>
  <li>If the structure is bad, what the probability is for determining the context and intent of what is being said.</li>
  <li>The correct meaning and application of a multifaceted word (Triumph), is by probability - according to the statistics.</li>
  <li>To determine where a conversation is likely to be going. </li>
  <li>What the correct grammar, and punctuation should be.</li>
  </ol>
  
  <p>So, in conclusion, you have two things to look for: Association and probability. </p>
  
  <p><em>When digitally databasing a language model, the possibility of word and sentence "strings" occurs, so that every variation of language structure in any given sentence can be determined before, during and after a text sample is being scribed. This intimate control over language model patterns, means that sensitive components such as "subject" and "object" can be determined easily by code.</em></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-08T10:07:56.357Z
Answers.Text: I think that is overly optimistic. Language is too messy for this to work.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-01-15T19:30:07.467Z
Answers.LastActivityDate: 2017-01-16T14:20:16.013Z
Answers.LastEditDate: 2017-01-16T14:20:16.013Z
Answers.UserDisplayName: Engage
Answers.Body: <blockquote>
    <p>Can there ever be a functionally complete set of grammar rules which can parse any statement in English (locale-specific) accurately and which can be possibly implemented for use in AI-based projects?</p>
  </blockquote>
  
  <p>Parse it yes, accurately most likely no.</p>
  
  <p>Why ? </p>
  
  <p>According to my understanding on how we derive meaning from sounds, there are 2 complementary strategies:</p>
  
  <p><strong>Grammar Rules:</strong>
  A rule based system for ordering words to facilitate communication, here meaning is derived from interaction of discrete sounds and their independent meaning, so you could parse a sentence based on a rule book.</p>
  
  <p>E.G. <strong><em>"This was a triumph"</em></strong> : the parser would extract a pronoun (<strong>This</strong>) with corresponding meaning ( a specific person or thing ) ; a verb (<strong>was</strong>) with corresponding meaning ( occurred ); ( <strong>a</strong>) and here we start with some parsing problems , what would the parser extract, a noun or an indefinite article ? An so we consult the grammar rule book, and settle for the meaning ( indefinite article any one of ), you have to parse the next word  and refer to it though, but let's gloss over that for now, and finally (<strong>triumph</strong>) a noun ( it could also be a verb, but thanks to the grammar rule book we settled for a noun with meaning: ( victory,conquest), so in the end we have ( joining the meanings ):</p>
  
  <p><strong>A specific thing occurred of victory.</strong> Close enough and I am glossing over a few other rules, but that's not the point, the other strategy is:</p>
  
  <p><strong>A lexical dictionary (or lexicon)</strong>
  Where words or sounds are associated with specific meaning. Here meaning is derived from one or more words or sounds as a unit. This introduces the problem to a parser, since well, it shouldn't parse anything.</p>
  
  <p>E.G. <strong><em>"Non Plus Ultra"</em></strong> And so the AI parser would recognize that this phrase is not to be parsed and instead matched with meaning :</p>
  
  <p>The highest point or culmination</p>
  
  <p>Lexical units introduce another issue in that they themselves could be part of the first example, and so you end up with recursion.</p>
  
  <blockquote>
    <p>if it is possible for a computer to parse a well-versed sentence written in English as if it were parsed by an adult English-speaking human?</p>
  </blockquote>
  
  <p>I believe it could be possible, most examples I've seen deal effectively with the grammar rule book or the lexicon part, but I am not aware of a combination of both, but in terms of programming, it could happen.</p>
  
  <p>Unfortunately even if you solve this problem, your AI would not really understand things in the strict sense, but rather present you with very elaborate synonyms, additionally context (as mentioned in the comments) plays a role into the grammar and lexicon strategies. </p>
  
  <blockquote>
    <p>If it cannot be represented using simple grammar rules, what kind of semantic structure can be used to generalize it?</p>
  </blockquote>
  
  <p>A mixed one where there are both grammar rules and a lexicon and both can change and be influenced based on the AI specific context and experience as well as a system for dealing with these objects could be one way.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-16T19:20:08.283Z
Answers.Text: My scope of English is restricted to formal usage. If for once we skip very complex entities like metaphors or idioms, would defining a context-sensitive grammar achieve the purpose?
Answers.UserDisplayName: skrtbhtngr
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-16T23:04:56.267Z
Answers.Text: @skrtbhtngr: well in theory yes, it would be a subset as formal language is, context will still be the hard problem though.
Answers.UserDisplayName: Keno
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-01-16T16:30:21.630Z
Answers.Text: Side case: "It was a Triumph" talking about a motor bike for various reason (first bike, burnt bike), Maybe we can argue it takes an uppercase to distinguish the brand from a noun.
Answers.UserDisplayName: Tensibai
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-08T13:25:39.607Z
Answers.Text: I agree that understanding the meaning of the text is the hard part. Just a simple sentence: "Peter went to the cinema." contains a lot of hidden information: Peter is male, he most probably went there to watch a film with his girlfriend, his location changed, etc... Building a model for example a graph based on the text is not enough, because it is not something static, it can describe multiple timelines, events and contexts, plus there is hidden information in every sentence you can infer and use to understand the previous or following sentences.
Answers.UserDisplayName: inf3rno
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-12T22:16:06.647Z
Answers.LastActivityDate: 2016-11-12T22:16:06.647Z
Answers.UserDisplayName: Keno
Answers.Body: <p>We've concluded that it is a two-faceted, circular problem: structure cannot be inferred without context but knowing the structure also helps infer the context. So, here is your complex solution: start with the context, which is determined by the combination of words in sentence (combinatorics and search problem), from there determine your structure, or "parse" (at this step you can also filter out some insignificant words or at least assign lesser weights to them), go back to the context, back to parsing, and on until you arrive at the meaning. Thus by iterative, recursive reduction the whole problem can be solved.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-02T16:18:23.760Z
Answers.LastActivityDate: 2016-12-02T16:18:23.760Z
Answers.UserDisplayName: postoronnim
Answers.Body: <p>I'm pretty sure that the answer is "no" in the strictest sense, since English simply doesn't have a formal definition.  That is, nobody controls English and publishes a formal grammar that everyone is required to adhere to. English is built up through an experiential process and it has contradictions and flaws, but the probabilistic nature of the human mind allows us to work around those.</p>
  
  <p>For example, that this "sentence":</p>
  
  <p><strong>This sentence no verb</strong></p>
  
  <p>Technically it's not a sentence at all, since it doesn't have a verb.  But did anybody have any problem understanding what it meant? Doubtful.  Try coming up with a formal rule for that though.  And that's just one example.</p>
  
  <p>Now, could you come up with a formal grammar that covers, maybe, 90% of cases, and is "good enough" for most practical uses? Possibly, maybe even probably.  But I am pretty sure it's not possible to get to 100%.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-14T10:01:39.583Z
Answers.Text: Your example is not grammatical, so why should a grammar contain a formal rule to describe it?
Answers.UserDisplayName: BlindKungFuMaster
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-15T21:39:47.590Z
Answers.Text: That's exactly my point.  It's clearly English and it's clearly understandable, but if fails to meet the commonly used definition of English grammar.  Hence it's an example of how real-world spoken English operates outside the bounds of a rigid grammar.
Answers.UserDisplayName: mindcrime
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-12T23:51:41.563Z
Answers.LastActivityDate: 2016-11-12T23:51:41.563Z
Answers.UserDisplayName: mindcrime
Body: <p>In programming languages, there is a set of grammar rules which govern the construction of valid statements and expressions. These rules help in parsing the programs written by the user.</p>
  
  <p>Can there ever be a functionally complete set of grammar rules which can parse any statement in English (locale-specific) <strong>accurately</strong> and which can be possibly implemented for use in AI-based projects?</p>
  
  <p>I know that there are a lot of NLP Toolkits available online, but they are not that effective. Most of them are trained using specific corpuses which sometimes fail to infer some complex correlations between various parts of an expression.</p>
  
  <p>In other words, what I am asking is that if it is possible for a computer to parse a well-versed sentence written in English as if it were parsed by an adult English-speaking human?</p>
  
  <p>EDIT:
  If it cannot be represented using simple grammar rules, what kind of semantic structure can be used to generalize it?</p>
  
  <p>EDIT2: This <a href="https://www.eecs.harvard.edu/shieber/Biblio/Papers/shieber85.pdf" rel="noreferrer">paper</a> proves the absence of context-freeness in natural languages. I am looking for a solution, even if it is too complex.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-11-12T00:43:02.270Z
Comments.Text: I don't have the references to back this up, but I think it has been proven mathematically that you can't do this-- that understanding English completely requires context cues that demand higher-level semantic reasoning that can't be contained in grammar rules.
Comments.UserDisplayName: antlersoft
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-08T13:00:43.427Z
Comments.Text: According to wikipedia most parts of natural languages are context free: https://en.wikipedia.org/wiki/Context-free_grammar#Linguistic_applications
Comments.UserDisplayName: inf3rno
Tags: <ai-design><natural-language-processing><semantics>
Title: Can the English language ever be generalized using a set of grammar rules?
UserDisplayName: skrtbhtngr
ThreadText: # Can the English language ever be generalized using a set of grammar rules?
  
  (Asked by: username_0 on 2016-11-11)
  
  In programming languages, there is a set of grammar rules which govern the construction of valid statements and expressions. These rules help in parsing the programs written by the user.
  
  Can there ever be a functionally complete set of grammar rules which can parse any statement in English (locale-specific) **accurately** and which can be possibly implemented for use in AI-based projects?
  
  I know that there are a lot of NLP Toolkits available online, but they are not that effective. Most of them are trained using specific corpuses which sometimes fail to infer some complex correlations between various parts of an expression.
  
  In other words, what I am asking is that if it is possible for a computer to parse a well-versed sentence written in English as if it were parsed by an adult English-speaking human?
  
  EDIT:
  If it cannot be represented using simple grammar rules, what kind of semantic structure can be used to generalize it?
  
  EDIT2: This [paper](https://www.eecs.harvard.edu/shieber/Biblio/Papers/shieber85.pdf) proves the absence of context-freeness in natural languages. I am looking for a solution, even if it is too complex.
  
  **Tags:** ai-design, natural-language-processing, semantics
  > **Comments:**
  > **username_1**: I don't have the references to back this up, but I think it has been proven mathematically that you can't do this-- that understanding English completely requires context cues that demand higher-level semantic reasoning that can't be contained in grammar rules.  
  > **username_2**: According to wikipedia most parts of natural languages are context free: https://en.wikipedia.org/wiki/Context-free_grammar#Linguistic_applications  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2016-11-12. Score: 7
  
  > Can there ever be a functionally complete set of grammar rules which can parse any statement in English (locale-specific) accurately and which can be possibly implemented for use in AI-based projects?
  
  Parse it yes, accurately most likely no.
  
  Why ?
  
  According to my understanding on how we derive meaning from sounds, there are 2 complementary strategies:
  
  **Grammar Rules:**
  A rule based system for ordering words to facilitate communication, here meaning is derived from interaction of discrete sounds and their independent meaning, so you could parse a sentence based on a rule book.
  
  E.G. ***"This was a triumph"*** : the parser would extract a pronoun (**This**) with corresponding meaning ( a specific person or thing ) ; a verb (**was**) with corresponding meaning ( occurred ); ( **a**) and here we start with some parsing problems , what would the parser extract, a noun or an indefinite article ? An so we consult the grammar rule book, and settle for the meaning ( indefinite article any one of ), you have to parse the next word and refer to it though, but let's gloss over that for now, and finally (**triumph**) a noun ( it could also be a verb, but thanks to the grammar rule book we settled for a noun with meaning: ( victory,conquest), so in the end we have ( joining the meanings ):
  
  **A specific thing occurred of victory.** Close enough and I am glossing over a few other rules, but that's not the point, the other strategy is:
  
  **A lexical dictionary (or lexicon)**
  Where words or sounds are associated with specific meaning. Here meaning is derived from one or more words or sounds as a unit. This introduces the problem to a parser, since well, it shouldn't parse anything.
  
  E.G. ***"Non Plus Ultra"*** And so the AI parser would recognize that this phrase is not to be parsed and instead matched with meaning :
  
  The highest point or culmination
  
  Lexical units introduce another issue in that they themselves could be part of the first example, and so you end up with recursion.
  
  > if it is possible for a computer to parse a well-versed sentence written in English as if it were parsed by an adult English-speaking human?
  
  I believe it could be possible, most examples I've seen deal effectively with the grammar rule book or the lexicon part, but I am not aware of a combination of both, but in terms of programming, it could happen.
  
  Unfortunately even if you solve this problem, your AI would not really understand things in the strict sense, but rather present you with very elaborate synonyms, additionally context (as mentioned in the comments) plays a role into the grammar and lexicon strategies.
  
  > If it cannot be represented using simple grammar rules, what kind of semantic structure can be used to generalize it?
  
  A mixed one where there are both grammar rules and a lexicon and both can change and be influenced based on the AI specific context and experience as well as a system for dealing with these objects could be one way.
  
  > **Comments:**
  > **username_0**: My scope of English is restricted to formal usage. If for once we skip very complex entities like metaphors or idioms, would defining a context-sensitive grammar achieve the purpose?  
  > **username_5**: @skrtbhtngr: well in theory yes, it would be a subset as formal language is, context will still be the hard problem though.  
  > **username_6**: Side case: "It was a Triumph" talking about a motor bike for various reason (first bike, burnt bike), Maybe we can argue it takes an uppercase to distinguish the brand from a noun.  
  > **username_2**: I agree that understanding the meaning of the text is the hard part. Just a simple sentence: "Peter went to the cinema." contains a lot of hidden information: Peter is male, he most probably went there to watch a film with his girlfriend, his location changed, etc... Building a model for example a graph based on the text is not enough, because it is not something static, it can describe multiple timelines, events and contexts, plus there is hidden information in every sentence you can infer and use to understand the previous or following sentences.  
  
  ---
  
  ### Answer by username_3 on 2017-01-15. Score: 3
  
  I strongly disagree with all the former comments. Not because they are wrong, -which they are not - but because they are misleading - though unintentionally.
  
  For example: If one looks at these problems from an academic position, the problems will always seem insurmountable. This is because everything is coldly assessed and calculated in isolation to everything else.
  
  The answer predominantly lies in ***word association***. You have to write a program that can process a vast database of digital books, to register every word and all the words in that language which are associated with it. Plus all the statistical information with each associated word and its associated punctuation.
  
  This will then give you the basis on which an AI can decide several things:
  
  1. Whether the structure of a given sentence is correct.
  2. If the structure is bad, what the probability is for determining the context and intent of what is being said.
  3. The correct meaning and application of a multifaceted word (Triumph), is by probability - according to the statistics.
  4. To determine where a conversation is likely to be going.
  5. What the correct grammar, and punctuation should be.
  
  So, in conclusion, you have two things to look for: Association and probability.
  
  *When digitally databasing a language model, the possibility of word and sentence "strings" occurs, so that every variation of language structure in any given sentence can be determined before, during and after a text sample is being scribed. This intimate control over language model patterns, means that sensitive components such as "subject" and "object" can be determined easily by code.*
  
  > **Comments:**
  > **username_4**: I think that is overly optimistic. Language is too messy for this to work.  
  
  ---
  
  ### Answer by username_7 on 2016-12-02. Score: 2
  
  We've concluded that it is a two-faceted, circular problem: structure cannot be inferred without context but knowing the structure also helps infer the context. So, here is your complex solution: start with the context, which is determined by the combination of words in sentence (combinatorics and search problem), from there determine your structure, or "parse" (at this step you can also filter out some insignificant words or at least assign lesser weights to them), go back to the context, back to parsing, and on until you arrive at the meaning. Thus by iterative, recursive reduction the whole problem can be solved.
  
  
  ---
  
  ### Answer by username_8 on 2016-11-12. Score: 1
  
  I'm pretty sure that the answer is "no" in the strictest sense, since English simply doesn't have a formal definition. That is, nobody controls English and publishes a formal grammar that everyone is required to adhere to. English is built up through an experiential process and it has contradictions and flaws, but the probabilistic nature of the human mind allows us to work around those.
  
  For example, that this "sentence":
  
  **This sentence no verb**
  
  Technically it's not a sentence at all, since it doesn't have a verb. But did anybody have any problem understanding what it meant? Doubtful. Try coming up with a formal rule for that though. And that's just one example.
  
  Now, could you come up with a formal grammar that covers, maybe, 90% of cases, and is "good enough" for most practical uses? Possibly, maybe even probably. But I am pretty sure it's not possible to get to 100%.
  
  > **Comments:**
  > **username_9**: Your example is not grammatical, so why should a grammar contain a formal rule to describe it?  
  > **username_8**: That's exactly my point.  It's clearly English and it's clearly understandable, but if fails to meet the commonly used definition of English grammar.  Hence it's an example of how real-world spoken English operates outside the bounds of a rigid grammar.  
  
  ---
  
  