Answers.Body: <p>There's some impressive work going on at IBM under the name of &quot;<a href="https://eorder.sheridan.com/3_0/app/orders/11030/files/assets/common/downloads/Slonim.pdf" rel="nofollow noreferrer">Project Debater</a>&quot;, which already produced some impressive results, that you can check in this <a href="https://www.youtube.com/watch?v=3_yy0dnIc58" rel="nofollow noreferrer">video</a>.</p>
  <p>The project is about creating a system capable of debating with human on random topics by scraping the internet to get ground knowledge (mainly scientific papers, not random gibberish of course). The system is pretty massive, it include speech synthesis and lot of other components, what's interesting for you are the parts I highlighted in red, especially those on the left that contribute to generate the counter arguments to the human opponent (rebuttal construction).</p>
  <p>I won't go into the details cause all those three topics constitute entire branches of Natural Language Processing. But you can easily see how combined together they try to accomplish a rough version of deductive reasoning:</p>
  <ul>
  <li>claim detection: identifying the main argument of a sentence/document</li>
  <li>evidence detection: identifying what facts are presented along with the argument</li>
  <li>stance detection: identifying the positive/neutral/negative stance of each evidence (and possibly opinion based facts presented) toward the claim has been made.</li>
  </ul>
  <p>Assuming that we managed to extract all this information from a text (big time, unfortunately not easy at all) then we can build reasoning out of it. For example:</p>
  <ul>
  <li>Public schools are worse than private ones for children education -&gt; detected negative claim</li>
  <li>Researches show that there's an equal amount of graduated students coming from public and private schools. -&gt; detected evidence</li>
  <li>The previous evidence has a positive stance toward the initial claim -&gt; stance detection</li>
  </ul>
  <p>Putting everything together, since we have a negative claim followed by a positive evidence toward the same fact, we just found out that the document contains a contradiction.</p>
  <p><a href="https://i.sstatic.net/XCD3o.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XCD3o.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-21T18:39:39.957Z
Answers.LastActivityDate: 2022-03-21T18:39:39.957Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>Yes, current large language models since GPT-4 are able to find errors in reasoning, probably zero-shot, not much prompt engineering needed.</p>
  <p>Giving a precise probability for something being wrong is much harder, even if the the model knows it, it is hard to query. Rough estimates should be easy.</p>
  <hr />
  <p>Some examples using GPT-4:</p>
  <p>Prompt:</p>
  <blockquote>
  <p>Alice had blue shoes. Therefore, Bob had to eat the cake.</p>
  <p>Analyze the reasoning.</p>
  </blockquote>
  <p>Result:</p>
  <blockquote>
  <p>The reasoning presented in the statement is illogical and does not
  follow a coherent thought process. There is no apparent connection or
  correlation between Alice having blue shoes and Bob having to eat the
  cake. The conclusion does not follow from the premise, and the
  argument lacks a proper cause-and-effect relationship.</p>
  </blockquote>
  <p>Starting a new chat:</p>
  <p>Prompt:</p>
  <blockquote>
  <p>Alice had birthday cake. Therefore, Alice had to eat the cake.</p>
  </blockquote>
  <p>Result:</p>
  <blockquote>
  <p>It is not necessarily true that Alice had to eat the cake just because she had a birthday cake. While it is common for people to eat cake on their birthdays, Alice could have chosen not to eat it, shared it with others, or saved it for later.</p>
  </blockquote>
  <p>Starting a new chat:</p>
  <p>Prompt:</p>
  <blockquote>
  <p>Alice had blue shoes. Therefore, Alice had to shoe the blues.</p>
  </blockquote>
  <p>Result:</p>
  <blockquote>
  <p>This statement is a play on words. &quot;Alice had blue shoes&quot; means that Alice owned a pair of blue-colored shoes. &quot;Shoe the blues&quot; is a pun on the phrase &quot;sing the blues,&quot; which means to express one's troubles or sadness, often through music. In this case, the pun implies that because Alice had blue shoes, she had to express her sadness or troubles, possibly through wearing her blue shoes.</p>
  </blockquote>
  <p>These examples are not cherry picked.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-26T18:46:01.870Z
Answers.Text: With that in mind, can any current models convert real-world natural argumentation into something more formal, like a list of premises and conclusions? An example would be taking an Online argument thread, or political debate, then summarising its arguments cleanly.
Answers.UserDisplayName: Michael
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T16:43:16.873Z
Answers.Text: @user253751 I added your examples to the answer, thanks for asking. It shows that I did not cherry pick the question. I did not cherry pick the answers either.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T16:44:35.010Z
Answers.Text: @Michael Yes, summarizing is one of the most important tasks it is used for,
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T16:49:51.717Z
Answers.Text: @Michael To demonstrate, I asked it "What are the main points in:" followed by the text of your original question above. The answer was a list of 5 items:
  1. Poor reasoning and ignorance can lead to suffering and evil, and manipulative individuals often use faulty logic.
  2. AI is sometimes trained using this broken thought process, raising concerns about its potential impact.
  3. Fact-checking and language transformers are being developed to address this issue, but there is still a need for AI systems that can analyze reasoning in text. ...
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T16:52:40.093Z
Answers.Text: @Michael ... 4. The desired AI system should be able to process large text bodies, identify fallacies and misleading reasoning, and provide an output explaining the errors.
  5. The question raised is whether there are any AI systems available or in development that can analyze fallacious inferences in natural language text.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T16:57:49.267Z
Answers.Text: @user253751 I tried to read it carefully and found it a very valid interpretation. What do you think is wrong? I mean, it does not take it as nonsense, but as poetic, with a convincing interpretation.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T17:11:43.697Z
Answers.Text: @user253751 To ask GPT-4 for advice for an example for one that is actually true and not a fallacy, I prompted it with "The text below is a question and some comments under one of the answers. Commentator user253751 is looking for a specific kind of example in the last comment. Can you find one?" followed by the text of the question, the word "Comments:" and the text of the comments under this answer up to the comment here you ask for it (starting "okay,...").
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T17:15:08.063Z
Answers.Text: The result was scary good - note that I did not clean up the comment thread in any way, or clarify the question:
  
  One possible example that is true and not a fallacy could be:
  
  "Alice studied diligently for her exam. Therefore, Alice was well-prepared for the exam."
  
  In this case, the premise is that Alice studied diligently, and the conclusion is that she was well-prepared. The reasoning is sound, as diligent studying typically leads to being well-prepared for an exam.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T17:38:40.950Z
Answers.Text: @Michael That is worth a different question, but as an example, I asked it to summarize the first third (28 minutes) of the transcript of the presidential debate between Trump and Biden, on Oct. 22, 2020.  - the model of GPT-8 allows for a context of about 8,000 tokens, roughly 32,000 words. The prompt was "Summarize the 5 most important points, one line each." followed by https://eu.usatoday.com/story/news/politics/elections/2020/10/23/debate-transcript-trump-biden-final-presidential-debate-nashville/3740152001/ up to [27:44] See the 5 items in the following 2 comments for the result:
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T17:39:15.203Z
Answers.Text: 1. Trump claims a COVID-19 vaccine is coming within weeks and will be distributed by the military, while Biden focuses on encouraging mask-wearing and rapid testing.
  2. Biden emphasizes the need for transparency in vaccine development and criticizes Trump's handling of the pandemic, calling the current situation a "dark winter."
  3. Trump defends his administration's response to COVID-19, arguing that they are learning to live with the virus and asserting that the virus will eventually go away.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T17:39:55.813Z
Answers.Text: 4. Biden states that he would not shut down the country, but instead focus on shutting down the virus, suggesting increased safety measures for reopening businesses and schools.
  5. Both candidates address foreign interference in U.S. elections, with Biden vowing to make interfering countries pay a price and Trump defending his administration's toughness on Russia.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-24T12:17:17.987Z
Answers.LastActivityDate: 2023-03-28T17:29:11.250Z
Answers.LastEditDate: 2023-03-28T17:29:11.250Z
Answers.UserDisplayName: Volker Siegel
Body: <p>Poor reasoning, and ignorance in general, is the source of a lot of <a href="https://philosophy.stackexchange.com/questions/64954/is-the-moral-compass-of-an-individual-inherent-or-is-this-aspect-of-their-nature/89759#89759">suffering and evil</a>. Covertly erroneous logic is often used in manipulation. And much of this broken thought is being used directly in the training of AI.</p>
  <p>There has been talk of, and development in, <em>fact-checking</em>, such as for language transformers. But what about <em>reasoning</em>?</p>
  <p>The function in mind is specifically being able to process a potentially large text body, analysing all logic and implied relations for fallacy and other misleading reasoning. Perhaps shades of colour could indicate level of error. A bonus would be output listing and explaining the mistakes, maybe like compiler errors -- &quot;fallacy x between premise y and conclusion z&quot;.</p>
  <p><strong>Are any AI systems available, or in development, for finding and analysing <em>fallacious inference</em> in natural language text?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-21T09:13:24.457Z
Comments.Text: It is very complicated to even recognise inferences; deciding whether they are fallacious would require understanding of how the world works, and AI is a long way away from that. So the answer, I'm afraid, is no.
Comments.UserDisplayName: Oliver Mason
Tags: <natural-language-processing><reference-request>
Title: Are any AI systems available, or in development, for finding and analysing fallacious inference in natural language text?
UserDisplayName: Michael
ThreadText: # Are any AI systems available, or in development, for finding and analysing fallacious inference in natural language text?
  
  (Asked by: username_0 on 2022-03-19)
  
  Poor reasoning, and ignorance in general, is the source of a lot of [suffering and evil](https://philosophy.stackexchange.com/questions/64954/is-the-moral-compass-of-an-individual-inherent-or-is-this-aspect-of-their-nature/89759#89759). Covertly erroneous logic is often used in manipulation. And much of this broken thought is being used directly in the training of AI.
  
  There has been talk of, and development in, *fact-checking*, such as for language transformers. But what about *reasoning*?
  
  The function in mind is specifically being able to process a potentially large text body, analysing all logic and implied relations for fallacy and other misleading reasoning. Perhaps shades of colour could indicate level of error. A bonus would be output listing and explaining the mistakes, maybe like compiler errors -- "fallacy x between premise y and conclusion z".
  
  **Are any AI systems available, or in development, for finding and analysing *fallacious inference* in natural language text?**
  
  **Tags:** natural-language-processing, reference-request
  > **Comments:**
  > **username_1**: It is very complicated to even recognise inferences; deciding whether they are fallacious would require understanding of how the world works, and AI is a long way away from that. So the answer, I'm afraid, is no.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-03-21. Score: 2
  
  There's some impressive work going on at IBM under the name of "[Project Debater](https://eorder.sheridan.com/3_0/app/orders/11030/files/assets/common/downloads/Slonim.pdf)", which already produced some impressive results, that you can check in this [video](https://www.youtube.com/watch?v=3_yy0dnIc58).
  
  The project is about creating a system capable of debating with human on random topics by scraping the internet to get ground knowledge (mainly scientific papers, not random gibberish of course). The system is pretty massive, it include speech synthesis and lot of other components, what's interesting for you are the parts I highlighted in red, especially those on the left that contribute to generate the counter arguments to the human opponent (rebuttal construction).
  
  I won't go into the details cause all those three topics constitute entire branches of Natural Language Processing. But you can easily see how combined together they try to accomplish a rough version of deductive reasoning:
  
  * claim detection: identifying the main argument of a sentence/document
  * evidence detection: identifying what facts are presented along with the argument
  * stance detection: identifying the positive/neutral/negative stance of each evidence (and possibly opinion based facts presented) toward the claim has been made.
  
  Assuming that we managed to extract all this information from a text (big time, unfortunately not easy at all) then we can build reasoning out of it. For example:
  
  * Public schools are worse than private ones for children education -> detected negative claim
  * Researches show that there's an equal amount of graduated students coming from public and private schools. -> detected evidence
  * The previous evidence has a positive stance toward the initial claim -> stance detection
  
  Putting everything together, since we have a negative claim followed by a positive evidence toward the same fact, we just found out that the document contains a contradiction.
  
  [![enter image description here](https://i.sstatic.net/XCD3o.png)](https://i.sstatic.net/XCD3o.png)
  
  
  ---
  
  ### Answer by username_3 on 2023-03-24. Score: 1
  
  Yes, current large language models since GPT-4 are able to find errors in reasoning, probably zero-shot, not much prompt engineering needed.
  
  Giving a precise probability for something being wrong is much harder, even if the the model knows it, it is hard to query. Rough estimates should be easy.
  
  ---
  
  Some examples using GPT-4:
  
  Prompt:
  
  > Alice had blue shoes. Therefore, Bob had to eat the cake.
  >
  > Analyze the reasoning.
  
  Result:
  
  > The reasoning presented in the statement is illogical and does not
  > follow a coherent thought process. There is no apparent connection or
  > correlation between Alice having blue shoes and Bob having to eat the
  > cake. The conclusion does not follow from the premise, and the
  > argument lacks a proper cause-and-effect relationship.
  
  Starting a new chat:
  
  Prompt:
  
  > Alice had birthday cake. Therefore, Alice had to eat the cake.
  
  Result:
  
  > It is not necessarily true that Alice had to eat the cake just because she had a birthday cake. While it is common for people to eat cake on their birthdays, Alice could have chosen not to eat it, shared it with others, or saved it for later.
  
  Starting a new chat:
  
  Prompt:
  
  > Alice had blue shoes. Therefore, Alice had to shoe the blues.
  
  Result:
  
  > This statement is a play on words. "Alice had blue shoes" means that Alice owned a pair of blue-colored shoes. "Shoe the blues" is a pun on the phrase "sing the blues," which means to express one's troubles or sadness, often through music. In this case, the pun implies that because Alice had blue shoes, she had to express her sadness or troubles, possibly through wearing her blue shoes.
  
  These examples are not cherry picked.
  
  > **Comments:**
  > **username_0**: With that in mind, can any current models convert real-world natural argumentation into something more formal, like a list of premises and conclusions? An example would be taking an Online argument thread, or political debate, then summarising its arguments cleanly.  
  > **username_3**: @user253751 I added your examples to the answer, thanks for asking. It shows that I did not cherry pick the question. I did not cherry pick the answers either.  
  > **username_3**: @Michael Yes, summarizing is one of the most important tasks it is used for,  
  > **username_3**: @Michael To demonstrate, I asked it "What are the main points in:" followed by the text of your original question above. The answer was a list of 5 items: 1. Poor reasoning and ignorance can lead to suffering and evil, and manipulative individuals often use faulty logic. 2. AI is sometimes trained using this broken thought process, raising concerns about its potential impact. 3. Fact-checking and language transformers are being developed to address this issue, but there is still a need for AI systems that can analyze reasoning in text. ...  
  > **username_3**: @Michael ... 4. The desired AI system should be able to process large text bodies, identify fallacies and misleading reasoning, and provide an output explaining the errors. 5. The question raised is whether there are any AI systems available or in development that can analyze fallacious inferences in natural language text.  
  > **username_3**: @user253751 I tried to read it carefully and found it a very valid interpretation. What do you think is wrong? I mean, it does not take it as nonsense, but as poetic, with a convincing interpretation.  
  > **username_3**: @user253751 To ask GPT-4 for advice for an example for one that is actually true and not a fallacy, I prompted it with "The text below is a question and some comments under one of the answers. Commentator user253751 is looking for a specific kind of example in the last comment. Can you find one?" followed by the text of the question, the word "Comments:" and the text of the comments under this answer up to the comment here you ask for it (starting "okay,...").  
  > **username_3**: The result was scary good - note that I did not clean up the comment thread in any way, or clarify the question:  One possible example that is true and not a fallacy could be:  "Alice studied diligently for her exam. Therefore, Alice was well-prepared for the exam."  In this case, the premise is that Alice studied diligently, and the conclusion is that she was well-prepared. The reasoning is sound, as diligent studying typically leads to being well-prepared for an exam.  
  > **username_3**: @Michael That is worth a different question, but as an example, I asked it to summarize the first third (28 minutes) of the transcript of the presidential debate between Trump and Biden, on Oct. 22, 2020.  - the model of GPT-8 allows for a context of about 8,000 tokens, roughly 32,000 words. The prompt was "Summarize the 5 most important points, one line each." followed by https://eu.usatoday.com/story/news/politics/elections/2020/10/23/debate-transcript-trump-biden-final-presidential-debate-nashville/3740152001/ up to [27:44] See the 5 items in the following 2 comments for the result:  
  > **username_3**: 1. Trump claims a COVID-19 vaccine is coming within weeks and will be distributed by the military, while Biden focuses on encouraging mask-wearing and rapid testing. 2. Biden emphasizes the need for transparency in vaccine development and criticizes Trump's handling of the pandemic, calling the current situation a "dark winter." 3. Trump defends his administration's response to COVID-19, arguing that they are learning to live with the virus and asserting that the virus will eventually go away.  
  > **username_3**: 4. Biden states that he would not shut down the country, but instead focus on shutting down the virus, suggesting increased safety measures for reopening businesses and schools. 5. Both candidates address foreign interference in U.S. elections, with Biden vowing to make interfering countries pay a price and Trump defending his administration's toughness on Russia.  
  
  ---
  
  