Answers.Body: <p>I like your choice of "induce" instead of "produce," because the delusions came from the users. This means the answer has to do mostly with human psychology; people come equipped with lots of mental machinery specialized for dealing with other humans and not very much mental machinery specialized for dealing with software. So ELIZA behaved in ways that some people classified it as a person and behaved accordingly, and others didn't.</p>
  
  <p>What features will trip up a person's internal person classification system seem like they vary heavily from person to person, and also with experience and familiarity. Going into more detail is, as mentioned in the comments, more appropriate for sites specializing on the human side of the keyboard.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-23T02:02:00.140Z
Answers.LastActivityDate: 2016-08-23T02:02:00.140Z
Answers.UserDisplayName: Matthew Gray
Body: <p><a href="http://www.alicebot.org/articles/wallace/eliza.html" rel="noreferrer">From Eliza to A.L.I.C.E.</a>:</p>
  
  <blockquote>
    <p>Weizenbaum tells us that he was shocked by the experience of releasing ELIZA (also known as "Doctor") to the nontechnical staff at the MIT AI Lab. Secretaries and nontechnical administrative staff thought the machine was a "real" therapist, and spent hours revealing their personal problems to the program. When Weizenbaum informed his secretary that he, of course, had access to the logs of all the conversations, she reacted with outrage at this invasion of her privacy. Weizenbaum was shocked by this and similar incidents to find that such a simple program could so easily deceive a naive user into revealing personal information.</p>
  </blockquote>
  
  <p>Wikipedia's article on the <a href="https://en.wikipedia.org/wiki/ELIZA_effect" rel="noreferrer">"ELIZA Effect"</a>:</p>
  
  <blockquote>
    <p>Though designed strictly as a mechanism to support "natural language conversation" with a computer, ELIZA's DOCTOR script was found to be surprisingly successful in eliciting emotional responses from users who, in the course of interacting with the program, began to ascribe understanding and motivation to the program's output. As Weizenbaum later wrote, <strong>"I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."</strong> Indeed, ELIZA's code had not been designed to evoke this reaction in the first place. Upon observation, researchers discovered users unconsciously assuming ELIZA's questions implied interest and emotional involvement in the topics discussed, <em>even when they consciously knew that ELIZA did not simulate emotion.</em></p>
  </blockquote>
  
  <p>ELIZA, despite its simplicity, was incredibly successful at its task of tricking other human beings. Even those who knew ELIZA was a bot would still talk to it. Obviously, ELIZA served as an inspiration for various other, more intelligent chatbots, such as <a href="http://www.nytimes.com/2015/08/04/science/for-sympathetic-ear-more-chinese-turn-to-smartphone-program.html?_r=0" rel="noreferrer">Xiaoice</a>. But I would like to know what <em>exactly</em> led to such a simple program like ELIZA to be so successful in the first place.</p>
  
  <p>This is very useful knowledge for a programmer since a simple program is one that would be easily maintainable.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-23T01:53:29.030Z
Comments.Text: I'm flagging to close this as off-topic because even though the question is interesting and *involves* AI, it is not a question *about* the AI, it is a question about humans and our real psychology, and probably belongs on https://psychology.stackexchange.com/.
Comments.UserDisplayName: Jason C
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-23T01:58:45.637Z
Comments.Text: (Also, it is a *very* broad question. As for the answer, though, check out the citations on the wiki page about the subject as a good starting point!)
Comments.UserDisplayName: Jason C
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-23T16:49:56.270Z
Comments.Text: I'd argue against closure, since some of the standards for judging whether or not we've achieved A.I. - like the Turing Test - are purely psychological. The Eliza Effect is a perfect illustration of why the Turing Test is itself delusional; simply making a mannequin that is good enough to pass for a human being in the dark does not make it a human being. It is thus directly relevant to the definition of A.I. If we didn't already have psychological tests for the definition and routinely discuss them here, I might be more supportive of closure.
Comments.UserDisplayName: SQLServerSteve
Tags: <history><turing-test><emotional-intelligence><chat-bots>
Title: Why was ELIZA able to induce "delusional thinking"?
UserDisplayName: Left SE On 10_6_19
ThreadText: # Why was ELIZA able to induce "delusional thinking"?
  
  (Asked by: username_0 on 2016-08-23)
  
  [From Eliza to A.L.I.C.E.](http://www.alicebot.org/articles/wallace/eliza.html):
  
  > Weizenbaum tells us that he was shocked by the experience of releasing ELIZA (also known as "Doctor") to the nontechnical staff at the MIT AI Lab. Secretaries and nontechnical administrative staff thought the machine was a "real" therapist, and spent hours revealing their personal problems to the program. When Weizenbaum informed his secretary that he, of course, had access to the logs of all the conversations, she reacted with outrage at this invasion of her privacy. Weizenbaum was shocked by this and similar incidents to find that such a simple program could so easily deceive a naive user into revealing personal information.
  
  Wikipedia's article on the ["ELIZA Effect"](https://en.wikipedia.org/wiki/ELIZA_effect):
  
  > Though designed strictly as a mechanism to support "natural language conversation" with a computer, ELIZA's DOCTOR script was found to be surprisingly successful in eliciting emotional responses from users who, in the course of interacting with the program, began to ascribe understanding and motivation to the program's output. As Weizenbaum later wrote, **"I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."** Indeed, ELIZA's code had not been designed to evoke this reaction in the first place. Upon observation, researchers discovered users unconsciously assuming ELIZA's questions implied interest and emotional involvement in the topics discussed, *even when they consciously knew that ELIZA did not simulate emotion.*
  
  ELIZA, despite its simplicity, was incredibly successful at its task of tricking other human beings. Even those who knew ELIZA was a bot would still talk to it. Obviously, ELIZA served as an inspiration for various other, more intelligent chatbots, such as [Xiaoice](http://www.nytimes.com/2015/08/04/science/for-sympathetic-ear-more-chinese-turn-to-smartphone-program.html?_r=0). But I would like to know what *exactly* led to such a simple program like ELIZA to be so successful in the first place.
  
  This is very useful knowledge for a programmer since a simple program is one that would be easily maintainable.
  
  **Tags:** history, turing-test, emotional-intelligence, chat-bots
  > **Comments:**
  > **username_1**: I'm flagging to close this as off-topic because even though the question is interesting and *involves* AI, it is not a question *about* the AI, it is a question about humans and our real psychology, and probably belongs on https://psychology.stackexchange.com/.  
  > **username_1**: (Also, it is a *very* broad question. As for the answer, though, check out the citations on the wiki page about the subject as a good starting point!)  
  > **username_2**: I'd argue against closure, since some of the standards for judging whether or not we've achieved A.I. - like the Turing Test - are purely psychological. The Eliza Effect is a perfect illustration of why the Turing Test is itself delusional; simply making a mannequin that is good enough to pass for a human being in the dark does not make it a human being. It is thus directly relevant to the definition of A.I. If we didn't already have psychological tests for the definition and routinely discuss them here, I might be more supportive of closure.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2016-08-23. Score: 4 | Accepted answer
  
  I like your choice of "induce" instead of "produce," because the delusions came from the users. This means the answer has to do mostly with human psychology; people come equipped with lots of mental machinery specialized for dealing with other humans and not very much mental machinery specialized for dealing with software. So ELIZA behaved in ways that some people classified it as a person and behaved accordingly, and others didn't.
  
  What features will trip up a person's internal person classification system seem like they vary heavily from person to person, and also with experience and familiarity. Going into more detail is, as mentioned in the comments, more appropriate for sites specializing on the human side of the keyboard.
  
  
  ---
  
  