Answers.Body: <p>It was essentially a lack of control over crowd-sourced training data.</p>
  
  <p>While <a href="https://en.wikipedia.org/wiki/Tay_(bot)" rel="noreferrer">Tay</a> was initially set up with some conversational ability, it seemed to be programmed to learn from interactions with other users. Once users became aware of this, they basically gamed the bot by exposing it to inappropriate language, which Tay's algorithms then picked up and repeated. According to the <a href="https://en.wikipedia.org/wiki/Tay_(bot)" rel="noreferrer">Wikipedia article</a> on the topic, it is not known for sure whether its <em>repeat after me</em> facility was solely at fault, or if there was other behaviour that caused it.</p>
  
  <p>It's not really an example of <em>catastrophic forgetting</em>; for once we don't know how Tay worked internally. I would think it's just that it was overwhelmed by new data coming in which was different from the pre-set. It seems unlikely that the kind of language it was exposed to was in any way known in advance and part of its training set (and labelled as 'inappropriate').</p>
  
  <p>Essentially, the lesson from this is to never trust any unvetted input data for training, unless you want to risk people abusing this trust as happened in this case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-17T09:03:41.213Z
Answers.LastActivityDate: 2019-07-17T09:03:41.213Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>Looking at what happened, it was something similar. Though, the case differs in my eyes from one perspective: if it could only do a few comedy jokes, that probably is not a profound starting point to excel in Twitter.</p>
  
  <p>Firstly, Twitter is about real life, not about comedy. Discussions are sometimes tough and you easily end up to Social Media Bubbles, where only a certain kind of speaking style and topics is cultivated. So, even humans get on the wrong track there; why not a newbie bot? And, with jokes you would probably catch something about language itself, but not about the topics. So, becoming a jerk instead of a nice comedian is a logical direction, where the bot even has to go, at least a little, to communicate on same level, and not alone. </p>
  
  <p>The comedian dataset compared to a Twitter dataset is also very small in a technical sense, so talking about a mini trend overkilling a megatrend in this case is probably not true, because of amounts of examples available. </p>
  
  <p>So, catastrophic things happened in learning, but not catastrophic learning with the definition of that term. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-17T08:20:39.973Z
Answers.LastActivityDate: 2019-07-17T09:25:35.587Z
Answers.LastEditDate: 2019-07-17T09:25:35.587Z
Answers.UserDisplayName: mico
Body: <p>Tay was a chatbot, who learned from Twitter users.</p>
  
  <blockquote>
    <p>Microsoft's AI fam from the internet that's got zero chill. The more you talk the smarter Tay gets. — Twitter tagline.</p>
  </blockquote>
  
  <p>Microsoft trained the AI to have a basic ability to communicate, and taught it a few jokes from hired comedians before setting it lose to learn from its conversations.</p>
  
  <p>This was a mistake.</p>
  
  <p>But why did Tay go so wrong? Was this an example of <a href="https://ai.stackexchange.com/q/13289/125">catastrophic forgetting</a>, where short, recent trends override large, less recent training, or was it something else entirely?</p>
  
Tags: <chat-bots><history><catastrophic-forgetting>
Title: Was the corruption of Microsoft's "Tay" chatbot an example of catastrophic forgetting?
UserDisplayName: wizzwizz4
ThreadText: # Was the corruption of Microsoft's "Tay" chatbot an example of catastrophic forgetting?
  
  (Asked by: username_0 on 2019-07-16)
  
  Tay was a chatbot, who learned from Twitter users.
  
  > Microsoft's AI fam from the internet that's got zero chill. The more you talk the smarter Tay gets. — Twitter tagline.
  
  Microsoft trained the AI to have a basic ability to communicate, and taught it a few jokes from hired comedians before setting it lose to learn from its conversations.
  
  This was a mistake.
  
  But why did Tay go so wrong? Was this an example of [catastrophic forgetting](https://ai.stackexchange.com/q/13289/125), where short, recent trends override large, less recent training, or was it something else entirely?
  
  **Tags:** chat-bots, history, catastrophic-forgetting
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-07-17. Score: 6
  
  It was essentially a lack of control over crowd-sourced training data.
  
  While [Tay](https://en.wikipedia.org/wiki/Tay_(bot)) was initially set up with some conversational ability, it seemed to be programmed to learn from interactions with other users. Once users became aware of this, they basically gamed the bot by exposing it to inappropriate language, which Tay's algorithms then picked up and repeated. According to the [Wikipedia article](https://en.wikipedia.org/wiki/Tay_(bot)) on the topic, it is not known for sure whether its *repeat after me* facility was solely at fault, or if there was other behaviour that caused it.
  
  It's not really an example of *catastrophic forgetting*; for once we don't know how Tay worked internally. I would think it's just that it was overwhelmed by new data coming in which was different from the pre-set. It seems unlikely that the kind of language it was exposed to was in any way known in advance and part of its training set (and labelled as 'inappropriate').
  
  Essentially, the lesson from this is to never trust any unvetted input data for training, unless you want to risk people abusing this trust as happened in this case.
  
  
  ---
  
  ### Answer by username_2 on 2019-07-17. Score: 6
  
  Looking at what happened, it was something similar. Though, the case differs in my eyes from one perspective: if it could only do a few comedy jokes, that probably is not a profound starting point to excel in Twitter.
  
  Firstly, Twitter is about real life, not about comedy. Discussions are sometimes tough and you easily end up to Social Media Bubbles, where only a certain kind of speaking style and topics is cultivated. So, even humans get on the wrong track there; why not a newbie bot? And, with jokes you would probably catch something about language itself, but not about the topics. So, becoming a jerk instead of a nice comedian is a logical direction, where the bot even has to go, at least a little, to communicate on same level, and not alone.
  
  The comedian dataset compared to a Twitter dataset is also very small in a technical sense, so talking about a mini trend overkilling a megatrend in this case is probably not true, because of amounts of examples available.
  
  So, catastrophic things happened in learning, but not catastrophic learning with the definition of that term.
  
  
  ---
  
  