Answers.Body: <p>Yes and no!</p>
  
  <p>There's no inherent reason that machine learning systems can't deal with extreme events. As a simple version, you can learn the parameters of a <a href="https://en.wikipedia.org/wiki/Weibull_distribution" rel="nofollow noreferrer">Weibull distribution</a>, or another extreme value model, from data.</p>
  
  <p>The bigger issue is with known-unknowns vs. unknown-unknowns. If you <em>know</em> that rare events are possible (as with, say, earthquake prediction), you can incorporate that knowledge into the models you develop, and you'll get something that works as well or better than humans in that domain. If you <em>don't</em> know that rare events are possible (as with, say, a stock market crash produced by correlated housing defaults), then your model will reflect that as well.</p>
  
  <p>I tend to think Taleb is being a bit unfair here: AI can't handle these kinds of events precisely because its creators (us) can't handle them! If we knew they were possible, then we could handle them pretty well, and AI could too.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-16T19:22:57.690Z
Answers.Text: Isn't the inability of these models (including human ones in the brain) to handle unknown-unknowns kind of his point? Since there is always a limit to the amount of accurate data we can gather in our samples and in instances of fat-tail distributions the effect of an outlier can be huge, whereas in a normal distribution the effect or damage of an extreme outlier will usually be pretty small. So it's as if he is saying this is a fundamental characteristic to knowledge and predictive systems, biological or machine based, hence why A.I. will be limited in certain domains.
Answers.UserDisplayName: Josiah Swaim
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-16T20:10:04.370Z
Answers.Text: Hmm. I think there are two issues. One is the claim that we can't handle fat-tailed distributions with AI. This is false. The other is that, if you don't know what distributions are suited to the problem you're studying (that is, if you don't really understand your problem), then you'll be surprised by unexpected events. This is true. I think Taleb is conflating the two issues, when really they are separate.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-16T18:38:44.547Z
Answers.LastActivityDate: 2019-06-19T20:50:42.203Z
Answers.LastEditDate: 2019-06-19T20:50:42.203Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>Yes, Taleb is right. There are practical and mathematical limits even an AI, no matter how powefull, cannot overcome. Taleb point is that some events are inherently unpredictable. The amount of data the we would need to correctly guess their distribution is so massive we could never build a machine large enough that would run for long enough to provide accurate predictions:</p>
  <p><a href="https://i.sstatic.net/gAzQ5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gAzQ5.png" alt="enter image description here" /></a></p>
  <p>Quote (and above image) from Taleb book on <a href="https://arxiv.org/pdf/2001.10488.pdf" rel="nofollow noreferrer">Statistical Consequences of Fat Tails</a> (emphasis mine)</p>
  <blockquote>
  <p>Once we leave the yellow zone, where the law of large numbers (LLN) largely
  works, and the central limit theorem (CLT) eventually ends up workin, then, we encounter convergence problems.</p>
  </blockquote>
  <blockquote>
  <p>Further up, in the top segment, there is no mean. We call it the Fuhgetaboudit. <strong>If you see something in that category, you go home and you don’t talk about it.</strong></p>
  </blockquote>
  <blockquote>
  <p>The traditional statisticians approach to thick tails has been to claim to assume a
  different distribution but keep doing business as usual, using same metrics, tests,
  and statements of significance. Once we leave the yellow zone, for which statistical
  techniques were designed (even then), things no longer work as planned. The
  next section presents a dozen issues, almost all terminal.</p>
  </blockquote>
  <p>Moreover, consider that not only you need a huge amount of data, but also that the interesting data-points for extremely rare events happen, as their name indicates, extremely rarely. So you might have to wait a literal eternity to ever have enough data to train the AI to make a prediction.</p>
  <p>That said, even while AI cannot escape the mathematical boundaries that make prediction of extreme events impossible, it might perhaps be able to learn to do what Taleb recommends us to do: When dealing with extreme events, avoid prediction, and instead come up with approaches that make allow you to survive regardless lack of predictive power.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T22:42:35.253Z
Answers.LastActivityDate: 2023-08-07T15:23:38.170Z
Answers.LastEditDate: 2023-08-07T15:23:38.170Z
Answers.UserDisplayName: Luxspes
Body: <p>So Taleb has two heuristics to generally describe data distributions. One is Mediocristan, which basically means things that are on a Gaussian distribution such as height and/or weight of people.</p>
  
  <p>The other is called Extremistan, which describes a more Pareto like or fat-tailed distribution. An example is wealth distribution, 1% of people own 50% of the wealth or something close to that and so predictability from limited data sets is much harder or even impossible. This is because you can add a single sample to your data set and the consequences are so large that it breaks the model, or has an effect so large that it cancels out any of the benefits from prior accurate predictions. In fact this is how he claims to have made money in the stock market, because everyone else was using bad, Gaussian distribution models to predict the market, which actually would work for a short period of time but when things went wrong, they went really wrong which would cause you to have net losses in the market.</p>
  
  <p>I found this video of Taleb being asked about AI. His claim is that A.I. doesn't work (as well) for things that fall into extremistan.</p>
  
  <p>Is he right? Will some things just be inherently unpredictable even with A.I.?</p>
  
  <p>Here is the video I am referring to <a href="https://youtu.be/B2-QCv-hChY?t=43m08s" rel="noreferrer">https://youtu.be/B2-QCv-hChY?t=43m08s</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-25T04:33:20.000Z
Comments.Text: Well, in Taleb's book "The Black Swan" about the events unknown-unknowns, you realize that these events are inherently unexpected. Therefore, based on that, yes, A.I. in this area won't work.
Comments.UserDisplayName: Manuel Ramos
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-10T11:08:53.150Z
Comments.Text: The video link appears to be broken; do you have a functioning link to the original video?
Comments.UserDisplayName: nutty about natty
Tags: <machine-learning><ai-design><probability><statistical-ai>
Title: Is Nassim Taleb right about AI not being able to accurately predict certain types of distributions?
UserDisplayName: Josiah Swaim
ThreadText: # Is Nassim Taleb right about AI not being able to accurately predict certain types of distributions?
  
  (Asked by: username_0 on 2018-08-16)
  
  So Taleb has two heuristics to generally describe data distributions. One is Mediocristan, which basically means things that are on a Gaussian distribution such as height and/or weight of people.
  
  The other is called Extremistan, which describes a more Pareto like or fat-tailed distribution. An example is wealth distribution, 1% of people own 50% of the wealth or something close to that and so predictability from limited data sets is much harder or even impossible. This is because you can add a single sample to your data set and the consequences are so large that it breaks the model, or has an effect so large that it cancels out any of the benefits from prior accurate predictions. In fact this is how he claims to have made money in the stock market, because everyone else was using bad, Gaussian distribution models to predict the market, which actually would work for a short period of time but when things went wrong, they went really wrong which would cause you to have net losses in the market.
  
  I found this video of Taleb being asked about AI. His claim is that A.I. doesn't work (as well) for things that fall into extremistan.
  
  Is he right? Will some things just be inherently unpredictable even with A.I.?
  
  Here is the video I am referring to <https://youtu.be/B2-QCv-hChY?t=43m08s>
  
  **Tags:** machine-learning, ai-design, probability, statistical-ai
  > **Comments:**
  > **username_1**: Well, in Taleb's book "The Black Swan" about the events unknown-unknowns, you realize that these events are inherently unexpected. Therefore, based on that, yes, A.I. in this area won't work.  
  > **username_2**: The video link appears to be broken; do you have a functioning link to the original video?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2018-08-16. Score: 8
  
  Yes and no!
  
  There's no inherent reason that machine learning systems can't deal with extreme events. As a simple version, you can learn the parameters of a [Weibull distribution](https://en.wikipedia.org/wiki/Weibull_distribution), or another extreme value model, from data.
  
  The bigger issue is with known-unknowns vs. unknown-unknowns. If you *know* that rare events are possible (as with, say, earthquake prediction), you can incorporate that knowledge into the models you develop, and you'll get something that works as well or better than humans in that domain. If you *don't* know that rare events are possible (as with, say, a stock market crash produced by correlated housing defaults), then your model will reflect that as well.
  
  I tend to think Taleb is being a bit unfair here: AI can't handle these kinds of events precisely because its creators (us) can't handle them! If we knew they were possible, then we could handle them pretty well, and AI could too.
  
  > **Comments:**
  > **username_0**: Isn't the inability of these models (including human ones in the brain) to handle unknown-unknowns kind of his point? Since there is always a limit to the amount of accurate data we can gather in our samples and in instances of fat-tail distributions the effect of an outlier can be huge, whereas in a normal distribution the effect or damage of an extreme outlier will usually be pretty small. So it's as if he is saying this is a fundamental characteristic to knowledge and predictive systems, biological or machine based, hence why A.I. will be limited in certain domains.  
  > **username_3**: Hmm. I think there are two issues. One is the claim that we can't handle fat-tailed distributions with AI. This is false. The other is that, if you don't know what distributions are suited to the problem you're studying (that is, if you don't really understand your problem), then you'll be surprised by unexpected events. This is true. I think Taleb is conflating the two issues, when really they are separate.  
  
  ---
  
  ### Answer by username_4 on 2023-07-29. Score: 0
  
  Yes, Taleb is right. There are practical and mathematical limits even an AI, no matter how powefull, cannot overcome. Taleb point is that some events are inherently unpredictable. The amount of data the we would need to correctly guess their distribution is so massive we could never build a machine large enough that would run for long enough to provide accurate predictions:
  
  [![enter image description here](https://i.sstatic.net/gAzQ5.png)](https://i.sstatic.net/gAzQ5.png)
  
  Quote (and above image) from Taleb book on [Statistical Consequences of Fat Tails](https://arxiv.org/pdf/2001.10488.pdf) (emphasis mine)
  
  > Once we leave the yellow zone, where the law of large numbers (LLN) largely
  > works, and the central limit theorem (CLT) eventually ends up workin, then, we encounter convergence problems.
  
  > Further up, in the top segment, there is no mean. We call it the Fuhgetaboudit. **If you see something in that category, you go home and you don’t talk about it.**
  
  > The traditional statisticians approach to thick tails has been to claim to assume a
  > different distribution but keep doing business as usual, using same metrics, tests,
  > and statements of significance. Once we leave the yellow zone, for which statistical
  > techniques were designed (even then), things no longer work as planned. The
  > next section presents a dozen issues, almost all terminal.
  
  Moreover, consider that not only you need a huge amount of data, but also that the interesting data-points for extremely rare events happen, as their name indicates, extremely rarely. So you might have to wait a literal eternity to ever have enough data to train the AI to make a prediction.
  
  That said, even while AI cannot escape the mathematical boundaries that make prediction of extreme events impossible, it might perhaps be able to learn to do what Taleb recommends us to do: When dealing with extreme events, avoid prediction, and instead come up with approaches that make allow you to survive regardless lack of predictive power.
  
  
  ---
  
  