DataCamp exercise about distributions I was studying some statistics in DataCamp and they assigned me this exercise that I can't solve. I tried speaking with people that know more statistics than me and we can't seem to agree in an answer. The exercise is as follows:

Since 1926, the Belmont Stakes is a 1.5 mile-long race of 3-year old thoroughbred horses. Secretariat ran the fastest Belmont Stakes in history in 1973. While that was the fastest year, 1970 was the slowest because of unusually wet and sloppy conditions. With these two outliers removed from the data set, compute the mean and standard deviation of the Belmont winners' times. Sample out of a Normal distribution with this mean and standard deviation using the np.random.normal() function and plot a CDF. Overlay the ECDF from the winning Belmont times. Are these close to Normally distributed?

The answer to this is yes, they are. I used the dataset they provided and calculated both the theoretical CDF and the ECDF. They match.

Unfortunately, Justin was not alive when Secretariat ran the Belmont in 1973. Do you think he will get to see a performance like that? To answer this, you are interested in how many years you would expect to wait until you see another performance like Secretariat's. How is the waiting time until the next performance as good or better than Secretariat's distributed? Choose the best answer.

The thing is, they say the distribution is exponencial, and I understand it. We can model the event by a Bernoulli trial, where we toss a coin that says if we see a fast horse or not in a given year. Knowing that the probability of seing such horse is small, this turns into a Poisson process and the time between poisson events is exponentially modelled.
My problem with this answer is that we only have a race per year and the time should be continuous to use the exponential distribution. Also, even though the probability of seing a fast horse is small, the number of trials is not high, so I don't even know why we can modell it as a Poisson process to begin with.
I appreciate your help!

Let's begin by answering the question as it stands.  Then we can respond to some of the points raised in comments.
The question wants you to make the following assumptions:

*

*The future will behave like the past, but "outliers" like Secretariat will not occur.


*The past results are characterized as a sequence of realizations of random variables that are

*

*independent

*identically distributed (thus, exhibit no trends over time)

*normal



*Your estimate of that common normal distributions is perfect -- there is no error in it.
With these assumptions, let $\mu$ be the common Normal mean, $\sigma$ the common Normal standard deviation, and $x_{min}$ Secretariat's time (144 seconds).  The chance of equaling or beating that time on each future run of the Belmont stakes therefore is the chance that a standard Normal variate $Z$ will be less than or equal to $p = (x_{min}- \mu)/\sigma,$ given by the standard Normal distribution function $\Phi.$
This describes a sequence of independent identically distributed Bernoulli$(p)$ variables.  The chance that the waiting time $T$ exceeds $N$ runs of the stakes (its survival function) is the chance the next $N$ values equal $0.$  That is the product of those chances (by the independence part of assumption $(2)$), given by
$$\Pr(T \gt N) = (1-p)^{N}.$$
This is a geometric distribution.  However, $p$ is tiny.  (It will be somewhere around $10^{-3},$ $10^{-4},$ or even less, depending on how you estimate $\mu$ and $\sigma.$)  Thus, to an excellent approximation,
$$\Pr(T \gt N) = (1-p)^{N} = ((1-p)^{1/p})^{pN} \approx \exp(-pN).$$
That is manifestly an exponential waiting time, consistent with your expectation that Poisson processes might play a role.  The expected waiting time is
$$E[T] = \sum_{N=0}^\infty \Pr(T \ge N) = \frac{1}{p} = \frac{1}{\Phi\left((x_{min}-\mu)/\sigma\right)}.$$
That will be hundreds to tens of thousands of runs (and therefore at least that many years).  Bear in mind that the assumptions $(1)$ - $(3)$ must continue to hold for at least Justin's lifetime for this to be a useful calculation.

Let's do a reality check.  The data at https://www.belmontstakes.com/history/past-winners/ give 94 winning times from 1926 through the present.  Suppose Justin expects to live another $N$ years.  Assume only the first two parts of $(2)$ -- namely, winning times are independent and identically distributed. In that case, the location of the best time during the entire time series of $94 + N$ results is randomly and uniformly distributed: it could occur at any time.  Consequently, Justin would compute a probability of $N/(94 + N)$ of observing the best time during the next $N$ years.  With this argument, they would have a greater than 50% chance of observing the best time provided $N$ exceeds $94.$  This is a couple of orders of magnitude shorter than the previous result.
Which should we believe?
We might appeal to the data.

This plot immediately shows most of the assumptions $(1) - (3)$ are implausible or must be modified.  There have been trends, leveling off around 1970 - 1990.  Consequently, "$94$" in the preceding calculations ought to be replaced by some value between c. $32$ and $52.$  If Justin is young, they should expect to see a new record time during their lifetime.
Relative to these trends, the times do display remarkably consistent randomness, as suggested by this time series plot of the residuals (differences between the winning times and their smoothed values):

Moreover, there is no significant evidence of lack of independence: the autocorrelations are negligible.

This justifies examining the univariate distribution of the residuals. Indeed, it is well described as a Normal distribution (the red curve), provided we ignore two of the $94$ data points.

These residuals (after removing the two extreme values) have a mean of $-0.015$ and a standard deviation of $1.268.$  The residual for the best time is $-4.732$ (almost five seconds better than expected that year based on the Loess fit).  The intended answer to the question, then, is tantamount to a mean waiting time of
$$E[T] = \frac{1}{\Phi\left((4.732-(-0.015))/1.268\right)} \approx 10^4.$$
The Belmont Stakes will never run that many times.
The post-Secretariat residuals have a bit more scatter than the older residuals, with a standard deviation of $1.5.$  An upper $95\%$ confidence limit for this estimate is $2.0.$  Using that instead of $1.268$ in the preceding calculation causes the estimated waiting time to drop two orders of magnitude from $10\,000$ to $100.$  That's still a long time, but it leaves some room for Justin to hope!  This shows how sensitive the answer is to what otherwise seems to be a minor technical issue, that of estimating the Normal parameters.

That still leaves several thorny issues:

*

*How should we treat the fact that the parameters of this distribution can only be estimated and are not very certain?
For instance, the solution is exquisitely sensitive to the estimate of $\sigma$ and that could be off by $30\%$ or more.  We can do this with the Delta method or bootstrapping.  You can read about these elsewhere on CV.


*What justifies ignoring the two "outliers"?
The correct answer is, nothing.  "Outlying" conditions will recur in the future -- one can almost guarantee that.  Since $2$ outliers have appeared in $94$ runs, we can predict (with 95% confidence) that between $0$ and $8$ will appear in the next $94$ runs, suggesting Justin has a chance of observing several unusual winning times (good or bad).  (The value $8$ is a nonparametric $95\%$ upper prediction limit.)


*Why can we assume that trends -- which clearly occurred in the past -- won't recur in the future?
For instance, it's difficult to believe that gradual improvements halted a third of a century ago in a competitive business like horse racing.  An alternative is that some countervailing effects may have slowed progress.  Maybe gradual warming of the climate adversely affects winning times?  Only a tiny, tiny effect is needed.  If the warming accelerates, the future curve might trend upwards.  That would make it less likely for any horse ever to beat Secretariat's time.

The moral of this post is that data analysis is not a matter of hiring a roomful of monkeys (pardon me, DataCamp graduates) to plug numbers into Poisson process calculators that spit out predictions and probabilities.  Data analysis, when practiced well, is a principled exploration of how data help us reason from assumptions to tentative conclusions.  As this case study shows, even slight differences in in the assumptions can lead to profound differences in the conclusions.  The value of a data analysis, then, lies not in its predictions but primarily in the care with which it is conducted, how well it exposes and examines the assumptions, and how it reveals their connections to the results.
