Body: <p>Let <span class="math-container">$A$</span> and <span class="math-container">$B$</span> be two models for a classification task. Let <span class="math-container">$x$</span> be a test set and <span class="math-container">$M$</span> be a metric for the classification task.  <span class="math-container">$X$</span> be a random variable on test sets.</p>
  <p>Now,</p>
  <p><span class="math-container">$M(A,x) = $</span> Score of model <span class="math-container">$A$</span> on test set <span class="math-container">$x$</span></p>
  <p><span class="math-container">$M(B,x)$</span> = Score of model <span class="math-container">$B$</span> on test set <span class="math-container">$x$</span></p>
  <p><span class="math-container">$\delta(x) =$</span> difference in performance of models wrt test set <span class="math-container">$x$</span>  <span class="math-container">$= M(A, x)-M(B,x)$</span></p>
  <p>Now, consider the following (statistical) hypothesis on the performance difference <span class="math-container">$\delta$</span></p>
  <p><span class="math-container">$$H_o : \delta(x)  \le 0$$</span>
  <span class="math-container">$$H_1 : \delta(x)  &gt; 0$$</span></p>
  <p>We define <span class="math-container">$p-$</span>value as follows</p>
  <p><span class="math-container">$$P(\delta(X) \ge \delta(x) | H_o  \text{is true} ) $$</span></p>
  <p>With this as context, I confused with the following paragraph (taken from p15 of <a href="https://web.stanford.edu/%7Ejurafsky/slp3/4.pdf" rel="nofollow noreferrer">Naive Bayes and Sentiment Classification</a>)</p>
  <blockquote>
  <p>So in our example, this <span class="math-container">$p-$</span>value is the probability that we would see
  <span class="math-container">$\delta(x)$</span> assuming <span class="math-container">$A$</span> is not better than B. If <span class="math-container">$\delta(x)$</span> is huge
  (let’s say <span class="math-container">$A$</span> has a very respectable <span class="math-container">$M$</span> of <span class="math-container">$.9$</span> and <span class="math-container">$B$</span> has a
  terrible <span class="math-container">$M$</span> of only <span class="math-container">$.2$</span> on <span class="math-container">$x$</span>), we might be surprised, since that
  would be extremely unlikely to occur if <span class="math-container">$H_0$</span> were in fact true, and so
  the <strong><span class="math-container">$p-$</span>value would be low</strong> (unlikely to have such a large <span class="math-container">$\delta$</span> if
  <span class="math-container">$A$</span> is in fact not better than <span class="math-container">$B$</span>). But if <span class="math-container">$\delta(x)$</span> is very small,
  it might be less surprising to us even if <span class="math-container">$H_0$</span> were true and <span class="math-container">$A$</span> is
  not really better than <span class="math-container">$B$</span>, and so the <span class="math-container">$p-$</span>value would be higher.</p>
  </blockquote>
  <p>It is told in the paragraph that <span class="math-container">$p-$</span>value is very low if <span class="math-container">$A's$</span> performance is better than <span class="math-container">$B$</span>.</p>
  <p>I am thinking that <span class="math-container">$p-$</span>value should be zero if <span class="math-container">$A's$</span> performance is better than <span class="math-container">$B$</span> since it is a disjoint event wrt <span class="math-container">$H_0$</span>. Where am I going wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-07T10:15:57.693Z
Comments.Text: I don't have a definitive answer and I am also not a statistician, but from my understanding of statistical tests, the p-value is a **probability** of your _test statistic_, which can, for example, be the t-statistic (for the t-test). Therefore, the p-value **depends on** the probability distribution of the associated test statistic, which depends on your **assumptions** about our data (the t-statistic is computed as a function of your data). I think to fully answer this question it's required to understand what the test statistic and the associated prob. distribution are here.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-07T10:19:57.770Z
Comments.Text: This is the type of question to which you should find more qualified people to answer on Cross Validated SE, as this is really a question about a statistics topic, although statistical tests are also used in AI and research areas that use or apply AI. I think we can consider this question on-topic here, as it's in the context of Naive Bayes and NLP, but it's something that we may want to discuss on meta.
Comments.UserDisplayName: nbro
Tags: <probability><statistical-ai>
Title: How can the probability of two disjoint events be non-zero?
UserDisplayName: hanugm
ThreadText: # How can the probability of two disjoint events be non-zero?
  
  (Asked by: username_0 on 2021-07-07)
  
  Let $A$ and $B$ be two models for a classification task. Let $x$ be a test set and $M$ be a metric for the classification task. $X$ be a random variable on test sets.
  
  Now,
  
  $M(A,x) = $ Score of model $A$ on test set $x$
  
  $M(B,x)$ = Score of model $B$ on test set $x$
  
  $\delta(x) =$ difference in performance of models wrt test set $x$ $= M(A, x)-M(B,x)$
  
  Now, consider the following (statistical) hypothesis on the performance difference $\delta$
  
  $$H\_o : \delta(x) \le 0$$
  $$H\_1 : \delta(x) > 0$$
  
  We define $p-$value as follows
  
  $$P(\delta(X) \ge \delta(x) | H\_o \text{is true} ) $$
  
  With this as context, I confused with the following paragraph (taken from p15 of [Naive Bayes and Sentiment Classification](https://web.stanford.edu/%7Ejurafsky/slp3/4.pdf))
  
  > So in our example, this $p-$value is the probability that we would see
  > $\delta(x)$ assuming $A$ is not better than B. If $\delta(x)$ is huge
  > (let’s say $A$ has a very respectable $M$ of $.9$ and $B$ has a
  > terrible $M$ of only $.2$ on $x$), we might be surprised, since that
  > would be extremely unlikely to occur if $H\_0$ were in fact true, and so
  > the **$p-$value would be low** (unlikely to have such a large $\delta$ if
  > $A$ is in fact not better than $B$). But if $\delta(x)$ is very small,
  > it might be less surprising to us even if $H\_0$ were true and $A$ is
  > not really better than $B$, and so the $p-$value would be higher.
  
  It is told in the paragraph that $p-$value is very low if $A's$ performance is better than $B$.
  
  I am thinking that $p-$value should be zero if $A's$ performance is better than $B$ since it is a disjoint event wrt $H\_0$. Where am I going wrong?
  
  **Tags:** probability, statistical-ai
  
  > **Comments:**
  > **username_1**: I don't have a definitive answer and I am also not a statistician, but from my understanding of statistical tests, the p-value is a **probability** of your _test statistic_, which can, for example, be the t-statistic (for the t-test). Therefore, the p-value **depends on** the probability distribution of the associated test statistic, which depends on your **assumptions** about our data (the t-statistic is computed as a function of your data). I think to fully answer this question it's required to understand what the test statistic and the associated prob. distribution are here.  
  > **username_1**: This is the type of question to which you should find more qualified people to answer on Cross Validated SE, as this is really a question about a statistics topic, although statistical tests are also used in AI and research areas that use or apply AI. I think we can consider this question on-topic here, as it's in the context of Naive Bayes and NLP, but it's something that we may want to discuss on meta.  
  
  ---
  
  ## 0 answers
  
  