Answers.Body: <p>Most model-fitting is stochastic, so you get different parameters every time you train, and you usually can't say that one algorithm will <em>always</em> give you a better-performing model.</p>
  <p>However, since you can retrain many times to get a distribution of models, you can use a statistical test like the <a href="https://en.wikipedia.org/wiki/Student%27s_t-test" rel="nofollow noreferrer">T-Test</a> to say &quot;algorithm A usually produces a better model than algorithm B,&quot; which is what they mean by &quot;statistically outperforms.&quot;</p>
  <p>p-value is usually set by consensus in the field. The higher the p-value, the less confidence you have that there's a statistical difference between the distribution of values being compared. 0.1 might be normal in a field where data is very expensive to collect (like risky, long-term studies of humans), but in machine learning, it's usually easy enough to retrain a model that 0.01 is common, and demonstrates very high confidence. To know more about selecting and interpreting the values, I recommend <a href="https://en.wikipedia.org/wiki/Statistical_significance" rel="nofollow noreferrer">Wikipedia's page on statistical significance</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T07:01:25.480Z
Answers.LastActivityDate: 2020-07-04T07:11:04.450Z
Answers.LastEditDate: 2020-07-04T07:11:04.450Z
Answers.UserDisplayName: alltom
Body: <p>I was reading this <a href="https://arxiv.org/abs/1911.08415?fbclid=IwAR06xkxojSP-Tb7xemUQWByzmmu_EM2WMorqVZOCw44JhB_pMA357E1OMoI#:%7E:text=GMAN%20adapts%20an%20encoder%2Ddecoder,temporal%20factors%20on%20traffic%20conditions" rel="nofollow noreferrer">paper</a> where they are stating the following:</p>
  <blockquote>
  <p>We also use the T-Test to test the significance of GMAN in 1 hour ahead prediction compared to Graph WaveNet. The p-value is less than 0.01, which demonstrates that GMAN statistically outperforms Graph WaveNet.</p>
  </blockquote>
  <p>What does &quot;Model A statistically outperforms B&quot; mean in this context? And how should the p-value threshold be selected?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-03T22:58:13.230Z
Comments.Text: You should read about hypothesis testing, null hypothesis, and p-values. These are basic statistical concepts. Maybe [Khan Academy](https://www.khanacademy.org/math/statistics-probability/significance-tests-one-sample) is a good place to start. The value of $p$ is typically a hyper-parameter and needs to be chosen before the experiment. To understand why, you should first understand the meaning of the p-value, which actually takes a little bit of time to grasp.
Comments.UserDisplayName: nbro
Tags: <deep-learning><terminology><papers><statistics>
Title: What does it mean when a model "statistically outperforms" another?
UserDisplayName: razvanc92
ThreadText: # What does it mean when a model "statistically outperforms" another?
  
  (Asked by: username_0 on 2020-07-03)
  
  I was reading this [paper](https://arxiv.org/abs/1911.08415?fbclid=IwAR06xkxojSP-Tb7xemUQWByzmmu_EM2WMorqVZOCw44JhB_pMA357E1OMoI#:%7E:text=GMAN%20adapts%20an%20encoder%2Ddecoder,temporal%20factors%20on%20traffic%20conditions) where they are stating the following:
  
  > We also use the T-Test to test the significance of GMAN in 1 hour ahead prediction compared to Graph WaveNet. The p-value is less than 0.01, which demonstrates that GMAN statistically outperforms Graph WaveNet.
  
  What does "Model A statistically outperforms B" mean in this context? And how should the p-value threshold be selected?
  
  **Tags:** deep-learning, terminology, papers, statistics
  
  > **Comments:**
  > **username_1**: You should read about hypothesis testing, null hypothesis, and p-values. These are basic statistical concepts. Maybe [Khan Academy](https://www.khanacademy.org/math/statistics-probability/significance-tests-one-sample) is a good place to start. The value of $p$ is typically a hyper-parameter and needs to be chosen before the experiment. To understand why, you should first understand the meaning of the p-value, which actually takes a little bit of time to grasp.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-04. Score: 4 | Accepted answer
  
  Most model-fitting is stochastic, so you get different parameters every time you train, and you usually can't say that one algorithm will *always* give you a better-performing model.
  
  However, since you can retrain many times to get a distribution of models, you can use a statistical test like the [T-Test](https://en.wikipedia.org/wiki/Student%27s_t-test) to say "algorithm A usually produces a better model than algorithm B," which is what they mean by "statistically outperforms."
  
  p-value is usually set by consensus in the field. The higher the p-value, the less confidence you have that there's a statistical difference between the distribution of values being compared. 0.1 might be normal in a field where data is very expensive to collect (like risky, long-term studies of humans), but in machine learning, it's usually easy enough to retrain a model that 0.01 is common, and demonstrates very high confidence. To know more about selecting and interpreting the values, I recommend [Wikipedia's page on statistical significance](https://en.wikipedia.org/wiki/Statistical_significance).
  
  
  ---
  
  