Answers.Body: <p>Its a very good question and it is also a good study. What you are saying must be correct.</p>
  <p>I thought it was possible they used the correct critical probability in the original submission and a reviewer requested it was changed. However I checked their medRxiv upload <a href="https://www.medrxiv.org/content/10.1101/2020.07.16.20155606v1.full.pdf" rel="nofollow noreferrer">here</a> and the authors clearly use this threshold prior their formal submission.</p>
  <blockquote>
  <p>We performed stepwise conditional analysis 479 to identify additional
  independent signals by adjusting for the most significant amino acid
  480 position in each step until none met the significance threshold (
  P = 5 10^-8)</p>
  </blockquote>
  <p>Thus I don't know.</p>
  <p>If this paper is central to your area of interest I would use the corrected critical probability threshold and re-assess and extend their conclusions because 5 decimal places is a huge a difference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T12:31:02.833Z
Answers.LastActivityDate: 2022-05-19T12:31:02.833Z
Answers.UserDisplayName: M__
Answers.Body: <p>As a direct answer to your question, I would argue that the authors, in choosing this threshold, have demonstrated a lack of understanding of their process of analysis. But that answer is not a full answer, and it's a problematic answer because [almost] everyone who does population genetics does similar things. <strong>This p-value is not appropriate, not merely because it is at a genome-wide significance level, but because <em>no</em> threshold is appropriate.</strong></p>
  <p>To explain my perspective more, thresholding to identify &quot;statistically-significant&quot; results, in general, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5187603/#__p13" rel="nofollow noreferrer">is not a good approach</a>. Regardless of what threshold is used, p-values only indicate the confidence in a statistic's value (as it relates to fitting a particular statistical model), not importance or the size of the value.</p>
  <p>The authors of this paper, as with many other authors who have written similar papers, make an obvious statistical error in the legend of figure 3, where they state: &quot;One-field classical allele HLA-B*57 (P = 9.84 × 10−138) (a) and amino acid position 97 in HLA-B (Pomnibus = 1.86 × 10−184) (b) showed the strongest association signal.&quot; This is an error, because they have only reported here the p-values, and p-values have no relevance to the strength or magnitude of a statistic.</p>
  <p>Given this, it doesn't make sense to rank by p-value, and it especially doesn't make sense to display p-values on a graph and highlight p-value peaks as indicating association peaks. Those regions are <em>confidence</em> peaks.</p>
  <p>In an attempt to provide some constructive light around this, I will answer the obvious follow-up questions are &quot;How <em>should</em> p-values be used?&quot; and &quot;If p-values shouldn't be displayed or ranked, then what should?&quot;</p>
  <p>My preference is to use p-values as a loose filter, to exclude obviously incorrect results. In the case of a GWAS, I might, for example, choose to exclude any statistic that has a p-value above 0.1, leaving the remainder as <em>potentially</em> correct. It could possibly be argued that this is still thresholding. It is, in a sense, but importantly I am not saying that all values that pass the threshold are good, just that the ones that are excluded are all bad; I allow for some rubbish to end up in the good pile. I liken it to using a computer to find the things that look like diamonds, so that I don't waste my time inspecting dirt.</p>
  <p>After that filtering is done, the p-values should be ignored, and the descriptive statistic used to rank, determine significance, association, or whatever the goal of the analysis is. For GWAS, a commonly-used value is beta, which can be positive or negative depending on the direction of association, and relates to the variant difference between groups.</p>
  <p>If a list needs to be cut down, then this descriptive statistic should be used for thresholding, depending on the analysis. For example, it might be appropriate to display the 100 most extreme beta values, or threshold to exclude any beta values that have a magnitude less than 0.01, or display a manhattan-like plot with beta on the <span class="math-container">$y$</span> axis, and chromosome position on the <span class="math-container">$x$</span> axis.</p>
  <p>There shouldn't be set rules, because every situation is different - there's a human element to this. What I'm saying is that it should be the job of the researcher, not the computer, to identify which data are significant. Make the data set small enough that it can be processed by a person, then let them do the rest. I think Hannah Fry explains this quite well in <a href="https://www.youtube.com/watch?v=Rzhpf1Ai7Z4&amp;t=1944s" rel="nofollow noreferrer">this talk</a>:</p>
  <blockquote>
  <p>Now this is the stuff the algorithms are amazing at: looking for tiny, tiny clues in seemingly completely disconnected data sets.... Just incredibly, incredibly sensitive these algorithms, right? Just absolutely amazing what they can do. And yet remember that sensitivity does not make a perfect algorithm. That's only one half of the equation.... [Humans] are terrible at sensitivity, right? We are really, really bad. We miss things all the time. But, specificity, being specific, that's like our superpower.... And rather than choosing between human and machine, which is kind of the rhetoric that we get so much, why don't we exploit each other's strengths and just create much more of a partnership?.... Because the algorithm never gets tired, so let it trawl through all of that data and just highlight a few key areas of concern. And then the human never misdiagnoses. So they can come in, and just sweep up and have the final say.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-20T04:28:23.313Z
Answers.LastActivityDate: 2022-05-22T03:01:51.780Z
Answers.LastEditDate: 2022-05-22T03:01:51.780Z
Answers.UserDisplayName: gringer
Body: <p>Typically, a p-value of <span class="math-container">$p&lt;=5*10^{-8}$</span> is used for <strong>genome-wide</strong> association testing, as there are roughly <span class="math-container">$1/p$</span> independent LD blocks in the human genome, so it correct for multiple testing across the genome.</p>
  <p>HLA assocation tests, <a href="https://www.nature.com/articles/s41588-021-00935-7" rel="nofollow noreferrer">such as in this study</a>, only look at a fraction of the genome, something like chr6:28–34 Mb, which is approximately 0.2% of the genome used to conduct a true 'genome-wide' association study. However, they still use the same genome-wide p-value threshold of <span class="math-container">$p&lt;=5*10^{-8}$</span>.</p>
  <p>Is this not too conservative, since there is a much reduced multiple testing burden? Should the threshold not be something like <span class="math-container">$p/0.002$</span>?</p>
  
Tags: <r><gwas><p-values>
Title: Why is the p-value significance threshold for HLA association tests $5*10^{-8}$?
ThreadText: # Why is the p-value significance threshold for HLA association tests $5*10^{-8}$?
  
  (Asked by: username_0 on 2022-05-19)
  
  Typically, a p-value of $p<=5\*10^{-8}$ is used for **genome-wide** association testing, as there are roughly $1/p$ independent LD blocks in the human genome, so it correct for multiple testing across the genome.
  
  HLA assocation tests, [such as in this study](https://www.nature.com/articles/s41588-021-00935-7), only look at a fraction of the genome, something like chr6:28–34 Mb, which is approximately 0.2% of the genome used to conduct a true 'genome-wide' association study. However, they still use the same genome-wide p-value threshold of $p<=5\*10^{-8}$.
  
  Is this not too conservative, since there is a much reduced multiple testing burden? Should the threshold not be something like $p/0.002$?
  
  **Tags:** r, gwas, p-values
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-05-19. Score: 1
  
  Its a very good question and it is also a good study. What you are saying must be correct.
  
  I thought it was possible they used the correct critical probability in the original submission and a reviewer requested it was changed. However I checked their medRxiv upload [here](https://www.medrxiv.org/content/10.1101/2020.07.16.20155606v1.full.pdf) and the authors clearly use this threshold prior their formal submission.
  
  > We performed stepwise conditional analysis 479 to identify additional
  > independent signals by adjusting for the most significant amino acid
  > 480 position in each step until none met the significance threshold (
  > P = 5 10^-8)
  
  Thus I don't know.
  
  If this paper is central to your area of interest I would use the corrected critical probability threshold and re-assess and extend their conclusions because 5 decimal places is a huge a difference.
  
  
  ---
  
  ### Answer by username_2 on 2022-05-20. Score: 1
  
  As a direct answer to your question, I would argue that the authors, in choosing this threshold, have demonstrated a lack of understanding of their process of analysis. But that answer is not a full answer, and it's a problematic answer because [almost] everyone who does population genetics does similar things. **This p-value is not appropriate, not merely because it is at a genome-wide significance level, but because *no* threshold is appropriate.**
  
  To explain my perspective more, thresholding to identify "statistically-significant" results, in general, [is not a good approach](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5187603/#__p13). Regardless of what threshold is used, p-values only indicate the confidence in a statistic's value (as it relates to fitting a particular statistical model), not importance or the size of the value.
  
  The authors of this paper, as with many other authors who have written similar papers, make an obvious statistical error in the legend of figure 3, where they state: "One-field classical allele HLA-B\*57 (P = 9.84 × 10−138) (a) and amino acid position 97 in HLA-B (Pomnibus = 1.86 × 10−184) (b) showed the strongest association signal." This is an error, because they have only reported here the p-values, and p-values have no relevance to the strength or magnitude of a statistic.
  
  Given this, it doesn't make sense to rank by p-value, and it especially doesn't make sense to display p-values on a graph and highlight p-value peaks as indicating association peaks. Those regions are *confidence* peaks.
  
  In an attempt to provide some constructive light around this, I will answer the obvious follow-up questions are "How *should* p-values be used?" and "If p-values shouldn't be displayed or ranked, then what should?"
  
  My preference is to use p-values as a loose filter, to exclude obviously incorrect results. In the case of a GWAS, I might, for example, choose to exclude any statistic that has a p-value above 0.1, leaving the remainder as *potentially* correct. It could possibly be argued that this is still thresholding. It is, in a sense, but importantly I am not saying that all values that pass the threshold are good, just that the ones that are excluded are all bad; I allow for some rubbish to end up in the good pile. I liken it to using a computer to find the things that look like diamonds, so that I don't waste my time inspecting dirt.
  
  After that filtering is done, the p-values should be ignored, and the descriptive statistic used to rank, determine significance, association, or whatever the goal of the analysis is. For GWAS, a commonly-used value is beta, which can be positive or negative depending on the direction of association, and relates to the variant difference between groups.
  
  If a list needs to be cut down, then this descriptive statistic should be used for thresholding, depending on the analysis. For example, it might be appropriate to display the 100 most extreme beta values, or threshold to exclude any beta values that have a magnitude less than 0.01, or display a manhattan-like plot with beta on the $y$ axis, and chromosome position on the $x$ axis.
  
  There shouldn't be set rules, because every situation is different - there's a human element to this. What I'm saying is that it should be the job of the researcher, not the computer, to identify which data are significant. Make the data set small enough that it can be processed by a person, then let them do the rest. I think Hannah Fry explains this quite well in [this talk](https://www.youtube.com/watch?v=Rzhpf1Ai7Z4&t=1944s):
  
  > Now this is the stuff the algorithms are amazing at: looking for tiny, tiny clues in seemingly completely disconnected data sets.... Just incredibly, incredibly sensitive these algorithms, right? Just absolutely amazing what they can do. And yet remember that sensitivity does not make a perfect algorithm. That's only one half of the equation.... [Humans] are terrible at sensitivity, right? We are really, really bad. We miss things all the time. But, specificity, being specific, that's like our superpower.... And rather than choosing between human and machine, which is kind of the rhetoric that we get so much, why don't we exploit each other's strengths and just create much more of a partnership?.... Because the algorithm never gets tired, so let it trawl through all of that data and just highlight a few key areas of concern. And then the human never misdiagnoses. So they can come in, and just sweep up and have the final say.
  
  
  ---
  
  