Answers.Body: <p>Generally its paired. The bootstrapping personally I find concerning. I presume you are using it to augment your data set.</p>
  <p>Bootstrapping at best tends to be a controversial statistical operation and could result in the non-significance you are observing. Anyway if its not significant between high and low - thats your answer.</p>
  <p>You could of course perform a single ML classifying low, and high cancer risk against control. That would provide a method to understand the associations of miRNA.</p>
  <hr />
  <p>Bootstrapping issue, I had a think about this. It depends on how the bootstrapping is done. In a paired sample this (I assume) samples with replacement at each paired position for both ROCs. If this is not the case - the test is junk. However, if this is the case, its actually a good test, because the paired structure of the data is always preserved no matter how many bootstrap samples are performed. So like if a 100 were done and the pairwise statistic (which we don't know what it is - but something like KS test) and its &gt;0.05 for all 100 samples - I don't think there's any doubt thats a robust result.</p>
  <p>Again if the two ROCs are bootstrapped independently, in my opinion the test is junk.</p>
  <p><strong>Reason</strong> The concern of bootstrapping is it causes weird distributions which were not present in the original data. This is only okay if there's lots of experience about interpreting the result. For example, in phylogenetics there's decades of experience interpreting bootstrapping and a consensus about what it means. This often isn't the case for most other areas of statistics.</p>
  <p>I note this particular issue is really ramping up the rep score ;0)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-22T12:31:59.363Z
Answers.Text: I have already done a discrimination between Control vs Tumor (low + high). The AUC is quite good with 0.87. Because my Tumor Group as a whole is strongly shifted towards high risk Cancer, I wanted to also show that there is no difference between the AUC of Cancer (high risk) vs Control and the AUC of Cancer (low risk) vs. Control. This way I can strengthen my results.
Answers.UserDisplayName: Mischa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-22T12:44:36.400Z
Answers.Text: I see, I think that bootstrappng will bias towards a 'no-significant' difference result. See what the referees say I guess.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-22T16:43:14.163Z
Answers.Text: Can you please explain briefly, why you think that a paired test should be used? I just want to understand it. Thank you.
Answers.UserDisplayName: Mischa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-23T14:04:38.597Z
Answers.Text: @Mischa answered above. Bootstrapping is an opinionated area of statistics outside phylogenetics.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-22T12:25:28.477Z
Answers.LastActivityDate: 2020-10-23T14:32:17.923Z
Answers.LastEditDate: 2020-10-23T14:32:17.923Z
Answers.UserDisplayName: M__
Body: <p>I want to compare two AUCs using a bootstraping method from pROC package (roc.test).</p>
  <p>I have done a classification using Logistic Regression and Serum miRNAs as prediction factors. The Controls are in both cases the same, the test group is different.</p>
  <pre><code>1. Cancer (high risk) vs. Control = AUC 1
  2. Cancer (low risk) vs. Control = AUC 2
  </code></pre>
  <p>If I want to compare this two AUCs - should I use a paired or unpaired test?</p>
  <p>I would tend to using a paired test, because of exact the same Control group (Control groups are correlated).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-22T08:02:47.317Z
Comments.Text: Not sure this can be answered. What are ROCAUCs? Bootstrapping is usually non-parametric, please confirm. ROC curves are a special case of data integrity used in machine learning, I'm not sure thats what you mean. Bootstrapping might be used for data augmentation, we'd need to know if that was the reason. As a simple answer paired tests are more powerful - so yes. The only time this is questioned is if it doesn't give the answer you are looking for.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-22T11:29:56.247Z
Comments.Text: @Michael ROCAUC is a AUCs of a ROC Curve - nothing special. You are right with the Bootstrapping method - it is non-parametric. 
  So actually I want to compare two AUCs. Because in both cases I am using the same control group I am not sure if I need a paired or an unpaired test. In both cases the results are not significant and this was what I was looking for. So I do not really care about the concrete p-value - I just want to use the right test for the described case.
Comments.UserDisplayName: Mischa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-22T12:18:28.457Z
Comments.Text: Okay I see. It has to be a paired test without bootstrapping.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-22T12:25:20.787Z
Comments.Text: @Michael Why without bootstrapping? My sample size is quite low with approx. 35/ group. Hence I do not really can assume nomality. This why I wanted to use Bootstrapping. DeLong would be another possibile method.
Comments.UserDisplayName: Mischa
Tags: <r><cancer><design><logistic-regression>
Title: Comparing AUCs: Discrimination of same Control from different Test Group - paired or unpaired? miRNA
ThreadText: # Comparing AUCs: Discrimination of same Control from different Test Group - paired or unpaired? miRNA
  
  (Asked by: username_0 on 2020-10-22)
  
  I want to compare two AUCs using a bootstraping method from pROC package (roc.test).
  
  I have done a classification using Logistic Regression and Serum miRNAs as prediction factors. The Controls are in both cases the same, the test group is different.
  
  ```
  1. Cancer (high risk) vs. Control = AUC 1
  2. Cancer (low risk) vs. Control = AUC 2
  
  ```
  
  If I want to compare this two AUCs - should I use a paired or unpaired test?
  
  I would tend to using a paired test, because of exact the same Control group (Control groups are correlated).
  
  **Tags:** r, cancer, design, logistic-regression
  
  > **Comments:**
  > **username_1**: Not sure this can be answered. What are ROCAUCs? Bootstrapping is usually non-parametric, please confirm. ROC curves are a special case of data integrity used in machine learning, I'm not sure thats what you mean. Bootstrapping might be used for data augmentation, we'd need to know if that was the reason. As a simple answer paired tests are more powerful - so yes. The only time this is questioned is if it doesn't give the answer you are looking for.  
  > **username_0**: @Michael ROCAUC is a AUCs of a ROC Curve - nothing special. You are right with the Bootstrapping method - it is non-parametric.  So actually I want to compare two AUCs. Because in both cases I am using the same control group I am not sure if I need a paired or an unpaired test. In both cases the results are not significant and this was what I was looking for. So I do not really care about the concrete p-value - I just want to use the right test for the described case.  
  > **username_1**: Okay I see. It has to be a paired test without bootstrapping.  
  > **username_0**: @Michael Why without bootstrapping? My sample size is quite low with approx. 35/ group. Hence I do not really can assume nomality. This why I wanted to use Bootstrapping. DeLong would be another possibile method.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-22. Score: 1 | Accepted answer
  
  Generally its paired. The bootstrapping personally I find concerning. I presume you are using it to augment your data set.
  
  Bootstrapping at best tends to be a controversial statistical operation and could result in the non-significance you are observing. Anyway if its not significant between high and low - thats your answer.
  
  You could of course perform a single ML classifying low, and high cancer risk against control. That would provide a method to understand the associations of miRNA.
  
  ---
  
  Bootstrapping issue, I had a think about this. It depends on how the bootstrapping is done. In a paired sample this (I assume) samples with replacement at each paired position for both ROCs. If this is not the case - the test is junk. However, if this is the case, its actually a good test, because the paired structure of the data is always preserved no matter how many bootstrap samples are performed. So like if a 100 were done and the pairwise statistic (which we don't know what it is - but something like KS test) and its >0.05 for all 100 samples - I don't think there's any doubt thats a robust result.
  
  Again if the two ROCs are bootstrapped independently, in my opinion the test is junk.
  
  **Reason** The concern of bootstrapping is it causes weird distributions which were not present in the original data. This is only okay if there's lots of experience about interpreting the result. For example, in phylogenetics there's decades of experience interpreting bootstrapping and a consensus about what it means. This often isn't the case for most other areas of statistics.
  
  I note this particular issue is really ramping up the rep score ;0)
  
  > **Comments:**
  > **username_0**: I have already done a discrimination between Control vs Tumor (low + high). The AUC is quite good with 0.87. Because my Tumor Group as a whole is strongly shifted towards high risk Cancer, I wanted to also show that there is no difference between the AUC of Cancer (high risk) vs Control and the AUC of Cancer (low risk) vs. Control. This way I can strengthen my results.  
  > **username_1**: I see, I think that bootstrappng will bias towards a 'no-significant' difference result. See what the referees say I guess.  
  > **username_0**: Can you please explain briefly, why you think that a paired test should be used? I just want to understand it. Thank you.  
  > **username_1**: @username_0 answered above. Bootstrapping is an opinionated area of statistics outside phylogenetics.  
  
  ---
  
  