Answers.Body: <p>An ROC curve, strictly speaking, does not represent &quot;<em><strong>a</strong></em> classifier&quot;, but a scoring rule that can be thresholded at different levels to yield many different classifiers. So, the line labeled &quot;perfect classifier&quot; is actually a bit of a misnomer - it's not actually a classifier, it's a scoring rule, that when combined with a threshold, yields a classifier with particular TPR/FPR.</p>
  <p>A &quot;perfect classifier&quot; yields a specific set of classifications that have 100% sensitivity and 100% specificity, which would be represented by the <em>single point</em> in the upper-left of the ROC space. Any other point in ROC space represents the existence of classification errors. Any of the other points on the &quot;perfect classifier&quot; line are classifiers that could be implemented with the &quot;perfect&quot; scoring rule, but which would be needlessly sub-optimal as they sacrifice sensitivity or specificity for no reason.</p>
  <p>For the other sub-optimal scoring curves, there may be domain reasons why one wishes to trade sensitivity for specificity or vice versa, meaning the whole ROC curve represents potentially useful classifiers. But with the perfect scoring curve, there is no reason to choose a threshold that gives less than 100% sensitivity/specificity, meaning the &quot;perfect ROC curve&quot; represents one classifier you'd actually use in practice (located at (0,1)), and many other classifiers which serve no practical purpose (which are located anywhere else on the ROC curve).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T15:06:10.140Z
Answers.LastActivityDate: 2024-10-14T16:43:26.670Z
Answers.LastEditDate: 2024-10-14T16:43:26.670Z
Answers.UserDisplayName: Nuclear Hoagie
Answers.Body: <p>“Perfect classifier” might take it too far, but “perfect separation” is a good description, where “perfect separation” means that there is some threshold where all predictions corresponding to one category are above that threshold while all predictions corresponding to the other category are on below that threshold. Consider, for example, if all predictions corresponding to category <span class="math-container">$0$</span> are between <span class="math-container">$0.1$</span> and <span class="math-container">$0.4$</span>, while all predictions corresponding to category <span class="math-container">$1$</span> are between <span class="math-container">$0.6$</span> and <span class="math-container">$0.9$</span>.</p>
  <p>In that case, the separating threshold gives a decision rule with perfect sensitivity and perfect specificity. However, other decision rules are possible. “Always classify as zero” gives perfect specificity at the expense of no sensitivity. “Always classify as one” gives perfect sensitivity at the expense of no specificity. These could correspond to thresholds of <span class="math-container">$2$</span> and <span class="math-container">$-1$</span>, respectively.</p>
  <p>If you use a threshold like <span class="math-container">$0.2$</span> or <span class="math-container">$0.7$</span>, however, you wind up with perfection in either sensitivity or specificity and and imperfect, but not zero, value of the other metric. This explains the points you see in the plot.</p>
  <p>Let's look at some plots (code given at the end).</p>
  <p><a href="https://i.sstatic.net/xMtemYiI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xMtemYiI.png" alt="boxplots" /></a></p>
  <p>From the box plots, we see that the lowest value of the blue group (1) is greater than the highest value of the red group (0): no overlap. That is, there is perfect separation between the two groups. Interpreting these as predictions for the subjects corresponding to groups 0 and 1, respectively, we can plot a ROC curve based on these.</p>
  <p><a href="https://i.sstatic.net/6HOvmX5B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6HOvmX5B.png" alt="ROC curve" /></a></p>
  <p>This shows the thresholds to which each point on the ROC plot corresponds. You can use many more values at the thresholds, as is common in software implementations of ROC curves (even when there is perfect separation).</p>
  <p><a href="https://i.sstatic.net/OkIpko18.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OkIpko18.png" alt="ROC curve many thresholds" /></a></p>
  <p>Finally, let's look at the box plots as we move the threshold from <span class="math-container">$-\infty$</span>...</p>
  <p><a href="https://i.sstatic.net/IYohHz4W.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IYohHz4W.png" alt="-Infinity threshold" /></a></p>
  <p>...to <span class="math-container">$0.25$</span>...</p>
  <p><a href="https://i.sstatic.net/6tnPVrBM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6tnPVrBM.png" alt="0.25 threshold" /></a></p>
  <p>...to <span class="math-container">$0.5$</span>...</p>
  <p><a href="https://i.sstatic.net/xVaRnHJi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xVaRnHJi.png" alt="0.5 threshold" /></a></p>
  <p>...to <span class="math-container">$0.75$</span>...</p>
  <p><a href="https://i.sstatic.net/M6NRYf0p.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/M6NRYf0p.png" alt="0.75 threshold" /></a></p>
  <p>...to <span class="math-container">$+\infty$</span>.</p>
  <p><a href="https://i.sstatic.net/eLESrgvI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eLESrgvI.png" alt="+Infinity threshold" /></a></p>
  <p>No matter where the threshold is, at least one group has its members predicted perfectly, so we always have at least one of sensitivity = 1 or specificity = 1. However, when we do not use a threshold that is between the highest prediction for group 0 and the lowest prediction for group 1, we do not get sensitivity = 1 and specificity = 1 simultaneously.</p>
  <p><strong>R CODE</strong></p>
  <pre><code>library(ModelMetrics)
  library(ggplot2)
  set.seed(2024)
  N &lt;- 1000
  p0 &lt;- runif(N, 0, 0.4)
  p1 &lt;- runif(N, 0.6, 1)
  y0 &lt;- rep(0, N)
  y1 &lt;- rep(1, N)
  p &lt;- c(p0, p1)
  y &lt;- c(y0, y1)
  
  d1 &lt;- data.frame(
    y = as.factor(y),
    p = p
  )
  g1 &lt;- ggplot(d1, aes(y = p, col = y)) +
    geom_boxplot() + 
    scale_color_manual(values=c(&quot;black&quot;,&quot;blue&quot;)) +
    theme(legend.position = &quot;bottom&quot;)
  
  thresholds &lt;- c(-Inf, 0.25, 0.5, 0.75, Inf)
  
  
  
  tprs &lt;- fprs &lt;- rep(NA, length(thresholds))
  
  for (i in 1:length(thresholds)){
    
    classifications &lt;- rep(0, length(p))
    
    # Determine which p-values are above threshold i
    # Assign those to category 1 and, otherwise, to category 0
    #
    idx &lt;- which(p &gt; thresholds[i])
    classifications[idx] &lt;- 1
    
    # Calculate TPR and FPR for ROC curve
    #
    tprs[i] &lt;- ModelMetrics::tpr(y, classifications)
    fprs[i] &lt;- 1 - ModelMetrics::tnr(y, classifications)
  }
  
  d2 &lt;- data.frame(
    FPR = fprs,
    TPR = tprs,
    Threshold = as.factor(thresholds)
  )
  
  g2 &lt;- ggplot(d2, aes(x = FPR, y = TPR, col = Threshold)) +
    geom_point() + 
    theme(legend.position = &quot;bottom&quot;)
  
  # Redo with thresholds at all values
  
  thresholds &lt;- c(-Inf, p, Inf)
  tprs &lt;- fprs &lt;- rep(NA, length(thresholds))
  
  for (i in 1:length(thresholds)){
    
    classifications &lt;- rep(0, length(p))
    
    # Determine which p-values are above threshold i
    # Assign those to category 1 and, otherwise, to category 0
    #
    idx &lt;- which(p &gt; thresholds[i])
    classifications[idx] &lt;- 1
    
    # Calculate TPR and FPR for ROC curve
    #
    tprs[i] &lt;- ModelMetrics::tpr(y, classifications)
    fprs[i] &lt;- 1 - ModelMetrics::tnr(y, classifications)
  }
  
  d3 &lt;- data.frame(
    FPR = fprs,
    TPR = tprs,
    Threshold = thresholds
  )
  
  g3 &lt;- ggplot(d3, aes(x = FPR, y = TPR, col = Threshold)) +
    geom_point() + 
    theme(legend.position = &quot;bottom&quot;)
  
  # Create a ROC curve using a standard package to compare to the results here
  #
  r &lt;- pROC::roc(y, p); r$thresholds`
  
  ################################################################################
  ################################################################################
  
  # Let's look at what happens when we apply the original five thresholds
  
  thresholds &lt;- c(-Inf, 0.25, 0.5, 0.75, Inf)
  Ds &lt;- Gs &lt;- list()
  for (i in 1:length(thresholds)){
  
      regions &lt;- data.frame(
      xmin = c(-Inf, -Inf),
      xmax = c(Inf, Inf),
      ymin = c(-Inf, thresholds[i]),
      ymax = c(thresholds[i], Inf),
      Classification = c(&quot;Classified as 0&quot;, &quot;Classified as 1&quot;)
    )
    Ds[[i]] &lt;- regions
    
    Gs[[i]] &lt;- ggplot() + 
      geom_rect(
        data = regions, aes(
          xmin = xmin, 
          ymin = ymin, 
          xmax = xmax,
          ymax = ymax, 
          fill = Classification
        )
      ) +
      geom_boxplot(
        data = d1, 
        aes(
          y = p,
          col = y
        )
      ) +
      scale_fill_manual(values=c(&quot;gold&quot;,&quot;lightgreen&quot;)) +
      scale_color_manual(values=c(&quot;black&quot;,&quot;blue&quot;)) +
      theme(legend.position = &quot;bottom&quot;)
    
  }
  
  # Look at the plots
  #
  g1
  g2
  g3
  Gs[[1]]
  Gs[[2]]
  Gs[[3]]
  Gs[[4]]
  Gs[[5]]
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T14:06:24.660Z
Answers.LastActivityDate: 2024-10-16T22:46:44.617Z
Answers.LastEditDate: 2024-10-16T22:46:44.617Z
Answers.UserDisplayName: Dave
Answers.Body: <p>Because this means you have already found all true positives. And after you found all true positives you are left with false positives only.</p>
  <p>Imagine three classifiers sorting candidates by probability of being positive (from left to right):</p>
  <pre><code>Random classifier:  0 1 0 1 0  0 0 1 0 1  0 1 0 1 0  1 0 1 0 1
  Better classifier:  1 0 1 1 1  1 1 0 1 1  0 0 1 0 0  0 0 0 1 0
  Perfect classifier: 1 1 1 1 1  1 1 1 1 1  0 0 0 0 0  0 0 0 0 0
  </code></pre>
  <p>Now imagine you are searching for fraud in insurance claims and you need one day for a claim to be checked. Using random classifier it would take you 20 days to catch all fraudulent claims. With better classifier you will find 8/10 (80%) positives after 10 days (starting from left) and 19 days for all fraudulent claims to be caught. Using perfect classifier you catch 100% of positives after just 10 days. What is left are false positives only, nothing but zeros.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T10:48:58.803Z
Answers.LastActivityDate: 2024-10-14T10:48:58.803Z
Answers.UserDisplayName: Pero
Answers.Body: <p>The &quot;PERFECT CLASSIFIER&quot; line is perfect because at the optimal threshold (somewhere near the top left corner, with TPR = 1 and FPR = 0), it makes no classification errors. The non-zero FPR points you see occur when you change the cutoff threshold score (for instance, due to imbalanced dataset or cost sensitive learning ), which includes more negatives in the positive class. The ROC curve is a summary of classifier behavior across <em>all</em> thresholds, but the classifier itself can still be perfect if you set the optimal threshold.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T00:59:09.803Z
Answers.Text: Question: Does the ROC curve represent a classifier with the same parameters under different thresholds ? Are we allowed to change the model parameters of the classifier in the ROC curve?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T01:43:04.173Z
Answers.Text: @DSPinfinity Indeed, see [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic): *Given a threshold parameter ${\displaystyle T}$, The ROC curve plots parametrically ${\displaystyle {\mbox{TPR}}(T)}$ versus ${\displaystyle {\mbox{FPR}}(T)}$ with 
   ${\displaystyle T}$ as the varying parameter.*
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T10:00:40.223Z
Answers.Text: are the model parameters fixed?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T15:25:02.520Z
Answers.Text: @DSPinfinity $T$ is a *varying* parameter of the model as explicitly described in my above comment.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T15:47:33.030Z
Answers.Text: @DSPinfinity: It's probably useful to consider two classifiers `f(x)` and `g(x)=2*f(x)`. Clearly, they have diffrent parameters, the same ROC curve, but `T_g`=`2*T_f`. And obviously, `g` is perfect if and only if `f` is perfect.
Answers.UserDisplayName: MSalters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-14T00:43:49.957Z
Answers.LastActivityDate: 2024-10-14T00:43:49.957Z
Answers.UserDisplayName: cinch
Body: <p>Consider the ROC curve below:</p>
  <p><a href="https://i.sstatic.net/65cLvmoB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/65cLvmoB.png" alt="enter image description here" /></a></p>
  <p>It is not clear to me why &quot;PERFECT CLASSIFIER&quot; line is really perfect because there are points on this line (see the green circles) for which false positive rate is non-zero.</p>
  
Tags: <machine-learning><classification>
Title: Unclear points regarding ROC curve in machine learning
UserDisplayName: DSPinfinity
ThreadText: # Unclear points regarding ROC curve in machine learning
  
  (Asked by: username_0 on 2024-10-13)
  
  Consider the ROC curve below:
  
  [![enter image description here](https://i.sstatic.net/65cLvmoB.png)](https://i.sstatic.net/65cLvmoB.png)
  
  It is not clear to me why "PERFECT CLASSIFIER" line is really perfect because there are points on this line (see the green circles) for which false positive rate is non-zero.
  
  **Tags:** machine-learning, classification
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2024-10-14. Score: 7 | Accepted answer
  
  The "PERFECT CLASSIFIER" line is perfect because at the optimal threshold (somewhere near the top left corner, with TPR = 1 and FPR = 0), it makes no classification errors. The non-zero FPR points you see occur when you change the cutoff threshold score (for instance, due to imbalanced dataset or cost sensitive learning ), which includes more negatives in the positive class. The ROC curve is a summary of classifier behavior across *all* thresholds, but the classifier itself can still be perfect if you set the optimal threshold.
  
  > **Comments:**
  > **username_0**: Question: Does the ROC curve represent a classifier with the same parameters under different thresholds ? Are we allowed to change the model parameters of the classifier in the ROC curve?  
  > **username_4**: @DSPinfinity Indeed, see [ROC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic): *Given a threshold parameter ${\displaystyle T}$, The ROC curve plots parametrically ${\displaystyle {\mbox{TPR}}(T)}$ versus ${\displaystyle {\mbox{FPR}}(T)}$ with   ${\displaystyle T}$ as the varying parameter.*  
  > **username_0**: are the model parameters fixed?  
  > **username_4**: @DSPinfinity $T$ is a *varying* parameter of the model as explicitly described in my above comment.  
  > **username_5**: @DSPinfinity: It's probably useful to consider two classifiers `f(x)` and `g(x)=2*f(x)`. Clearly, they have diffrent parameters, the same ROC curve, but `T_g`=`2*T_f`. And obviously, `g` is perfect if and only if `f` is perfect.  
  
  ---
  
  ### Answer by username_1 on 2024-10-14. Score: 5
  
  An ROC curve, strictly speaking, does not represent "***a*** classifier", but a scoring rule that can be thresholded at different levels to yield many different classifiers. So, the line labeled "perfect classifier" is actually a bit of a misnomer - it's not actually a classifier, it's a scoring rule, that when combined with a threshold, yields a classifier with particular TPR/FPR.
  
  A "perfect classifier" yields a specific set of classifications that have 100% sensitivity and 100% specificity, which would be represented by the *single point* in the upper-left of the ROC space. Any other point in ROC space represents the existence of classification errors. Any of the other points on the "perfect classifier" line are classifiers that could be implemented with the "perfect" scoring rule, but which would be needlessly sub-optimal as they sacrifice sensitivity or specificity for no reason.
  
  For the other sub-optimal scoring curves, there may be domain reasons why one wishes to trade sensitivity for specificity or vice versa, meaning the whole ROC curve represents potentially useful classifiers. But with the perfect scoring curve, there is no reason to choose a threshold that gives less than 100% sensitivity/specificity, meaning the "perfect ROC curve" represents one classifier you'd actually use in practice (located at (0,1)), and many other classifiers which serve no practical purpose (which are located anywhere else on the ROC curve).
  
  
  ---
  
  ### Answer by username_2 on 2024-10-14. Score: 4
  
  “Perfect classifier” might take it too far, but “perfect separation” is a good description, where “perfect separation” means that there is some threshold where all predictions corresponding to one category are above that threshold while all predictions corresponding to the other category are on below that threshold. Consider, for example, if all predictions corresponding to category $0$ are between $0.1$ and $0.4$, while all predictions corresponding to category $1$ are between $0.6$ and $0.9$.
  In that case, the separating threshold gives a decision rule with perfect sensitivity and perfect specificity. However, other decision rules are possible. “Always classify as zero” gives perfect specificity at the expense of no sensitivity. “Always classify as one” gives perfect sensitivity at the expense of no specificity. These could correspond to thresholds of $2$ and $-1$, respectively.
  If you use a threshold like $0.2$ or $0.7$, however, you wind up with perfection in either sensitivity or specificity and and imperfect, but not zero, value of the other metric. This explains the points you see in the plot.
  Let's look at some plots (code given at the end).
  
  From the box plots, we see that the lowest value of the blue group (1) is greater than the highest value of the red group (0): no overlap. That is, there is perfect separation between the two groups. Interpreting these as predictions for the subjects corresponding to groups 0 and 1, respectively, we can plot a ROC curve based on these.
  
  This shows the thresholds to which each point on the ROC plot corresponds. You can use many more values at the thresholds, as is common in software implementations of ROC curves (even when there is perfect separation).
  
  Finally, let's look at the box plots as we move the threshold from $-\infty$...
  
  ...to $0.25$...
  
  ...to $0.5$...
  
  ...to $0.75$...
  
  ...to $+\infty$.
  
  No matter where the threshold is, at least one group has its members predicted perfectly, so we always have at least one of sensitivity = 1 or specificity = 1. However, when we do not use a threshold that is between the highest prediction for group 0 and the lowest prediction for group 1, we do not get sensitivity = 1 and specificity = 1 simultaneously.
  R CODE
  library(ModelMetrics)
  library(ggplot2)
  set.seed(2024)
  N <- 1000
  p0 <- runif(N, 0, 0.4)
  p1 <- runif(N, 0.6, 1)
  y0 <- rep(0, N)
  y1 <- rep(1, N)
  p <- c(p0, p1)
  y <- c(y0, y1)
  
  d1 <- data.frame(
    y = as.factor(y),
    p = p
  )
  g1 <- ggplot(d1, aes(y = p, col = y)) +
    geom_boxplot() + 
    scale_color_manual(values=c("black","blue")) +
    theme(legend.position = "bottom")
  
  thresholds <- c(-Inf, 0.25, 0.5, 0.75, Inf)
  
  
  
  tprs <- fprs <- rep(NA, length(thresholds))
  
  for (i in 1:length(thresholds)){
    
    classifications <- rep(0, length(p))
    
    # Determine which p-values are above threshold i
    # Assign those to category 1 and, otherwise, to category 0
    #
    idx <- which(p > thresholds[i])
    classifications[idx] <- 1
    
    # Calculate TPR and FPR for ROC curve
    #
    tprs[i] <- ModelMetrics::tpr(y, classifications)
    fprs[i] <- 1 - ModelMetrics::tnr(y, classifications)
  }
  
  d2 <- data.frame(
    FPR = fprs,
    TPR = tprs,
    Threshold = as.factor(thresholds)
  )
  
  g2 <- ggplot(d2, aes(x = FPR, y = TPR, col = Threshold)) +
    geom_point() + 
    theme(legend.position = "bottom")
  
  # Redo with thresholds at all values
  
  thresholds <- c(-Inf, p, Inf)
  tprs <- fprs <- rep(NA, length(thresholds))
  
  for (i in 1:length(thresholds)){
    
    classifications <- rep(0, length(p))
    
    # Determine which p-values are above threshold i
    # Assign those to category 1 and, otherwise, to category 0
    #
    idx <- which(p > thresholds[i])
    classifications[idx] <- 1
    
    # Calculate TPR and FPR for ROC curve
    #
    tprs[i] <- ModelMetrics::tpr(y, classifications)
    fprs[i] <- 1 - ModelMetrics::tnr(y, classifications)
  }
  
  d3 <- data.frame(
    FPR = fprs,
    TPR = tprs,
    Threshold = thresholds
  )
  
  g3 <- ggplot(d3, aes(x = FPR, y = TPR, col = Threshold)) +
    geom_point() + 
    theme(legend.position = "bottom")
  
  # Create a ROC curve using a standard package to compare to the results here
  #
  r <- pROC::roc(y, p); r$thresholds`
  
  ################################################################################
  ################################################################################
  
  # Let's look at what happens when we apply the original five thresholds
  
  thresholds <- c(-Inf, 0.25, 0.5, 0.75, Inf)
  Ds <- Gs <- list()
  for (i in 1:length(thresholds)){
  
      regions <- data.frame(
      xmin = c(-Inf, -Inf),
      xmax = c(Inf, Inf),
      ymin = c(-Inf, thresholds[i]),
      ymax = c(thresholds[i], Inf),
      Classification = c("Classified as 0", "Classified as 1")
    )
    Ds[[i]] <- regions
    
    Gs[[i]] <- ggplot() + 
      geom_rect(
        data = regions, aes(
          xmin = xmin, 
          ymin = ymin, 
          xmax = xmax,
          ymax = ymax, 
          fill = Classification
        )
      ) +
      geom_boxplot(
        data = d1, 
        aes(
          y = p,
          col = y
        )
      ) +
      scale_fill_manual(values=c("gold","lightgreen")) +
      scale_color_manual(values=c("black","blue")) +
      theme(legend.position = "bottom")
    
  }
  
  # Look at the plots
  #
  g1
  g2
  g3
  Gs[[1]]
  Gs[[2]]
  Gs[[3]]
  Gs[[4]]
  Gs[[5]]
  
  
  ---
  
  ### Answer by username_3 on 2024-10-14. Score: 2
  
  Because this means you have already found all true positives. And after you found all true positives you are left with false positives only.
  Imagine three classifiers sorting candidates by probability of being positive (from left to right):
  Random classifier:  0 1 0 1 0  0 0 1 0 1  0 1 0 1 0  1 0 1 0 1
  Better classifier:  1 0 1 1 1  1 1 0 1 1  0 0 1 0 0  0 0 0 1 0
  Perfect classifier: 1 1 1 1 1  1 1 1 1 1  0 0 0 0 0  0 0 0 0 0
  
  Now imagine you are searching for fraud in insurance claims and you need one day for a claim to be checked. Using random classifier it would take you 20 days to catch all fraudulent claims. With better classifier you will find 8/10 (80%) positives after 10 days (starting from left) and 19 days for all fraudulent claims to be caught. Using perfect classifier you catch 100% of positives after just 10 days. What is left are false positives only, nothing but zeros.
  
  
  ---
  
  