After training a binary classifier, are TPR & FPR independent of a test set? Assume I trained a binary classifier $f$ and I was able to extract an optimal decision value $t$ such that the binary output $\widehat{y}$ of the classifier is:
$$
\widehat{y} = \begin{cases}
    1, & f(x) \geq t \\
    0, & f(x) < t
\end{cases}
$$
Intrinsic to this classifier (at threshold $t$) are the True Positive Rate and False Positive Rate:

*

*$TPR = \Pr(\widehat{y}=1|y=1)$

*$FPR = \Pr(\widehat{y}=1|y=0)$
where $y$ is the true label of the sample.
Let's now assume that the classifier is deployed into a staging environment (call it $S$) for several weeks, where it will be presented with a fraction $N = \Pr(y=0|S)$ of negative and $P = \Pr(y=1|S)$ of positive examples.
My question is this: is it correct to expect the following fraction of positive classifications given the data in $S$?
$$
\begin{align}
\Pr(\widehat{y}=1|S) &= \Pr(\widehat{y}=1|y=1,S) \cdot \Pr(y=1|S) \\
&+ \Pr(\widehat{y}=1|y=0,S) \cdot \Pr(y=0|S) \\
&=  TPR \cdot P + FPR\cdot N
\end{align}
$$
That is, are $TPR, FPR$ independent of $S$?

*

*$\Pr(\widehat{y}=1|y=1,S) = \Pr(\widehat{y}=1|y=1) = TPR$

*$\Pr(\widehat{y}=1|y=0,S) = \Pr(\widehat{y}=1|y=0) = FPR$
My intuition is that this holds provided that the sample distribution in $S$ is similar to the hold-out test set used to compute $TPR$, but I need some help with formalizing this concept. Moreover, are there any other caveats?

Nope. Not really sure you can conclude anything useful here. If you think about your model as assigning probabilities of observations being true then I think the issue is more clear.
Example
Let's say you have a perfectly calibrated model (of the observations that are predicted to have 70% probability of being true, 70% of them will actually be true). You can have two datasets of N observations each that are both balanced (50% True). But, one dataset may be much more deterministic and all predictions are either 10% or 90%. The other may be more random and all predictions might be either 40% or 60%. Let's use a threshold of $S=0.5$. For the two datasets, the TPRs would be
$$
TPR_1
=\frac
{
\left(0*\frac{N}{2}*0.1\right)
+
\left(1*\frac{N}{2}*0.9\right)
}
{N/2}=0.9$$
$$
TPR_2
=\frac
{
\left(0*\frac{N}{2}*0.4\right)
+
\left(1*\frac{N}{2}*0.6\right)
}
{N/2}=0.6$$
