Given scores for a classifier and an expected positive rate, generate targets that achieve a given AUC score Say I have output scores (maybe they're logits or poorly calibrated probabilities) from a trained classifier like y_hat = [1, 2, 3, 4], an expected positive rate of 25%, and I want to generate targets that satisfy the expected positive rate and give the scores an AUC of 66.6%. Then a vector of targets like y = [0, 0, 1, 0] would give me what I need, because
mean(y) = 0.25
auc(y, y_hat) = 0.666

Is there an efficient way to generate, or preferably sample, targets like this? In reality, my score arrays have hundreds of millions of rows.

Here's an approach that's based the probabilistic interpretation of AUC, which is the likelihood that a random positive sample has a higher score than a random negative sample. To begin, order all of your samples according to your y_hat score measure. Now, assign positive target labels to the top target-prevalence % of samples. This first step gives you target labels at the desired prevalence with an AUC of 1  (all positive samples rank higher than all negative samples).
Next, swap target values until you sufficiently reduce the AUC. To do this most simply, change the lowest-ranked positive targets to negatives, and the lowest-ranked negative targets to positives. You need to swap enough labels to reach the desired AUC - to do this, you need to swap a portion of the positive targets equal to 1 minus the target AUC value. For example, to reach an AUC of 0.7, you need to swap labels between the bottom-ranked 30% of the positive samples and an equivalent number of bottom-ranked negative samples.
Ultimately, you get a target vector that goes Positive-Negative-Positive according to your ranked measure, with all negatives in the middle, AUC% of positives at the top, and 1-AUC% of positives at the bottom. Exactly AUC% of positive samples rank above all the negative samples, giving your desired AUC, and since you only swapped targets starting from the desired prevalence of positives, you also have the correct prevalence.
Note that this may not be a terribly "realistic" target vector, as the ROC curve only has 4 points on it (and 2 are the degenerate all-positive or all-negative classifiers). The situation is also far less "random" than usual, since you could actually identify positive cases as ones with the highest or lowest scores. To make something more realistic, you could continue to swap target labels between the Negative group and the upper and lower Positive groups symmetrically - move a Positive sample from the lower group up to outrank X negatives (increasing AUC), and also a Positive sample from the upper group down to under-rank X negatives (decreasing it again by the same amount).
