Calculating "accuracy", "recall" etc. without classification I have a set of models, that I'm comparing to each other with respect to prediction of a binary event. I'm using a few proper scores (Brier, log), but I also need accuracy, recall, sensitivity etc., since the people I work with are more likely to understand them.
However, I'd like to do this without applying an arbitrary classification threshold, so I'm currently I'm calculating the following, with forecast probability $f_k$ and event indicator $e_k$ for observation $k$:

*

*True positives $TP = \sum_k f_k[e_k = 1]$;

*True negatives $TN = \sum_k (1-f_k)[e_k = 0]$;

*False positives $FP = \sum_k f_k[e_k = 0]$;

*False negatives $FN = \sum_k (1-f_k)[e_k = 1]$.

This gives me an "average confusion" matrix, with real "counts" rather than integer ones, which I use to calculate the metrics as usual. Are there any issues / biases I'm introducing by doing this, in addition to the ones inherent to using improper scoring rules?

I think the main problem is that you will get the same problems as for the underlying KPIs, just in a probabilistic flavor. For instance, let's assume that conditional on your predictors, a given instance has a certain true probability or prevalence of belonging to the target class, and let us assume we calculate an "expected probabilistic accuracy" by the sum of the expectations of your $TP+TN$ fractions:

If the true probability of belonging to the target class is lower than $0.5$, maximizing our expected "probablistic accuracy" will pull our forecast $f$ towards $0$. And if the true probability is higher than $0.5$, we can maximize the expected "probabilistic accuracy" by $f\to 1$. The same problems will occur with precision, recall etc.

I also need accuracy, recall, sensitivity etc., since the people I work with are more likely to understand them.

To be honest, I think people only think they understand accuracy etc., because they seem to be easy to grasp. Just as a flat Earth seems "obvious", or Newtonian mechanics. However, I think that a flat Earth and Newtonian mechanics are more often useful approximations than accuracy and so forth.
R code:
true_probability <- c(0.3, 0.8)
opar <- par(mfrow=c(1,length(true_probability)),las=1,mai=c(.8,.8,.5,.1))
Forecast <- seq(0,1,by=0.01)
for ( ii in seq_along(true_probability) ) {
    ETP <- forecast*true_probability[ii]
    ETN <- (1-forecast)*(1-true_probability[ii])
    Accuracy <- ETP+ETN
    plot(Forecast,Accuracy,type="l",main=paste("True probability", true_probability[ii]))
}

